Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/14014#discussion_r71277147
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
---
@@ -442,13 +445,23 @@ private[parquet] class ParquetRowConverter(
private val elementConverter: Converter = {
val repeatedType = parquetSchema.getType(0)
val elementType = catalystSchema.elementType
- val parentName = parquetSchema.getName
- if (isElementType(repeatedType, elementType, parentName)) {
+ // At this stage, we're not sure whether the repeated field maps to
the element type or is
+ // just the syntactic repeated group of the 3-level standard LIST
layout. Here we try to
+ // convert the repeated field into a Catalyst type to see whether
the converted type matches
+ // the Catalyst array element type.
+ val guessedElementType = schemaConverter.convertField(repeatedType)
+
+ if (DataType.equalsIgnoreCompatibleNullability(guessedElementType,
elementType)) {
+ // If the repeated field corresponds to the element type, creates
a new converter using the
+ // type of the repeated field.
newConverter(repeatedType, elementType, new ParentContainerUpdater
{
override def set(value: Any): Unit = currentArray += value
})
} else {
+ // If the repeated field corresponds to the syntactic group in the
standard 3-level Parquet
+ // LIST layout, creates a new converter using the only child field
of the repeated field.
+ assert(!repeatedType.isPrimitive &&
repeatedType.asGroupType().getFieldCount == 1)
new ElementConverter(repeatedType.asGroupType().getType(0),
elementType)
--- End diff --
Can we add examples at here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
