chenhao-db commented on code in PR #49263:
URL: https://github.com/apache/spark/pull/49263#discussion_r1895976141
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala:
##########
@@ -390,6 +394,11 @@ object ParquetReadSupport extends Logging {
.named(parquetRecord.getName)
}
+ private def clipVariantSchema(parquetType: GroupType, variantStruct:
StructType): GroupType = {
+ // TODO(SHREDDING): clip `parquetType` to retain the necessary columns.
Review Comment:
It doesn't. in this function, we will have custom logic to clip
`parquetType` to retain the the necessary columns for reading `variantStruct`.
But this part will be in a future PR to avoid making the single PR too big.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala:
##########
@@ -390,6 +394,11 @@ object ParquetReadSupport extends Logging {
.named(parquetRecord.getName)
}
+ private def clipVariantSchema(parquetType: GroupType, variantStruct:
StructType): GroupType = {
+ // TODO(SHREDDING): clip `parquetType` to retain the necessary columns.
Review Comment:
It doesn't. in this function, we will have custom logic to clip
`parquetType` to retain the necessary columns for reading `variantStruct`. But
this part will be in a future PR to avoid making the single PR too big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
