hudi-agent commented on code in PR #20136:
URL: https://github.com/apache/hudi/pull/20136#discussion_r4135363981
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieParquetReadSupport.scala:
##########
@@ -62,12 +66,26 @@ object HoodieParquetReadSupport {
*
* @param requestedSchema the initial parquet schema requested by Spark
* @param fileSchema the actual parquet schema of the file
+ * @param dropMissingTopLevelFields whether a top-level field the file does
not have is dropped as
+ * well (the row-based reader) or kept (the
vectorized reader)
* @return a potentially updated schema with empty struct fields removed
*/
- def trimParquetSchema(requestedSchema: MessageType, fileSchema:
MessageType): MessageType = {
+ def trimParquetSchema(requestedSchema: MessageType,
+ fileSchema: MessageType,
+ dropMissingTopLevelFields: Boolean): MessageType = {
val trimmedFields = requestedSchema.getFields.asScala.map(field => {
if (fileSchema.containsField(field.getName)) {
Review Comment:
🤖 nit: the comment explaining why the field is dropped is quite long (7+
lines) for a single branch — consider trimming it to the core rationale and
moving the Spark 4.1/#20135 specifics to the method-level javadoc instead.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
