Joy-2000 commented on issue #20438: URL: https://github.com/apache/hudi/issues/20438#issuecomment-6443171174
@cshuo Thanks for the detailed analysis and the comparison with 0.14.1. I checked the profiled Hudi 1.1.1 run. In this invocation of `projectIter`, `from == to` is true: the fields, their order, and their types are identical. The input row class is `ColumnarBatchRow`. Thus this Hudi projection does not reorder columns; it materializes the columnar row view as an `UnsafeRow`. The table configuration is: - No partition fields - No precombine field (I have not confirmed whether a separate ordering field is configured) - `hoodie.table.recordkey.fields=cs_item_sk,cs_order_number` - `hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator` I have not yet confirmed the exact partition-read options. Given the nonpartitioned table configuration, I am also unsure why the mixed-partition branch is reached. Could an effective `partitionSchema` be present here despite the table configuration? If Hudi's work here is only `ColumnarBatchRow` to `UnsafeRow` materialization, could this conversion or Spark's subsequent `UnsafeProjection` be avoided safely? Debug screenshots of the schemas and row type: <img width="2560" height="1379" alt="Image" src="https://github.com/user-attachments/assets/8daa6a2c-a0af-4cc4-bfa1-a919f8e6b13c"; /> <img width="5120" height="2832" alt="Image" src="https://github.com/user-attachments/assets/4bb77a10-27b3-478d-b6ae-16df89c95401"; /> <img width="2560" height="1417" alt="Image" src="https://github.com/user-attachments/assets/c605cd98-4412-438c-84fe-dcfbc1c6f24f"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
