vinishjail97 opened a new pull request, #985: URL: https://github.com/apache/incubator-xtable/pull/985
Fixes #927. #931 routed `--partitionspec` to the Parquet source, but a VALUE (identity) spec such as `order_date:VALUE`, the example in the issue and in the CLI help, still failed with an NPE because the Parquet source parsed every partition field as a date. - Parquet source: identity fields are read from their Hive style directory, unescaped, and typed by the source column; DATE directories become days since epoch. `__HIVE_DEFAULT_PARTITION__` maps to null. - A time transform without a path format (`col:DAY`) now fails fast with the expected form instead of an NPE later. - RunSync had the same gap as the spark-runtime before #931: it forwarded the spec only under the Hudi key. It now uses the Parquet key for a PARQUET source. Tests: unit tests for the extractor (8 of 9 fail on main) and RunSync key routing; a new ITParquetConversionSource case syncs `region:VALUE,order_date:VALUE` to Iceberg, Delta and Hudi with a full then an incremental sync. Also re-ran the issue's repro with the rebuilt spark-runtime bundle on Spark 3.4.3: Iceberg gets identity(order_date) with two partitions, Delta is partitioned on order_date. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
