vinishjail97 opened a new issue, #959: URL: https://github.com/apache/incubator-xtable/issues/959
### Feature Request / Improvement Spark's `partitionBy` and Hive both write a partition column only into the directory name (`region=eu/`) and drop it from the data files. This is the most common layout for plain parquet tables, but the parquet source cannot sync it with partitioning today, which #937 documents as a limitation: - `ParquetPartitionSpecExtractor.spec()` resolves the partition field against the schema read from one file footer, so a column that is only in the path is not found. - `ParquetPartitionValueExtractor` parses every partition value with `SimpleDateFormat`, so an identity (`VALUE`) partition on a string or integer such as `sr_returned_date_sk=2451457` cannot be expressed even when the column is in the files. Without a partition config the table is synced as flat, so the partition column is missing from the target schema. A possible approach, which has been prototyped: - For `VALUE` fields, accept the column type as the third part of `xtable.parquet.source.partition_field_spec_config`, for example `region:VALUE:STRING` or `sr_returned_date_sk:VALUE:INT`. When the column is not in the footer schema, add it to the table schema as a nullable top-level field and mark the partition field as path-only in `transformOptions`. - Parse all-`VALUE` partition paths with the shared `PathBasedPartitionValuesExtractor`, which already handles `key=` prefixes and `__HIVE_DEFAULT_PARTITION__`. Each target already reads such a column correctly. Delta keeps partition values in `partitionValues` by design, Iceberg fills identity partition columns missing from a file from the manifest's partition tuple, and Hudi's Spark reader extracts them from the partition path. Validation of the prototype: TPC-DS 100GB `store_returns` written by Spark (2,004 files, 28,801,611 rows) synced to Iceberg and Hudi with `sr_returned_date_sk:VALUE:INT`. Every row read back from both targets matches Spark's own partition discovery over the same files, including 1,007,811 rows in `__HIVE_DEFAULT_PARTITION__`, and a filter on one partition value returns exactly that partition's rows. A local IT covers Iceberg, Delta and Hudi. Mixing `VALUE` and time-based partition fields is not handled by the prototype. Related: #901, #927. Anyone in the community is welcome to pick this up; it would be good to land in the upcoming 0.5.0 release. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
