vinishjail97 commented on PR #650:
URL: https://github.com/apache/incubator-xtable/pull/650#issuecomment-2719325688
> Thanks for working on the PR @unical1988, added comments.
>
> There seems to be some confusion about extracting partition values, let me
know what you think of this.
>
> ```
> basePath/
> p1/.. (Can be recursive partitions for parquet files)
> p2/ ..
> p3/..
> .hoodie/ (Hudi Metadata)
> metadata/ (Iceberg metadata)
> _delta_log/ (Delta metadata)
> ```
>
> To extract the partition fields (emphasis on fields here not the actual
values) we can it in two ways:
>
> 1. Assume table is not partitioned, this would just sync the parquet files
in the target formats using the physical paths you have extracted in one of the
classes. When you read those tables, partition pruning won't work.
> 2. Ask user input (from YAML configuration) for the partition fields from
the parquet file schema. Many of these analytical datasets are partitioned by
date either through an actual date column in the parquet file or a timestamp
field through which the date is actually extracted.
```
public class InputPartitionColumn {
String fieldName;
PartitionTransformType transformType;
}
InputPartitionKeyConfig should be part of Table object in DatasetConfig.
1. No transform -> The values for partition keys in the parquet file are
concatenated and partitionPath is generated. Configuring this in InternalTable
object.
2. Transformation -> timestamp -> transform(timestamp) ->
year/date/month/xyz.parquet
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
