huadongliu opened a new issue #2490:
URL: https://github.com/apache/iceberg/issues/2490
The table is created with
`SortOrder.builderFor(schema).withOrderId(1).asc("userId", NULLS_LAST).build()`
and populated with
`df.write().format("iceberg").mode("append").save(tableLocation)`. The table
has below schema and partition spec. I am trying to improve `userId` join and
query by sorting them in parquet partitions.
```
Schema schema = new Schema(
optional(1, "userId", Types.StringType.get()),
optional(2, "eventTime", Types.TimestampType.withZone()),
optional(3, "count", Types.IntegerType.get())
);
PartitionSpec spec = PartitionSpec.builderFor(schema)
.day("eventTime")
.truncate("userId", 2)
.build();
```
userId is not sorted in parquet data files. Is Spark iceberg write supposed
to respect SortOrder?
`df.sortWithinPartitions("userId").write().format("iceberg").mode("append").save(tableLocation)`
seems a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
