JingsongLi commented on code in PR #9370:
URL: https://github.com/apache/paimon/pull/9370#discussion_r3932071223
##########
paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java:
##########
@@ -256,7 +259,8 @@ protected CatalogEnvironment newCatalogEnvironment(String
branch) {
public RowKeyExtractor createRowKeyExtractor() {
switch (bucketMode()) {
case HASH_FIXED:
- return new FixedBucketRowKeyExtractor(schema());
+ return new FixedBucketRowKeyExtractor(
+ schema(), PartitionBucketMapping.loadFromTable(this));
Review Comment:
[P2] This turns every `createRowKeyExtractor()` into a full
partition-manifest scan. In the Flink sink this method is called once on the
client for `RowDataChannelComputer` and again inside every writer subtask when
`StoreSinkWriteImpl` calls `table.newWrite`; each writer also performs another
full `PartitionBucketMapping.loadFromScan` in `FileSystemWriteRestore` (and
that restore is constructed even when the coordinator later replaces it). With
P writer subtasks, startup is therefore roughly 2P full scans/partition-map
builds, not the single extra scan described by the option/docs. On the large
partitioned tables this feature targets, that can create an object-store
request storm and duplicate the full mapping in every task. Please build the
mapping once per table/job and pass/share it with both routing and restore, or
make the writer-side lookup partition-scoped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
