JingsongLi commented on code in PR #11015:
URL: https://github.com/apache/paimon/pull/11015#discussion_r4493510672
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -351,6 +353,54 @@ protected DataStreamSink<?> buildDynamicBucketSink(
protected DataStreamSink<?> buildForFixedBucket(DataStream<InternalRow>
input) {
int bucketNums = table.bucketSpec().getNumBuckets();
+ Options options = Options.fromMap(table.options());
+ // sink.striped-bucket-write.enabled is the sole switch;
writers-per-bucket only applies
+ // when it is
+ // on
+ if
(options.get(FlinkConnectorOptions.SINK_STRIPED_BUCKET_WRITE_ENABLED)) {
Review Comment:
[P2] Reject striped writes while append bucket ordering is enabled
Ordinary fixed-bucket append tables default to `bucket-append-ordered=true`,
whose documented streaming contract preserves append order within a
partition/bucket. The new branch accepts that configuration but sends
successive rows to independent writers, whose files are read whole;
`write-only=true` does not disable the ordering contract.
I reproduced this with a single streaming Flink source emitting values 0–19
into one append bucket. The default sink reads back 0–19 from an actual
streaming DELTA scan. Enabling striping with two writers, still with all
required guards satisfied, reads `0,2,4,...,18,1,3,5,...,19`. Please require an
explicit `bucket-append-ordered=false` for striped append, or implement
coordinated ordering. A diagnostic guard rejects this conflicting configuration
while leaving the default and explicitly unordered write paths working.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StripedBucketWriteChannelComputer.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.sink.ChannelComputer;
+import org.apache.paimon.utils.MurmurHashUtils;
+
+/**
+ * A {@link RowDataChannelComputer} that stripes each bucket's writes across
multiple writer
+ * channels, so that write parallelism can exceed the bucket count without
rescaling the table.
+ *
+ * <p>Each bucket is routed to several channels. For a primary-key table the
record's primary key
+ * picks the channel, so records sharing a key still land on the same channel;
for an append-only
+ * table records are distributed round-robin. A single writer per bucket is
equivalent to {@link
+ * RowDataChannelComputer}.
+ */
+public class StripedBucketWriteChannelComputer extends RowDataChannelComputer {
+
+ private static final long serialVersionUID = 1L;
+
+ /** Derive the number of stripes from the channel count and the bucket
count. */
+ public static final int AUTO = 0;
+
+ /** Route every bucket to all channels, so every writer writes every
bucket. */
+ public static final int MATCH_PARALLELISM = -1;
+
+ private final int writesPerBucket;
+ private final int numBuckets;
+ private final boolean hasPrimaryKey;
+
+ private transient int numStripes;
+ private transient int nextStripe;
+
+ public StripedBucketWriteChannelComputer(TableSchema schema, int
writesPerBucket) {
+ super(schema);
+ this.writesPerBucket = writesPerBucket;
+ this.numBuckets = new CoreOptions(schema.options()).bucket();
+ this.hasPrimaryKey = !schema.primaryKeys().isEmpty();
+ }
+
+ @Override
+ public void setup(int numChannels) {
+ super.setup(numChannels);
+ this.numStripes = resolveNumStripes();
+ }
+
+ private int resolveNumStripes() {
+ if (writesPerBucket > 0) {
+ return writesPerBucket;
+ }
+ if (writesPerBucket == MATCH_PARALLELISM) {
+ return numChannels;
+ }
+ // AUTO: ceil(numChannels / numBuckets), but never less than one
writer per bucket
+ return Math.max(1, (numChannels + numBuckets - 1) / numBuckets);
+ }
+
+ @Override
+ public int channel(InternalRow record) {
+ extractor.setRecord(record);
+ BinaryRow partition = extractor.partition();
+ int bucket = extractor.bucket();
+ int stripe =
+ hasPrimaryKey
+ ? keyStripe(extractor.trimmedPrimaryKey())
+ : Math.floorMod(nextStripe++, numStripes);
+ // Widen to long so that bucket * numStripes cannot overflow for large
bucket counts
+ long bucketSlot = (long) bucket * numStripes + stripe;
+ int startChannel = ChannelComputer.startChannel(partition,
numChannels);
+ return (int) ((startChannel + bucketSlot) % numChannels);
+ }
+
+ private int keyStripe(BinaryRow key) {
+ // Avalanche the key hash so the stripe is independent of the bucket
modulo, which also
+ // derives from the key hash; otherwise shared factors between
numStripes and numBuckets
+ // skew it.
+ return Math.floorMod(MurmurHashUtils.fmix(key.hashCode()), numStripes);
Review Comment:
[P2] Make stripe hashing agree with logical primary-key equality
Hashing the raw `BinaryRow` bytes can send logically equal floating-point
keys to different writers. With `PK(tenant INT, k DOUBLE)` and
`bucket-key=tenant`, the accepted quiet-NaN payloads `0x7ff8000000000001` and
`0x7ff8000000000003` compare equal in Paimon's generated key comparator but
hash to different stripes. The real Flink row converter/wrapper preserves these
payload bits before partitioning.
I reproduced the storage effect with the actual router and public
writers/commits, then reopened the table. After priming the two writers with
different numbers of other keys, the first NaN upsert has sequence 100 on one
writer; the later equal-key upsert has sequence 1 on another. The current
striped path returns the stale value 10 instead of the later value 20. The
original bucket-only route returns 20; changing only a routing-row copy to
canonicalize NaNs also returns 20, while writing the original records
unchanged. All 102 logical rows remain present. Please hash according to the
key comparator's equality (including NaN normalization), so equal primary keys
cannot acquire independent sequence streams.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StripedBucketWriteChannelComputer.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.sink.ChannelComputer;
+import org.apache.paimon.utils.MurmurHashUtils;
+
+/**
+ * A {@link RowDataChannelComputer} that stripes each bucket's writes across
multiple writer
+ * channels, so that write parallelism can exceed the bucket count without
rescaling the table.
+ *
+ * <p>Each bucket is routed to several channels. For a primary-key table the
record's primary key
+ * picks the channel, so records sharing a key still land on the same channel;
for an append-only
+ * table records are distributed round-robin. A single writer per bucket is
equivalent to {@link
+ * RowDataChannelComputer}.
+ */
+public class StripedBucketWriteChannelComputer extends RowDataChannelComputer {
+
+ private static final long serialVersionUID = 1L;
+
+ /** Derive the number of stripes from the channel count and the bucket
count. */
+ public static final int AUTO = 0;
+
+ /** Route every bucket to all channels, so every writer writes every
bucket. */
+ public static final int MATCH_PARALLELISM = -1;
+
+ private final int writesPerBucket;
+ private final int numBuckets;
+ private final boolean hasPrimaryKey;
+
+ private transient int numStripes;
+ private transient int nextStripe;
+
+ public StripedBucketWriteChannelComputer(TableSchema schema, int
writesPerBucket) {
+ super(schema);
+ this.writesPerBucket = writesPerBucket;
+ this.numBuckets = new CoreOptions(schema.options()).bucket();
+ this.hasPrimaryKey = !schema.primaryKeys().isEmpty();
+ }
+
+ @Override
+ public void setup(int numChannels) {
+ super.setup(numChannels);
+ this.numStripes = resolveNumStripes();
+ }
+
+ private int resolveNumStripes() {
+ if (writesPerBucket > 0) {
+ return writesPerBucket;
+ }
+ if (writesPerBucket == MATCH_PARALLELISM) {
+ return numChannels;
+ }
+ // AUTO: ceil(numChannels / numBuckets), but never less than one
writer per bucket
+ return Math.max(1, (numChannels + numBuckets - 1) / numBuckets);
+ }
+
+ @Override
+ public int channel(InternalRow record) {
+ extractor.setRecord(record);
+ BinaryRow partition = extractor.partition();
+ int bucket = extractor.bucket();
+ int stripe =
+ hasPrimaryKey
+ ? keyStripe(extractor.trimmedPrimaryKey())
+ : Math.floorMod(nextStripe++, numStripes);
Review Comment:
[P2] Advance append stripes independently for each logical bucket
`nextStripe` is shared by every bucket. Bucket arrival patterns can
therefore correlate with the stripe counter: with two buckets and two stripes,
alternating bucket-0/bucket-1 records always choose stripe 0/stripe 1
respectively. Each bucket uses only one writer although the graph has four
writer subtasks, so the new option does not raise actual write parallelism for
this common input pattern.
I verified the real append bucket extractor's keys (here 1 and 0),
explicitly set `bucket-append-ordered=false`, and ran 2000 alternating records
through an actual Flink sink. All rows are retained, but only one file/writer
per bucket is produced. Changing only the counter to advance per bucket
produces two files/writers per bucket with the same rows. Please use progress
per partition/bucket, or another selection that avoids this correlation, and
assert actual append writer fan-out in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]