This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 125dc0b10d7 [test](paimon) Port the branch-4.x Paimon write tests
master is missing (#68858)
125dc0b10d7 is described below
commit 125dc0b10d7b59228dabdd5f86c004b6c5debc86
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 10 16:27:01 2026 +0800
[test](paimon) Port the branch-4.x Paimon write tests master is missing
(#68858)
### What problem does this PR solve?
Issue Number: None
Related PR: #65868, #66498, #67395
Original author: @suxiaogang223 (the cases follow the tests of
branch-4.1 #65868 and #66498, rewritten for master's Paimon connector;
the commit carries Co-authored-by)
Problem Summary:
**In short.** master #67395 forward-ported the branch-4.1 Paimon write
path, and comparing it with branch-4.2 found the write behavior in place
on master. What did not come along is most of the 4.x tests.
`PaimonConnectorTransaction`, which commits every Paimon write on the
FE, has no test at all, and several other rules are unpinned. This PR
ports those tests. It changes no production code.
**Background.** #67395 carries the branch-4.1 Paimon write PRs #65868,
#66325, #66498, #66810, #66612, #66990, #67023, #67085 and #68167,
rebuilt on the connector write framework of #68320. The BE tests and the
BE-side Java writer tests came over unchanged. The FE tests did not:
they target fe-core classes that master replaced with the Paimon
connector and the generic write framework (`PaimonTransaction`,
`PhysicalPaimonTableSink`, `PaimonMetadataOps`).
**The problem, and what it cost.** Nothing fails on master if any of the
following regresses:
- **FE commit.** A commit whose response is lost is retried once, then
reconciled against the snapshots Paimon published for this commit user
and transaction id. If no snapshot is found, the transaction ends in
OUTCOME_UNKNOWN and its files are never aborted, because the commit may
still land. A failure before the commit started stays abortable. A close
failure after a successful commit does not change the outcome. Payloads
that a BE reports twice are de-duplicated by their exact bytes, not by
their hash. The commit user carries the Doris cluster id. A broken rule
here either loses committed data (aborting files of a commit that
landed) or leaks it (keeping files of a commit that never landed).
- **Single writer.** Some bucket modes must be written by a single
writer: fixed-bucket append tables that are not write-only (their
writers also compact), primary-key fixed-bucket tables, and dynamic or
cross-partition dynamic buckets. Two writers on one bucket corrupt it.
- **MODIFY COLUMN.** Doris shows a Paimon TIMESTAMP(9) as DATETIMEV2(6).
A MODIFY COLUMN that keeps that type (for example, to change the
comment) must not narrow the remote column to TIMESTAMP(6).
- **Short-circuit evaluation.** MERGE INTO labels its branches with
`ShortCircuitIf`. `NestedCaseWhenCondToLiteral` and
`NullSafeEqualToEqual` must keep the function when they rewrite its
condition, or the branch values lose their mandatory short-circuit
evaluation.
- **Catalog flavors.** master writes through HMS and REST Paimon
catalogs, but only filesystem and JDBC catalog writes have end-to-end
cases.
- **MTMV.** The MTMV null-partition case checks that four partition
names are present, but not that there are exactly four.
**How this PR fixes it.** Tests only:
| Test | What it pins |
|---|---|
| `PaimonConnectorTransactionTest` (new, 7) | the six branch-4.x commit
cases above, plus payloads reported twice committing once |
| `PaimonWritePlanProviderTest` (+2) | which bucket modes get a single
writer (GATHER) and which do not |
| `PaimonConnectorMetadataColumnEvolutionTest` (+1) | MODIFY COLUMN
keeps the remote TIMESTAMP(9) |
| `NestedCaseWhenCondToLiteralTest`, `NullSafeEqualToEqualTest` (+1
each) | both rewrites keep a ShortCircuitIf |
| `test_paimon_hms_catalog_write`, `test_paimon_rest_catalog_write` (p2,
new) | append and primary-key writes through HMS and REST catalogs |
| `test_paimon_mtmv` | the MV gets exactly four null-partition variants
|
The transaction test needs no mocking framework, which this module does
not use. Its table is a real on-disk Paimon table, so reconciliation
reads the snapshot that the first attempt really published; only the
committer the table hands out is scripted. The single-writer fixtures
use a DATE bucket key. master cannot route that type natively, so these
fixtures reach the single-writer decision instead of native bucket
routing.
**Results.** Each run below made the listed changes to the production
code locally and ran the affected test classes:
| Changes to production code | Result |
|---|---|
| no reconciliation after two failed attempts; payloads de-duplicated by
hash | `publishedSnapshotReconcilesAsCommitted` and
`commitPayloadDedupUsesExactContent` fail, the other 5 pass |
| OUTCOME_UNKNOWN aborts on rollback; a failed committer open is retried
| `missingSnapshotLeavesOutcomeUnknown` and
`preCommitFailureRemainsAbortable` fail, the other 5 pass |
| a close failure after commit counts as a commit failure |
`closeFailureAfterCommitDoesNotChangeOutcome` fails (the commit is
opened twice), the other 6 pass |
| no single-writer rule for fixed buckets; MODIFY COLUMN always converts
the type | both single-writer cases and
`modifyColumnKeepsRemoteTimestampPrecisionBehindLossyProjection` fail,
the 13 existing cases pass |
| the two rewrites rebuild a plain `If` | both
`testPreserveShortCircuitIf` fail, the 14 existing cases pass |
The comparison also found behavior differences, which are handled in
separate PRs: the Paimon metadata cache, static partition overwrite
values, Java short time zone names, and the JNI read batch size.
**Classes, and how they call each other.**
```
PluginDrivenInsertExecutor
└─ PaimonConnectorTransaction
├─ addCommitData(bytes) BE final report: TPaimonCommitMessage,
kept once per exact payload
├─ commit() newCommit(commitUser).filterAndCommit,
retried once;
│ then
snapshotManager().findSnapshotsForIdentifiers(commitUser, [txnId])
└─ rollback() COMMITTED / COMMITTING / OUTCOME_UNKNOWN:
keep files; PREPARED: abort
PaimonWritePlanProvider.getWriteDistribution
native fixed-bucket route ─> BUCKET_UNAWARE: any ─> single writer: GATHER
─> primary key: hash ─> unpartitioned
```
---
...PaimonConnectorMetadataColumnEvolutionTest.java | 28 ++
.../paimon/PaimonConnectorTransactionTest.java | 342 +++++++++++++++++++++
.../paimon/PaimonWritePlanProviderTest.java | 66 ++++
.../rules/NestedCaseWhenCondToLiteralTest.java | 20 ++
.../expression/rules/NullSafeEqualToEqualTest.java | 21 ++
.../paimon/test_paimon_hms_catalog_write.out | 13 +
.../paimon/test_paimon_rest_catalog_write.out | 13 +
.../paimon/test_paimon_hms_catalog_write.groovy | 107 +++++++
.../paimon/test_paimon_rest_catalog_write.groovy | 101 ++++++
.../suites/mtmv_p0/test_paimon_mtmv.groovy | 1 +
10 files changed, 712 insertions(+)
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataColumnEvolutionTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataColumnEvolutionTest.java
index a835520fe37..c2fa7337e31 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataColumnEvolutionTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataColumnEvolutionTest.java
@@ -28,6 +28,7 @@ import org.apache.paimon.types.DataField;
import org.apache.paimon.types.IntType;
import org.apache.paimon.types.RowType;
import org.apache.paimon.types.SmallIntType;
+import org.apache.paimon.types.TimestampType;
import org.apache.paimon.types.TinyIntType;
import org.apache.paimon.types.VarCharType;
import org.junit.jupiter.api.Assertions;
@@ -140,6 +141,33 @@ public class PaimonConnectorMetadataColumnEvolutionTest {
SchemaChange.UpdateColumnPosition.class,
ops.lastSchemaChanges.get(3));
}
+ @Test
+ public void
modifyColumnKeepsRemoteTimestampPrecisionBehindLossyProjection() {
+ // Doris shows a Paimon TIMESTAMP(9) as DATETIMEV2(6). A MODIFY COLUMN
that keeps the projected
+ // type (here it only changes the comment) must not narrow the remote
column to TIMESTAMP(6).
+ List<DataField> fields = Arrays.asList(
+ new DataField(0, "id", new IntType(false)),
+ new DataField(1, "ts", new TimestampType(9)));
+ FakePaimonTable table = new FakePaimonTable(
+ "t", new RowType(fields), Collections.emptyList(),
Collections.singletonList("id"));
+ RecordingPaimonCatalogOps tsOps = new RecordingPaimonCatalogOps();
+ tsOps.table = table;
+ tsOps.latestSchema = Optional.of(new
PaimonCatalogOps.PaimonSchemaSnapshot(
+ fields, Collections.emptyList(),
Collections.singletonList("id")));
+ PaimonConnectorMetadata tsMetadata = new PaimonConnectorMetadata(
+ tsOps, PaimonCatalogProperties.of(Collections.emptyMap()), new
RecordingConnectorContext());
+ PaimonTableHandle tsHandle = new PaimonTableHandle(
+ "db", "t", Collections.emptyList(),
Collections.singletonList("id"));
+ tsHandle.setPaimonTable(table);
+
+ tsMetadata.modifyColumn(null, tsHandle, new ConnectorColumn(
+ "ts", ConnectorType.of("DATETIMEV2", 6, 0), "event time",
true, null), null);
+
+ Assertions.assertEquals(1, tsOps.lastSchemaChanges.size());
+ Assertions.assertInstanceOf(SchemaChange.UpdateColumnComment.class,
tsOps.lastSchemaChanges.get(0),
+ "the projected type did not change, so the remote TIMESTAMP(9)
must stay");
+ }
+
@Test
public void reorderColumnsEmitsOrderedMoves() {
metadata.reorderColumns(null, handle, Arrays.asList("name", "ID",
"score"));
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorTransactionTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorTransactionTest.java
new file mode 100644
index 00000000000..34b46e29fcf
--- /dev/null
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorTransactionTest.java
@@ -0,0 +1,342 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.thrift.TPaimonCommitMessage;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.consumer.ConsumerManager;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.io.CompactIncrement;
+import org.apache.paimon.io.DataIncrement;
+import org.apache.paimon.io.DataOutputViewStreamWrapper;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaManager;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.DelegatedFileStoreTable;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.FileStoreTableFactory;
+import org.apache.paimon.table.sink.CommitMessage;
+import org.apache.paimon.table.sink.CommitMessageImpl;
+import org.apache.paimon.table.sink.CommitMessageSerializer;
+import org.apache.paimon.table.sink.TableCommitImpl;
+import org.apache.paimon.types.DataField;
+import org.apache.paimon.types.DataTypes;
+import org.apache.thrift.TSerializer;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.ByteArrayOutputStream;
+import java.lang.reflect.Field;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-state handling of {@link PaimonConnectorTransaction}, ported from
the branch-4.x
+ * {@code PaimonTransactionTest}: a commit whose outcome was lost is
reconciled against the snapshots
+ * Paimon actually published, a transaction whose outcome stays unknown is
never aborted, a failure
+ * before the commit started stays abortable, and the commit payloads the BEs
report are
+ * de-duplicated by their exact content.
+ *
+ * <p>No mocking framework (this module has none by convention). The table is
a real on-disk Paimon
+ * table, so reconciliation reads real snapshot files; only the committer it
hands out is scripted,
+ * to fail or to record where a test needs it.
+ */
+public class PaimonConnectorTransactionTest {
+
+ private static final long TRANSACTION_ID = 12345L;
+ private static final DataField ID = new DataField(0, "id",
DataTypes.INT());
+
+ @TempDir
+ Path warehouse;
+
+ private FileStoreTable table;
+
+ @BeforeEach
+ public void createTable() throws Exception {
+ org.apache.paimon.fs.Path location =
+ new org.apache.paimon.fs.Path("file://" + warehouse +
"/db.db/tbl");
+ LocalFileIO fileIO = LocalFileIO.create();
+ new SchemaManager(fileIO, location).createTable(new Schema(
+ Collections.singletonList(ID), Collections.emptyList(),
Collections.emptyList(),
+ Collections.emptyMap(), ""));
+ table = FileStoreTableFactory.create(fileIO, location);
+ }
+
+ @Test
+ public void publishedSnapshotReconcilesAsCommitted() throws Exception {
+ // The first attempt publishes its snapshot but the response is lost,
and the retry fails as
+ // well. The snapshot carrying this commit user and transaction id
proves the commit landed.
+ ScriptedTable target = new ScriptedTable(table);
+ target.failingAttempts = 2;
+ target.publishOnFirstAttempt = true;
+ PaimonConnectorTransaction transaction = boundTransaction(target);
+ transaction.addCommitData(commitFragment(commitPayload(0)));
+
+ transaction.commit();
+
+ Assertions.assertEquals(2, target.commitAttempts);
+ transaction.rollback();
+ Assertions.assertEquals(0, target.aborts, "a committed transaction
must keep its files");
+ }
+
+ @Test
+ public void missingSnapshotLeavesOutcomeUnknown() throws Exception {
+ // Both attempts fail and no snapshot carries this transaction: the
commit may still land
+ // later, so the files must not be aborted.
+ ScriptedTable target = new ScriptedTable(table);
+ target.failingAttempts = 2;
+ PaimonConnectorTransaction transaction = boundTransaction(target);
+ transaction.addCommitData(commitFragment(commitPayload(0)));
+
+ DorisConnectorException failure =
+ Assertions.assertThrows(DorisConnectorException.class,
transaction::commit);
+
+ Assertions.assertEquals("commit attempt 2 failed",
failure.getCause().getMessage());
+ Assertions.assertEquals("commit attempt 1 failed",
+ failure.getCause().getSuppressed()[0].getMessage(),
+ "the first failure must be reported together with the retry
failure");
+ transaction.rollback();
+ Assertions.assertEquals(0, target.aborts,
+ "a transaction whose outcome is unknown must not abort its
files");
+ }
+
+ @Test
+ public void preCommitFailureRemainsAbortable() throws Exception {
+ ScriptedTable target = new ScriptedTable(table);
+ target.newCommitFailures = 1;
+ PaimonConnectorTransaction transaction = boundTransaction(target);
+ transaction.addCommitData(commitFragment(commitPayload(0)));
+
+ Assertions.assertThrows(DorisConnectorException.class,
transaction::commit);
+
+ Assertions.assertEquals(0, target.commitAttempts, "a failed committer
open must not be retried");
+ transaction.rollback();
+ Assertions.assertEquals(1, target.aborts);
+ Assertions.assertEquals(1, target.abortedMessages.size());
+ }
+
+ @Test
+ public void closeFailureAfterCommitDoesNotChangeOutcome() throws Exception
{
+ ScriptedTable target = new ScriptedTable(table);
+ target.closeFailure = new IllegalStateException("committer close
failed");
+ PaimonConnectorTransaction transaction = boundTransaction(target);
+ transaction.addCommitData(commitFragment(commitPayload(0)));
+
+ transaction.commit();
+
+ Assertions.assertEquals(1, target.newCommitCalls, "a committed
transaction must not be retried");
+ Assertions.assertEquals(1, target.commitAttempts);
+ transaction.rollback();
+ Assertions.assertEquals(0, target.aborts);
+ }
+
+ @Test
+ public void commitPayloadDedupUsesExactContent() throws Exception {
+ byte[] first = new byte[] {0, 31};
+ byte[] sameHash = new byte[] {1, 0};
+ Assertions.assertEquals(Arrays.hashCode(first),
Arrays.hashCode(sameHash));
+ PaimonConnectorTransaction transaction = boundTransaction(new
ScriptedTable(table));
+
+ transaction.addCommitData(commitFragment(first));
+ transaction.addCommitData(commitFragment(sameHash));
+ transaction.addCommitData(commitFragment(Arrays.copyOf(first,
first.length)));
+
+ Assertions.assertEquals(2, storedPayloads(transaction),
+ "a re-reported payload is dropped, a different payload with
the same hash is kept");
+ }
+
+ @Test
+ public void reReportedPayloadsCommitOnce() throws Exception {
+ ScriptedTable target = new ScriptedTable(table);
+ PaimonConnectorTransaction transaction = boundTransaction(target);
+ byte[] payload = commitPayload(0);
+
+ transaction.addCommitData(commitFragment(payload));
+ transaction.addCommitData(commitFragment(Arrays.copyOf(payload,
payload.length)));
+ transaction.addCommitData(commitFragment(commitPayload(1)));
+ transaction.commit();
+
+ Assertions.assertEquals(2, target.committedMessages.size());
+ }
+
+ @Test
+ public void commitUserIsNamespacedByDorisCluster() {
+ Assertions.assertEquals(
+ PaimonConnectorTransaction.commitUser(10001, TRANSACTION_ID),
+ PaimonConnectorTransaction.commitUser(10001, TRANSACTION_ID));
+ Assertions.assertNotEquals(
+ PaimonConnectorTransaction.commitUser(10001, TRANSACTION_ID),
+ PaimonConnectorTransaction.commitUser(10002, TRANSACTION_ID));
+ RecordingConnectorContext context = new RecordingConnectorContext();
+ Assertions.assertEquals(
+ PaimonConnectorTransaction.commitUser(context.getClusterId(),
TRANSACTION_ID),
+ new PaimonConnectorTransaction(TRANSACTION_ID,
context).getCommitUser());
+ }
+
+ private static PaimonConnectorTransaction boundTransaction(FileStoreTable
target) {
+ PaimonConnectorTransaction transaction =
+ new PaimonConnectorTransaction(TRANSACTION_ID, new
RecordingConnectorContext());
+ transaction.bind(PaimonWriteBinding.create(
+ new PaimonTableHandle("db", "tbl", Collections.emptyList(),
Collections.emptyList()),
+ target, Collections.emptyMap(), false, Collections.emptyMap(),
Collections.emptySet()));
+ return transaction;
+ }
+
+ /** A DPCM-framed payload with one empty commit message, as the BE writer
reports it. */
+ private static byte[] commitPayload(int bucket) throws Exception {
+ CommitMessage message = new CommitMessageImpl(BinaryRow.EMPTY_ROW,
bucket, 2,
+ DataIncrement.emptyIncrement(),
CompactIncrement.emptyIncrement());
+ CommitMessageSerializer serializer = new CommitMessageSerializer();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ serializer.serializeList(Collections.singletonList(message), new
DataOutputViewStreamWrapper(output));
+ byte[] serialized = output.toByteArray();
+ ByteBuffer framed = ByteBuffer.allocate(12 + serialized.length);
+ framed.put(new byte[] {'D', 'P', 'C', 'M'});
+ framed.putInt(serializer.getVersion());
+ framed.putInt(serialized.length);
+ framed.put(serialized);
+ return framed.array();
+ }
+
+ /** One entry of the connector commit data a BE reports: a binary-protocol
TPaimonCommitMessage. */
+ private static byte[] commitFragment(byte[] payload) throws Exception {
+ return new TSerializer(new TBinaryProtocol.Factory())
+ .serialize(new TPaimonCommitMessage().setPayload(payload));
+ }
+
+ @SuppressWarnings("unchecked")
+ private static int storedPayloads(PaimonConnectorTransaction transaction)
throws ReflectiveOperationException {
+ Field field =
PaimonConnectorTransaction.class.getDeclaredField("commitPayloads");
+ field.setAccessible(true);
+ return ((List<byte[]>) field.get(transaction)).size();
+ }
+
+ /**
+ * The real table, except that the committer it opens follows the test's
script. Reconciliation
+ * still reads the real snapshot manager. The script is transient because
the write binding
+ * serializes the table it is given.
+ */
+ private static final class ScriptedTable extends DelegatedFileStoreTable {
+ transient int newCommitFailures;
+ transient int failingAttempts;
+ transient boolean publishOnFirstAttempt;
+ transient RuntimeException closeFailure;
+ transient int newCommitCalls;
+ transient int commitAttempts;
+ transient int aborts;
+ transient List<CommitMessage> abortedMessages = new ArrayList<>();
+ transient List<CommitMessage> committedMessages = new ArrayList<>();
+
+ ScriptedTable(FileStoreTable real) {
+ super(real);
+ }
+
+ @Override
+ public TableCommitImpl newCommit(String commitUser) {
+ newCommitCalls++;
+ if (newCommitFailures > 0) {
+ newCommitFailures--;
+ throw new IllegalStateException("cannot open the Paimon
committer");
+ }
+ return new ScriptedCommit(this, commitUser);
+ }
+
+ @Override
+ public FileStoreTable copy(Map<String, String> dynamicOptions) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public FileStoreTable copy(TableSchema newTableSchema) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public FileStoreTable copyWithoutTimeTravel(Map<String, String>
dynamicOptions) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public FileStoreTable copyWithLatestSchema() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public FileStoreTable switchToBranch(String branchName) {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+ private static final class ScriptedCommit extends TableCommitImpl {
+ private final ScriptedTable target;
+ private final String commitUser;
+
+ ScriptedCommit(ScriptedTable target, String commitUser) {
+ super(target.wrapped().store().newCommit(commitUser,
target.wrapped()), null, null, null, null,
+ new ConsumerManager(target.fileIO(), target.location()),
+ CoreOptions.ExpireExecutionMode.SYNC, target.name(),
false, 1);
+ this.target = target;
+ this.commitUser = commitUser;
+ }
+
+ @Override
+ public int filterAndCommit(Map<Long, List<CommitMessage>>
commitIdentifiersAndMessages) {
+ int attempt = ++target.commitAttempts;
+ if (attempt > target.failingAttempts) {
+
commitIdentifiersAndMessages.values().forEach(target.committedMessages::addAll);
+ return commitIdentifiersAndMessages.size();
+ }
+ if (attempt == 1 && target.publishOnFirstAttempt) {
+ try (TableCommitImpl real =
target.wrapped().newCommit(commitUser)) {
+ real.ignoreEmptyCommit(false);
+ for (Long identifier :
commitIdentifiersAndMessages.keySet()) {
+ real.commit(identifier, Collections.emptyList());
+ }
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ throw new IllegalStateException("commit attempt " + attempt + "
failed");
+ }
+
+ @Override
+ public void abort(List<CommitMessage> commitMessages) {
+ target.aborts++;
+ target.abortedMessages.addAll(commitMessages);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ if (target.closeFailure != null) {
+ throw target.closeFailure;
+ }
+ }
+ }
+}
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWritePlanProviderTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWritePlanProviderTest.java
index efdb88d6673..8995f4fc5c0 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWritePlanProviderTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWritePlanProviderTest.java
@@ -24,8 +24,15 @@ import
org.apache.doris.connector.spi.handle.ConnectorTableHandle;
import org.apache.doris.connector.spi.handle.ConnectorWriteHandle;
import org.apache.doris.connector.spi.handle.WriteOperation;
import org.apache.doris.connector.spi.write.ConnectorChangelogMode;
+import org.apache.doris.connector.spi.write.ConnectorWriteDistribution;
import org.apache.paimon.CoreOptions;
+import org.apache.paimon.fs.Path;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.BucketMode;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.FileStoreTableFactory;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataTypes;
import org.junit.jupiter.api.Assertions;
@@ -33,6 +40,7 @@ import org.junit.jupiter.api.Test;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@@ -121,6 +129,64 @@ public class PaimonWritePlanProviderTest {
() -> columns.add(columns.get(0)));
}
+ @Test
+ public void fixedBucketAppendTableWithoutWriteOnlyRequiresSingleWriter() {
+ // A fixed-bucket table whose bucket key cannot be routed natively
(DATE) falls back to the
+ // planner distribution. Unless the table is write-only, its writers
also compact, and two
+ // writers compacting the same bucket conflict, so the write must be
gathered to one writer.
+ Assertions.assertEquals(ConnectorWriteDistribution.Mode.GATHER,
writeDistribution(
+ Collections.emptyList(), Collections.emptyList(),
fixedBucketOptions(false),
+ BucketMode.HASH_FIXED));
+
Assertions.assertEquals(ConnectorWriteDistribution.Mode.EXTERNAL_UNPARTITIONED,
writeDistribution(
+ Collections.emptyList(), Collections.emptyList(),
fixedBucketOptions(true),
+ BucketMode.HASH_FIXED));
+ }
+
+ @Test
+ public void primaryKeyAndDynamicBucketTablesRequireSingleWriter() {
+ Assertions.assertEquals(ConnectorWriteDistribution.Mode.GATHER,
writeDistribution(
+ Collections.emptyList(), Collections.singletonList("dt"),
fixedBucketOptions(true),
+ BucketMode.HASH_FIXED));
+ Assertions.assertEquals(ConnectorWriteDistribution.Mode.GATHER,
writeDistribution(
+ Collections.emptyList(), Collections.singletonList("id"),
dynamicBucketOptions(),
+ BucketMode.HASH_DYNAMIC));
+ // The primary key leaves out the partition column, so an upsert can
move a key across
+ // partitions and the bucket of a key comes from the cross-partition
index.
+ Assertions.assertEquals(ConnectorWriteDistribution.Mode.GATHER,
writeDistribution(
+ Collections.singletonList("region"),
Collections.singletonList("id"), dynamicBucketOptions(),
+ BucketMode.KEY_DYNAMIC));
+ Assertions.assertEquals(ConnectorWriteDistribution.Mode.EXECUTION_ANY,
writeDistribution(
+ Collections.emptyList(), Collections.emptyList(),
dynamicBucketOptions(),
+ BucketMode.BUCKET_UNAWARE));
+ }
+
+ private static ConnectorWriteDistribution.Mode
writeDistribution(List<String> partitionKeys,
+ List<String> primaryKeys, Map<String, String> options, BucketMode
expectedBucketMode) {
+ List<DataField> fields = Arrays.asList(
+ new DataField(0, "id", DataTypes.INT().notNull()),
+ new DataField(1, "dt", DataTypes.DATE().notNull()),
+ new DataField(2, "region", DataTypes.STRING().notNull()));
+ FileStoreTable table =
FileStoreTableFactory.create(LocalFileIO.create(),
+ new Path("file:///tmp/paimon-write-distribution/tbl"),
+ new TableSchema(0L, fields, 2, partitionKeys, primaryKeys,
options, ""));
+ Assertions.assertEquals(expectedBucketMode, table.bucketMode(),
"fixture bucket mode");
+ PaimonTableHandle handle = new PaimonTableHandle("db", "tbl",
partitionKeys, primaryKeys);
+ handle.setPaimonTable(table);
+ return new PaimonWritePlanProvider(null, null,
null).getWriteDistribution(null, handle).getMode();
+ }
+
+ private static Map<String, String> fixedBucketOptions(boolean writeOnly) {
+ Map<String, String> options = new HashMap<>();
+ options.put(CoreOptions.BUCKET.key(), "2");
+ options.put(CoreOptions.BUCKET_KEY.key(), "dt");
+ options.put(CoreOptions.WRITE_ONLY.key(), Boolean.toString(writeOnly));
+ return options;
+ }
+
+ private static Map<String, String> dynamicBucketOptions() {
+ return Collections.singletonMap(CoreOptions.BUCKET.key(), "-1");
+ }
+
private static List<ConnectorColumn> columns(String... names) {
return Arrays.stream(names)
.map(name -> new ConnectorColumn(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NestedCaseWhenCondToLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NestedCaseWhenCondToLiteralTest.java
index 04af10abaa9..deaf5c788b6 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NestedCaseWhenCondToLiteralTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NestedCaseWhenCondToLiteralTest.java
@@ -25,6 +25,7 @@ import org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.WhenClause;
import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.ShortCircuitIf;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.types.IntegerType;
@@ -294,6 +295,25 @@ class NestedCaseWhenCondToLiteralTest extends
ExpressionRewriteTestHelper {
Assertions.assertEquals(Maps.newHashMap(), replacer.conditionLiterals);
}
+ @Test
+ void testPreserveShortCircuitIf() {
+ // MERGE INTO labels its branches with ShortCircuitIf; replacing a
nested condition must keep the
+ // function, or the branch values lose their mandatory short-circuit
evaluation.
+ SlotReference slot = new SlotReference("a", IntegerType.INSTANCE);
+ Expression condition = new GreaterThan(slot, IntegerLiteral.of(1));
+ ShortCircuitIf nestedIf = new ShortCircuitIf(
+ condition, IntegerLiteral.of(1), IntegerLiteral.of(0));
+ If outerIf = new If(condition, nestedIf, IntegerLiteral.of(-1));
+
+ Expression rewritten = outerIf.accept(
+ new NestedCaseWhenCondToLiteral.NestedCondReplacer(), null);
+ Expression rewrittenNestedIf = ((If) rewritten).getTrueValue();
+
+ Assertions.assertInstanceOf(ShortCircuitIf.class, rewrittenNestedIf);
+ Assertions.assertEquals(BooleanLiteral.TRUE,
+ ((ShortCircuitIf) rewrittenNestedIf).getCondition());
+ }
+
private static class TestNestedCondReplacer extends
NestedCaseWhenCondToLiteral.NestedCondReplacer {
private final Map<Expression, Map<Expression, BooleanLiteral>>
expressionReplaceMap = Maps.newHashMap();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
index c2896e50fe6..613f17b4d20 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/NullSafeEqualToEqualTest.java
@@ -21,9 +21,11 @@ import
org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
import org.apache.doris.nereids.rules.expression.ExpressionRewriteTestHelper;
import org.apache.doris.nereids.rules.expression.ExpressionRuleExecutor;
import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.ShortCircuitIf;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
@@ -34,6 +36,7 @@ import org.apache.doris.nereids.types.StringType;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
+import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
class NullSafeEqualToEqualTest extends ExpressionRewriteTestHelper {
@@ -177,4 +180,22 @@ class NullSafeEqualToEqualTest extends
ExpressionRewriteTestHelper {
assertRewriteAfterTypeCoercion("(a / b > 1 and c > 1 or (d > 1)) <=>
true",
"(a / b > 1) <=> true and (c > 1 and c is not null) or (d > 1
and d is not null)");
}
+
+ @Test
+ void testPreserveShortCircuitIf() {
+ // Rewriting the condition of a ShortCircuitIf must keep the function,
so its branches still
+ // evaluate lazily.
+ executor = new ExpressionRuleExecutor(ImmutableList.of(
+ bottomUp(NullSafeEqualToEqual.INSTANCE)
+ ));
+ SlotReference left = new SlotReference("a", StringType.INSTANCE,
false);
+ SlotReference right = new SlotReference("b", StringType.INSTANCE,
false);
+ ShortCircuitIf shortCircuitIf = new ShortCircuitIf(
+ new NullSafeEqual(left, right), IntegerLiteral.of(1),
IntegerLiteral.of(0));
+
+ Expression rewritten = executor.rewrite(shortCircuitIf, context);
+
+ Assertions.assertInstanceOf(ShortCircuitIf.class, rewritten);
+ Assertions.assertInstanceOf(EqualTo.class, ((ShortCircuitIf)
rewritten).getCondition());
+ }
}
diff --git
a/regression-test/data/external_table_p2/paimon/test_paimon_hms_catalog_write.out
b/regression-test/data/external_table_p2/paimon/test_paimon_hms_catalog_write.out
new file mode 100644
index 00000000000..0b8ab9001cc
--- /dev/null
+++
b/regression-test/data/external_table_p2/paimon/test_paimon_hms_catalog_write.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !hms_append --
+1 alice 95.5 east
+2 bob 87 west
+3 charlie 92.3 east
+4 diana 88 north
+5 erin 86.5 south
+6 \N \N east
+
+-- !hms_pk --
+1 100 click_updated 99
+1 200 view 2
+2 100 click 3
diff --git
a/regression-test/data/external_table_p2/paimon/test_paimon_rest_catalog_write.out
b/regression-test/data/external_table_p2/paimon/test_paimon_rest_catalog_write.out
new file mode 100644
index 00000000000..b9f0934e935
--- /dev/null
+++
b/regression-test/data/external_table_p2/paimon/test_paimon_rest_catalog_write.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !rest_append --
+1 alice 95.5 east
+2 bob 87 west
+3 charlie 92.3 east
+4 diana 88 north
+5 erin 86.5 south
+6 \N \N east
+
+-- !rest_pk --
+1 100 click_updated 99
+1 200 view 2
+2 100 click 3
diff --git
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog_write.groovy
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog_write.groovy
new file mode 100644
index 00000000000..1a469399023
--- /dev/null
+++
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog_write.groovy
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_paimon_hms_catalog_write", "p2,external") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable paimon test.")
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String hmsPort = context.config.otherConfigs.get("hive3HmsPort")
+ String hdfsPort = context.config.otherConfigs.get("hive3HdfsPort")
+ String catalogName = "test_paimon_hms_catalog_write"
+ String dbName = "hdfs_db"
+ String appendTable = "test_paimon_hms_write_append"
+ String primaryKeyTable = "test_paimon_hms_write_pk"
+
+ sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+ sql """
+ CREATE CATALOG `${catalogName}` PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hmsPort}',
+ 'warehouse' =
'hdfs://${externalEnvIp}:${hdfsPort}/user/hive/warehouse',
+ 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfsPort}'
+ )
+ """
+ sql """SWITCH `${catalogName}`"""
+ sql """USE `${dbName}`"""
+
+ sql """DROP TABLE IF EXISTS `${appendTable}`"""
+ sql """DROP TABLE IF EXISTS `${primaryKeyTable}`"""
+
+ sql """
+ CREATE TABLE `${appendTable}` (
+ id INT NULL,
+ name STRING NULL,
+ score DOUBLE NULL,
+ region STRING NULL
+ ) ENGINE=paimon
+ PARTITION BY (region) ()
+ """
+ sql """
+ INSERT INTO `${appendTable}` VALUES
+ (1, 'alice', 95.5, 'east'),
+ (2, 'bob', 87.0, 'west')
+ """
+ sql """
+ INSERT INTO `${appendTable}` VALUES
+ (3, 'charlie', 92.3, 'east'),
+ (4, 'diana', 88.0, 'north')
+ """
+ sql """
+ INSERT INTO `${appendTable}` (region, score, name, id)
+ VALUES ('south', 86.5, 'erin', 5)
+ """
+ sql """
+ INSERT INTO `${appendTable}` (region, id)
+ VALUES ('east', 6)
+ """
+ order_qt_hms_append """
+ SELECT id, name, score, region
+ FROM `${appendTable}`
+ ORDER BY id
+ """
+
+ sql """
+ CREATE TABLE `${primaryKeyTable}` (
+ user_id INT NOT NULL,
+ event_time BIGINT NOT NULL,
+ event_type STRING NULL,
+ value DOUBLE NULL
+ ) ENGINE=paimon
+ PROPERTIES (
+ 'primary-key' = 'user_id,event_time',
+ 'bucket' = '2',
+ 'bucket-key' = 'user_id'
+ )
+ """
+ sql """
+ INSERT INTO `${primaryKeyTable}` VALUES
+ (1, 100, 'click', 1.0),
+ (1, 200, 'view', 2.0),
+ (2, 100, 'click', 3.0),
+ (1, 100, 'click_updated', 99.0)
+ """
+ order_qt_hms_pk """
+ SELECT user_id, event_time, event_type, value
+ FROM `${primaryKeyTable}`
+ ORDER BY user_id, event_time
+ """
+}
diff --git
a/regression-test/suites/external_table_p2/paimon/test_paimon_rest_catalog_write.groovy
b/regression-test/suites/external_table_p2/paimon/test_paimon_rest_catalog_write.groovy
new file mode 100644
index 00000000000..40f04e5b48a
--- /dev/null
+++
b/regression-test/suites/external_table_p2/paimon/test_paimon_rest_catalog_write.groovy
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_paimon_rest_catalog_write", "p2,external") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable paimon test.")
+ return
+ }
+
+ String catalogProperties =
context.config.otherConfigs.get("paimonDlfRestCatalog")
+ String catalogName = "test_paimon_rest_catalog_write"
+ String dbName = "new_dlf_paimon_db"
+ String appendTable = "test_paimon_rest_write_append"
+ String primaryKeyTable = "test_paimon_rest_write_pk"
+
+ sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+ sql """
+ CREATE CATALOG `${catalogName}` PROPERTIES (
+ ${catalogProperties}
+ )
+ """
+ sql """SWITCH `${catalogName}`"""
+ sql """USE `${dbName}`"""
+
+ sql """DROP TABLE IF EXISTS `${appendTable}`"""
+ sql """DROP TABLE IF EXISTS `${primaryKeyTable}`"""
+
+ sql """
+ CREATE TABLE `${appendTable}` (
+ id INT NULL,
+ name STRING NULL,
+ score DOUBLE NULL,
+ region STRING NULL
+ ) ENGINE=paimon
+ PARTITION BY (region) ()
+ """
+ sql """
+ INSERT INTO `${appendTable}` VALUES
+ (1, 'alice', 95.5, 'east'),
+ (2, 'bob', 87.0, 'west')
+ """
+ sql """
+ INSERT INTO `${appendTable}` VALUES
+ (3, 'charlie', 92.3, 'east'),
+ (4, 'diana', 88.0, 'north')
+ """
+ sql """
+ INSERT INTO `${appendTable}` (region, score, name, id)
+ VALUES ('south', 86.5, 'erin', 5)
+ """
+ sql """
+ INSERT INTO `${appendTable}` (region, id)
+ VALUES ('east', 6)
+ """
+ order_qt_rest_append """
+ SELECT id, name, score, region
+ FROM `${appendTable}`
+ ORDER BY id
+ """
+
+ sql """
+ CREATE TABLE `${primaryKeyTable}` (
+ user_id INT NOT NULL,
+ event_time BIGINT NOT NULL,
+ event_type STRING NULL,
+ value DOUBLE NULL
+ ) ENGINE=paimon
+ PROPERTIES (
+ 'primary-key' = 'user_id,event_time',
+ 'bucket' = '2',
+ 'bucket-key' = 'user_id'
+ )
+ """
+ sql """
+ INSERT INTO `${primaryKeyTable}` VALUES
+ (1, 100, 'click', 1.0),
+ (1, 200, 'view', 2.0),
+ (2, 100, 'click', 3.0),
+ (1, 100, 'click_updated', 99.0)
+ """
+ order_qt_rest_pk """
+ SELECT user_id, event_time, event_type, value
+ FROM `${primaryKeyTable}`
+ ORDER BY user_id, event_time
+ """
+}
diff --git a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy
index 69c4118fcd3..b1a10caa69f 100644
--- a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy
@@ -263,6 +263,7 @@ suite("test_paimon_mtmv",
"p0,external,mtmv,external_docker,external_docker_dori
assertTrue(showNullPartitionsResult.toString().contains("p_NULL"))
assertTrue(showNullPartitionsResult.toString().contains("p_bj"))
assertTrue(showNullPartitionsResult.toString().contains("pn_NULL"))
+ assertEquals(4, showNullPartitionsResult.size())
sql """
REFRESH MATERIALIZED VIEW ${mvName} auto;
"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]