github-actions[bot] commented on code in PR #69571:
URL: https://github.com/apache/doris/pull/69571#discussion_r4367401232
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -2292,6 +2301,108 @@ static FileStoreTable
legacyOrcSplitTable(FileStoreTable table, DataSplit split)
return base;
}
+ /** Positions in the partition row of the partition columns that hold
binary values. */
+ private static int[] binaryPartitionFields(Table table) {
+ RowType partitionType = table.rowType().project(table.partitionKeys());
+ return java.util.stream.IntStream.range(0,
partitionType.getFieldCount()).filter(field -> {
+ DataTypeRoot typeRoot =
partitionType.getTypeAt(field).getTypeRoot();
+ return typeRoot == DataTypeRoot.BINARY || typeRoot ==
DataTypeRoot.VARBINARY;
+ }).toArray();
+ }
+
+ /**
+ * Paimon finds the files of a partition under the directory its values
name. With {@code partition.legacy-name}
+ * = true, the default, it names a partition after the string of each
value, which for a non-NULL binary value
+ * is the Java array object ({@code [B@1b6d3586}) rather than its bytes,
and every resolution of the name
+ * differs. A file written under such a name, or looked up under one, is
never found: Paimon's own read fails,
+ * while the BE skips a missing file of an external table and would
silently drop the partition's rows. A file
+ * with an external path is found by the path its metadata records.
Without a schema dictionary (system tables
+ * other than {@code $ro}, which the JNI reader reads with Paimon's own
reader) the schemas of the files are not
+ * consulted.
+ *
+ * <p>The schema a file refers to does not prove how the file was named,
because dynamic options
+ * ({@code spark.paimon.*}, Flink hints, {@code Table.copy}) keep the
schema ID: a file whose schema uses legacy
+ * names fails only if it is also missing at its planned path. With {@code
index-file-in-data-file-dir} = true, a
+ * deletion vector written without an external data path is looked up
under the directory of its partition too,
+ * whatever the path of its data file, and the BE skips the whole data
file when it cannot open the deletion
+ * vector, so every deletion vector must exist. Each path is checked once
per scan.
+ */
+ private void checkBinaryPartitionFiles(PaimonTableHandle handle, Table
table, DataSplit split,
+ int[] binaryPartitionFields, Supplier<Table> schemaTable,
+ Map<FileStoreTable, Map<Long, Boolean>>
legacyPartitionNameSchemas, Map<String, Boolean> fileExistence) {
+ BinaryRow partition = split.partition();
+ OptionalInt nonNullField = Arrays.stream(binaryPartitionFields)
+ .filter(field -> !partition.isNullAt(field)).findFirst();
+ if (!nonNullField.isPresent()) {
+ return;
+ }
+ int binaryField = nonNullField.getAsInt();
+ FileStoreTable splitTable = schemaTable.get() == null
+ ? null : legacyOrcSplitTable((FileStoreTable)
schemaTable.get(), split);
+ Table optionsTable = splitTable == null ? table : splitTable;
Review Comment:
[P2] Resolve fallback options for audit-log binary partitions
For `$audit_log`, `resolveSchemaDictTable` returns null, so this selects the
system wrapper's options. [Paimon's audit
wrapper](https://github.com/apache/paimon/blob/release-2.1.0/paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java)
exposes the main branch's options while still planning and reading the
fallback branch. For example, write only a NULL binary partition on main with
the default `partition.legacy-name=true`; create a `backup` branch, set its
persisted naming option to `false`, and write a non-NULL binary partition
there. After setting main's `scan.fallback-branch=backup`, `SELECT id FROM
t$audit_log` now throws the legacy-name error for the backup split, even though
its byte-named file exists and the JNI reader selects backup correctly. Resolve
the pinned system source and the split's producing branch independently of
native-schema-dictionary availability, and add this wrapper/fallback case to
the tests. This uses ordinary persisted branch optio
ns, with no dynamic writer override or schema-ID provenance ambiguity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
