github-actions[bot] commented on code in PR #68786:
URL: https://github.com/apache/doris/pull/68786#discussion_r4236606278
##########
fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcConnectorMetadata.java:
##########
@@ -163,7 +163,14 @@ public Optional<ConnectorTableStatistics>
getTableStatistics(
public Map<String, ConnectorColumnHandle> getColumnHandles(
ConnectorSession session, ConnectorTableHandle handle) {
if (handle instanceof PassthroughQueryTableHandle) {
- return Collections.emptyMap();
+ Map<String, ConnectorColumnHandle> handles = new LinkedHashMap<>();
+ ConnectorTableSchema schema = getColumnsFromQuery(
Review Comment:
[P3] Reuse the query TVF schema when building column handles. TVF binding
already calls `getColumnsFromQuery` for this SQL, but each call here checks out
another FE JDBC connection and prepares the full query just to read its
metadata. `PluginDrivenScanNode` builds handles for the scan and again for scan
properties, so an ordinary TVF can prepare the same SQL three times before BE
reads any rows; EXPLAIN pays the same cost. Carry the bound schema or cache
these handles for the statement.
##########
fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcScanPlanProvider.java:
##########
@@ -67,7 +74,8 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
String querySql;
if (handle instanceof PassthroughQueryTableHandle) {
// Query passthrough from TVF — use the raw SQL directly
- querySql = ((PassthroughQueryTableHandle) handle).getQuery();
+ querySql = new JdbcQueryBuilder(dbType).wrapPassthroughQuery(
+ ((PassthroughQueryTableHandle) handle).getQuery(),
columns, noBackslashEscapes.getAsBoolean());
Review Comment:
[P3] Defer the SQL-mode lookup until a TVF query needs rewriting. For a
MySQL/OceanBase TVF with no TIMESTAMPTZ column, `wrapPassthroughQuery` returns
the original SQL, yet this eager supplier call checks out a FE JDBC connection
and executes `SELECT @@SESSION.sql_mode`. Scan planning and scan-property
serialization each call it, adding two remote round trips to an unchanged query
and making EXPLAIN wait on them. Check whether a timestamp projection is needed
before querying SQL mode, and reuse the result for both calls.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1927,6 +1951,84 @@ private static boolean projectsVariant(
.anyMatch(index -> containsVariant(rowType.getTypeAt(index)));
}
+ private static Set<Integer> scanReadFieldIds(RowType rowType,
List<ConnectorColumnHandle> columns,
+ Optional<ConnectorExpression> filter) {
+ Set<String> names =
columns.stream().filter(PaimonColumnHandle.class::isInstance)
+ .map(PaimonColumnHandle.class::cast)
+ .filter(column -> !column.isMetadataColumn())
+ .map(column -> column.getName().toLowerCase(Locale.ROOT))
+ .collect(Collectors.toSet());
+ filter.ifPresent(expression -> collectFilterColumnNames(expression,
names));
+ return rowType.getFields().stream()
+ .filter(field ->
names.contains(field.name().toLowerCase(Locale.ROOT)))
+ .map(DataField::id).collect(Collectors.toSet());
+ }
+
+ private static void collectFilterColumnNames(ConnectorExpression
expression, Set<String> names) {
+ if (expression instanceof ConnectorColumnRef) {
+ names.add(((ConnectorColumnRef)
expression).getColumnName().toLowerCase(Locale.ROOT));
+ }
+ expression.getChildren().forEach(child ->
collectFilterColumnNames(child, names));
+ }
+
+ static FileStoreTable legacyOrcSplitTable(FileStoreTable table, DataSplit
split) {
+ FileStoreTable base =
PaimonTableDecorators.unwrapToFallbackOrBase(table);
+ if (base instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
base;
+ boolean fallback = ((FallbackReadFileStoreTable.FallbackSplit)
split).isFallback();
+ // Match the SDK reader's branch selection before reading either
options or historical schemas.
+ return PaimonReaderOptions.isWrappedFirst(pair) != fallback ?
pair.wrapped() : pair.other();
+ }
+ return base;
+ }
+
+ static boolean requiresLegacyOrcTimestampReader(Table table,
Optional<List<RawFile>> rawFiles,
+ Set<Integer> readFieldIds, Map<Long, Boolean> schemaTimestamps) {
+ if (readFieldIds.isEmpty() || !rawFiles.isPresent()
+ || rawFiles.get().stream().noneMatch(f ->
f.path().endsWith(".orc"))
+ || !new org.apache.paimon.options.Options(table.options()).get(
+
org.apache.paimon.format.OrcOptions.ORC_TIMESTAMP_LTZ_LEGACY_TYPE)) {
+ return false;
+ }
+ // Only decoded fields require SDK timezone conversion. Unread LTZ
columns must not disable
+ // native splitting or metadata columns; stable field IDs also scope
historical schemas after renames.
+ if (readsTimestampLtz(table.rowType(), readFieldIds)) {
Review Comment:
[P2] Check each ORC file's schema before forcing the legacy LTZ reader. With
`orc.timestamp-ltz.legacy.type=true`, a selected nullable LTZ column added
after an older ORC file makes this current-row-type check return true even
though that file has no LTZ bytes. The split is sent to JNI unnecessarily; for
a primary-key file without `deleteRowCount`, `supportsJniPhysicalMetadata` is
false, so a query also selecting `__paimon_file_path` fails during planning
instead of using the native reader. Check the historical schema IDs first and
route to JNI only for files containing a read LTZ field; cover an added-column
evolution case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
