github-actions[bot] commented on code in PR #68786:
URL: https://github.com/apache/doris/pull/68786#discussion_r4228357016
##########
fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcQueryBuilder.java:
##########
@@ -190,6 +191,161 @@ public String buildQuery(String remoteDbName, String
remoteTableName,
return sql.toString();
}
+ private String timestampProjection(String expression,
org.apache.doris.connector.spi.ConnectorType type,
+ int depth) {
+ if ("TIMESTAMPTZ".equals(type.getTypeName())) {
+ if (dbType == JdbcDbType.MYSQL || dbType == JdbcDbType.OCEANBASE) {
+ // MySQL drivers can apply a cached timezone even in
getString() for binary results.
+ // A server-side text projection preserves the UTC session
fields and microseconds.
+ return "CAST(" + expression + " AS CHAR)";
+ }
+ if (dbType == JdbcDbType.CLICKHOUSE) {
+ return "toUnixTimestamp64Micro(toDateTime64(" + expression +
", 6))";
+ }
+ if (dbType == JdbcDbType.TRINO || dbType == JdbcDbType.PRESTO) {
+ return "(" + expression + " AT TIME ZONE 'UTC')";
+ }
+ }
+ if ("ARRAY".equals(type.getTypeName()) && containsInstant(type)) {
+ String element = "doris_ts_" + depth;
+ String converted = timestampProjection(element,
type.getChildren().get(0), depth + 1);
+ if (dbType == JdbcDbType.CLICKHOUSE) {
+ return "arrayMap(" + element + " -> " + converted + ", " +
expression + ")";
+ }
+ if (dbType == JdbcDbType.TRINO || dbType == JdbcDbType.PRESTO) {
+ return "transform(" + expression + ", " + element + " -> " +
converted + ")";
+ }
+ }
+ return expression;
+ }
+
+ private static boolean
containsInstant(org.apache.doris.connector.spi.ConnectorType type) {
+ return "TIMESTAMPTZ".equals(type.getTypeName()) ||
type.getChildren().stream().anyMatch(
+ JdbcQueryBuilder::containsInstant);
+ }
+
+ public String wrapPassthroughQuery(String query,
List<ConnectorColumnHandle> columns) {
+ if (columns.stream().noneMatch(c -> c instanceof JdbcColumnHandle
+ && containsInstant(((JdbcColumnHandle) c).getType()))
+ || (dbType != JdbcDbType.CLICKHOUSE && dbType !=
JdbcDbType.TRINO && dbType != JdbcDbType.PRESTO
+ && dbType != JdbcDbType.MYSQL && dbType !=
JdbcDbType.OCEANBASE)) {
+ return query;
+ }
+ // Project before driver decoding: a named-zone DST fold has already
lost its offset afterward.
+ StringJoiner projections = new StringJoiner(", ");
+ for (ConnectorColumnHandle column : columns) {
+ JdbcColumnHandle jdbcColumn = (JdbcColumnHandle) column;
+ String name = JdbcIdentifierQuoter.quoteRemoteIdentifier(dbType,
jdbcColumn.getRemoteName());
+ projections.add(timestampProjection(name, jdbcColumn.getType(), 0)
+ " AS " + name);
+ }
+ String inner = query.trim().replaceAll(";+$", "");
Review Comment:
[P2] Handle a semicolon before a trailing TVF comment. With
`enable.mapping.timestamp_tz=true`, a MySQL query TVF such as `SELECT ts FROM
t; -- label` can pass `getColumnsFromQuery` on the original SQL, but this trim
leaves `;` before the comment. The projection then sends `FROM (SELECT ts FROM
t; -- label\n)`, which MySQL cannot parse as a derived table, so the scan fails
after schema discovery. Strip the terminal statement delimiter while preserving
trailing comments before nesting, and cover the combined shape. This is
distinct from the repaired Trino `WITH SESSION` case.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -447,9 +461,24 @@ private int readAndProcessNextBatch() throws IOException {
rows++;
columnValue.setOffsetRow(record);
for (int i = 0; i < fields.length; i++) {
- columnValue.setIdx(
- i, types[i], paimonDataTypeList.get(i),
variantProjections.get(i));
- appendData(i, columnValue);
+ int readIndex = outputToReadIndex[i];
+ if (readIndex < 0) {
+ // Physical positions come from the SDK, so
filtering/deletion vectors cannot
+ // turn a returned-row counter into an incorrect
file row index.
+ if (!(recordIterator instanceof
FileRecordIterator)) {
Review Comment:
[P2] Guard metadata reads for legacy primary-key merge iterators. A Paimon
1.4.2 primary-key ORC file with no historical `deleteRowCount` is marked
raw-convertible, so FE permits a selected legacy LTZ field plus
`__paimon_file_path` or `__paimon_row_index` through the JNI fallback. Paimon's
raw reader rejects that file and its merge reader returns
`ValueContentRowDataRecordIterator`; this check then throws on the first row.
Match Paimon's actual raw-reader eligibility before admitting metadata columns,
and cover an upgraded primary-key file. The existing mixed-projection thread
concerns the append-only raw-file path, which now works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
