FrankChen021 commented on code in PR #19698:
URL: https://github.com/apache/druid/pull/19698#discussion_r3607330467
##########
server/src/main/java/org/apache/druid/metadata/SQLMetadataConnector.java:
##########
@@ -1038,6 +1045,73 @@ public void createAuditTable()
}
}
+ @Override
+ public void exportTable(
+ final String tableName,
+ final String outputPath
+ )
+ {
+ exportTableWithJdbc(tableName, outputPath);
+ }
+
+ /**
+ * Exports a table to a CSV file using generic JDBC.
+ * Binary columns are hex-encoded and booleans are written as true/false
strings.
+ * Subclasses may override {@link #exportTable} with a database-specific
implementation
+ * while this method remains available for testing or fallback.
+ */
+ protected void exportTableWithJdbc(
+ final String tableName,
+ final String outputPath
+ )
+ {
+ retryWithHandle(
+ (HandleCallback<Void>) handle -> {
+ try (Statement stmt = handle.getConnection().createStatement();
Review Comment:
[P1] Stream PostgreSQL results instead of buffering the table
PostgreSQL JDBC buffers the complete `ResultSet` by default. This plain
`Statement` runs on an auto-commit connection and never sets a fetch size;
PostgreSQL cursor-based fetching requires `autoCommit=false` and a positive
fetch size. Exporting a production segments table can therefore exhaust heap
before rows are written. Run the query in a transaction and apply
`getStreamingFetchSize()`.
##########
services/src/main/java/org/apache/druid/cli/ExportMetadata.java:
##########
@@ -245,12 +245,18 @@ private void exportTable(
} else {
pathFormatString = "%s/%s.csv";
}
+ final String exportTableName = isDerby() ?
StringUtils.toUpperCase(tableName) : tableName;
dbConnector.exportTable(
- StringUtils.toUpperCase(tableName),
+ exportTableName,
Review Comment:
[P1] Preserve current segment columns in the importable CSV
The generic raw export preserves all columns, but `run()` immediately passes
the file through `rewriteSegmentsExport`, which emits only columns 0 through 8.
Current segment tables also contain required `used_status_last_updated` and
additional fingerprint/upgrade columns. A PostgreSQL migration therefore loses
those values, and the documented `COPY` into a freshly created current table
fails because `used_status_last_updated` is `NOT NULL` without a default. The
new test covers only the raw file; the final CSV and import commands need to
preserve the current schema.
##########
server/src/main/java/org/apache/druid/metadata/SQLMetadataConnector.java:
##########
@@ -1038,6 +1045,73 @@ public void createAuditTable()
}
}
+ @Override
+ public void exportTable(
+ final String tableName,
+ final String outputPath
+ )
+ {
+ exportTableWithJdbc(tableName, outputPath);
+ }
+
+ /**
+ * Exports a table to a CSV file using generic JDBC.
+ * Binary columns are hex-encoded and booleans are written as true/false
strings.
+ * Subclasses may override {@link #exportTable} with a database-specific
implementation
+ * while this method remains available for testing or fallback.
+ */
+ protected void exportTableWithJdbc(
+ final String tableName,
+ final String outputPath
+ )
+ {
+ retryWithHandle(
+ (HandleCallback<Void>) handle -> {
+ try (Statement stmt = handle.getConnection().createStatement();
+ ResultSet rs = stmt.executeQuery(StringUtils.format("SELECT
* FROM %s", tableName));
+ FileOutputStream fos = new FileOutputStream(outputPath);
+ OutputStreamWriter writer = new OutputStreamWriter(fos,
StandardCharsets.UTF_8)) {
+ final ResultSetMetaData meta = rs.getMetaData();
+ final int columnCount = meta.getColumnCount();
+ while (rs.next()) {
+ for (int i = 1; i <= columnCount; i++) {
+ if (i > 1) {
+ writer.write(',');
+ }
+ final int colType = meta.getColumnType(i);
+ if (colType == Types.BINARY || colType == Types.VARBINARY
+ || colType == Types.LONGVARBINARY || colType ==
Types.BLOB
+ || (colType == Types.OTHER &&
"bytea".equalsIgnoreCase(meta.getColumnTypeName(i)))) {
+ final byte[] bytes = rs.getBytes(i);
+ if (bytes != null) {
+ writer.write(BaseEncoding.base16().encode(bytes));
+ }
+ } else if (colType == Types.BOOLEAN || colType ==
Types.BIT) {
+ final boolean val = rs.getBoolean(i);
+ if (!rs.wasNull()) {
+ writer.write(String.valueOf(val));
+ }
+ } else {
+ final String val = rs.getString(i);
+ if (val != null) {
+ if (val.contains(",") || val.contains("\"") ||
val.contains("\n") || val.contains("\r")) {
Review Comment:
[P2] Preserve CSV escaping through the rewrite stage
Although the raw writer correctly quotes commas and double quotes, each
`ExportMetadata` rewrite parses those fields and concatenates them into the
final CSV without re-escaping. Legal datasource and metadata identifiers may
contain these characters, so the final importable file gains extra columns or
malformed quoting even though the raw file is valid. Use a CSV writer or shared
escaping routine in the rewrite stage and add an end-to-end test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
