FrankChen021 commented on code in PR #19698:
URL: https://github.com/apache/druid/pull/19698#discussion_r3712426826
##########
docs/operations/export-metadata.md:
##########
@@ -151,14 +164,18 @@ In the example command above:
After running the tool, the output directory will contain
`<table-name>_raw.csv` and `<table-name>.csv` files.
-The `<table-name>_raw.csv` files are intermediate files used by the tool,
containing the table data as exported by Derby without modification.
+The `<table-name>_raw.csv` files are intermediate files used by the tool,
containing the table data as exported from the source database without
deep-storage rewrites. BLOB columns are hex-encoded and booleans are written as
`true`/`false` strings.
The `<table-name>.csv` files are used for import into another database such as
MySQL and PostgreSQL and have any configured deep storage location rewrites
applied.
Example import commands for Derby, MySQL, and PostgreSQL are shown below.
These example import commands expect `/tmp/csv` and its contents to be
accessible from the server. For other options, such as importing from the
client filesystem, please refer to the database's documentation.
+The segments table is exported in a fixed column order, independent of the
physical column order of the source table: `id`, `dataSource`, `created_date`,
`start`, `end`, `partitioned`, `version`, `used`, `payload`, followed by
whichever of the optional columns `used_status_last_updated`,
`indexing_state_fingerprint`, `upgraded_from_segment_id`, `schema_fingerprint`,
and `num_rows` exist in the source table, in that order. Adjust the segments
column list in the import commands below to contain exactly the columns of the
source table: omit any optional column the source table does not have (segments
tables from older Druid versions may have only the first nine columns), and add
`schema_fingerprint,num_rows` at the end if the source table has them. Apply
the same adjustment to the columns declared with `FORCE_NULL` in the PostgreSQL
command.
Review Comment:
[P1] Provide a column-mapped Derby import for legacy exports
The new guidance says to omit optional columns from the import list, but the
Derby example uses SYSCS_IMPORT_TABLE, which has no column list and imports
every destination column in physical order. A nine-column legacy export
therefore still cannot be loaded into a current 12/14-column Derby segments
table, and the documented adjustment cannot be performed. Use SYSCS_IMPORT_DATA
with an explicit mapping, or provide another working legacy-schema procedure.
##########
server/src/main/java/org/apache/druid/metadata/SQLMetadataConnector.java:
##########
@@ -1038,6 +1045,206 @@ public void createAuditTable()
}
}
+ @Override
+ public void exportTable(
+ final String tableName,
+ final String outputPath
+ )
+ {
+ exportTable(tableName, outputPath, null);
+ }
+
+ /**
+ * Exports a table to a CSV file, emitting the given columns in the given
order.
+ *
+ * @param columns columns to export in the desired order, or null to export
all columns in the
+ * order reported by the database
+ */
+ public void exportTable(
+ final String tableName,
+ final String outputPath,
+ @Nullable final List<String> columns
+ )
+ {
+ exportTableWithJdbc(tableName, outputPath, columns);
+ }
+
+ /**
+ * Returns the columns of the given table, in the order reported by the
database.
+ * Returns an empty list if the table does not exist or the metadata cannot
be read.
+ *
+ * The lookup is scoped to the schema of the current connection, which is
the schema an unqualified
+ * table name resolves to. The table name is folded to the case in which the
database stores
+ * unquoted identifiers, and escaped so that it is matched literally rather
than as a
+ * {@link DatabaseMetaData#getColumns} search pattern.
+ */
+ public List<String> getTableColumns(final String tableName)
+ {
+ return getDBI().withHandle(handle -> {
+ final List<String> columns = new ArrayList<>();
+ try {
+ if (tableExists(handle, tableName)) {
+ final Connection conn = handle.getConnection();
+ final DatabaseMetaData dbMetaData = conn.getMetaData();
+ try (ResultSet rs = dbMetaData.getColumns(
+ null,
+ escapeMetaDataSearchString(dbMetaData, conn.getSchema()),
Review Comment:
[P2] Look up columns in the configured PostgreSQL schema
PostgreSQLConnector.tableExists scopes the relation to
druid.metadata.postgres.dbTableSchema, but getTableColumns scopes
DatabaseMetaData.getColumns to Connection.getSchema(). These can differ, for
example when a role-named schema precedes public in search_path while the Druid
tables remain in public, so tableExists succeeds but column discovery returns
empty and exportSegmentsTable aborts. Use the connector's configured table
schema or resolve the actual schema containing the unqualified relation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
