JingsongLi opened a new pull request, #1438: URL: https://github.com/apache/paimon-rust/pull/1438
### Purpose Follow-up to #1432. Generated `_metadata` currently appears in `SELECT *`, changing output schemas, `DISTINCT` and imports. Headerless CSV column names and negated glob classes also differ from Databricks. ### Brief change log - Hide generated metadata from SQLContext wildcard projections while preserving explicit selections, source-column precedence, CTEs, views and INSERT/CTAS sources. Reuse query-local providers so schema planning and execution bind the same source and credentials. - Expose file block metadata, with each input file represented as one logical block. - Use `_c0`, `_c1`, etc. for inferred headerless CSV columns; support empty `quote`, explicit `mode => 'FAILFAST'` and `schemaEvolutionMode => 'none'`. - Match Hadoop `[^...]` glob negation in paths and filename filters. Reject unsupported brace expansion rather than treating braces literally. ### Tests - 563 tests passed across the library, read_files suites, SQLContext, query authorization and runtime catalog callbacks. - Added behavioral regressions for all five formats, wildcard modifiers, explicit metadata exports, CTE ordering/recursion, alias column lists, imports, source rebinding, DISTINCT, CSV options and glob matching. - `cargo +1.98.0 fmt --all -- --check`. - Strict package Clippy across all targets. ### API and Format Generated metadata is hidden only by the SQLContext SQL compatibility layer; explicit source columns named `_metadata` remain visible. No storage format changes. This remains a batch-reading subset. CSV/JSON still default to strict parsing; Databricks defaults to PERMISSIVE and rescued-data behavior, which is not implemented here. An aliased parenthesized join containing a direct read_files source is rejected for wildcard selection; alias the individual sources instead. For explicitly strict queries, use a fixed schema/header plus `mode => 'FAILFAST', schemaEvolutionMode => 'none'` to disable Databricks rescue behavior. See the [official read_files documentation](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_files) and [format options](https://docs.databricks.com/aws/en/spark/api-options). ### Documentation Updated the SQL guide with metadata visibility, CSV/glob behavior, explicit strict mode and remaining compatibility limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
