JingsongLi opened a new pull request, #1438:
URL: https://github.com/apache/paimon-rust/pull/1438

   ### Purpose
   
   Follow-up to #1432. Generated `_metadata` currently appears in `SELECT *`, 
changing output schemas, `DISTINCT` and imports. Headerless CSV column names 
and negated glob classes also differ from Databricks.
   
   ### Brief change log
   
   - Hide generated metadata from SQLContext wildcard projections while 
preserving explicit selections, source-column precedence, CTEs, views and 
INSERT/CTAS sources. Reuse query-local providers so schema planning and 
execution bind the same source and credentials.
   - Expose file block metadata, with each input file represented as one 
logical block.
   - Use `_c0`, `_c1`, etc. for inferred headerless CSV columns; support empty 
`quote`, explicit `mode => 'FAILFAST'` and `schemaEvolutionMode => 'none'`.
   - Match Hadoop `[^...]` glob negation in paths and filename filters. Reject 
unsupported brace expansion rather than treating braces literally.
   
   ### Tests
   
   - 563 tests passed across the library, read_files suites, SQLContext, query 
authorization and runtime catalog callbacks.
   - Added behavioral regressions for all five formats, wildcard modifiers, 
explicit metadata exports, CTE ordering/recursion, alias column lists, imports, 
source rebinding, DISTINCT, CSV options and glob matching.
   - `cargo +1.98.0 fmt --all -- --check`.
   - Strict package Clippy across all targets.
   
   ### API and Format
   
   Generated metadata is hidden only by the SQLContext SQL compatibility layer; 
explicit source columns named `_metadata` remain visible. No storage format 
changes.
   
   This remains a batch-reading subset. CSV/JSON still default to strict 
parsing; Databricks defaults to PERMISSIVE and rescued-data behavior, which is 
not implemented here. An aliased parenthesized join containing a direct 
read_files source is rejected for wildcard selection; alias the individual 
sources instead.
   
   For explicitly strict queries, use a fixed schema/header plus `mode => 
'FAILFAST', schemaEvolutionMode => 'none'` to disable Databricks rescue 
behavior. See the [official read_files 
documentation](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_files)
 and [format options](https://docs.databricks.com/aws/en/spark/api-options).
   
   ### Documentation
   
   Updated the SQL guide with metadata visibility, CSV/glob behavior, explicit 
strict mode and remaining compatibility limits.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to