JingsongLi opened a new pull request, #1432:
URL: https://github.com/apache/paimon-rust/pull/1432

   ### Purpose
   
   Allow SQL and Python callers to query files without first creating a catalog 
table, using a documented subset of Databricks `read_files(path, option => 
value, ...)`. File access requires an explicit application-authorized directory 
registration.
   
   ### Brief change log
   
   - Add CSV, Parquet, JSON/NDJSON, text and binaryFile reads, format 
inference, schemas, named options, directory globs and file metadata.
   - Match supported Databricks CSV defaults and escape behavior, preserve 
positional CSV columns, widen inferred scalar/JSON types, and enforce multiline 
options while streaming CSV, NDJSON and ordinary text.
   - Capture stores and credentials per SQLContext and query, including Parquet 
schema/statistics caches, without replacing the shared RuntimeEnv registry.
   - Resolve local opens and discovery through retained directory capabilities, 
reject special files without blocking, and prevent symlink replacement from 
escaping registered directories.
   - Add Rust and Python behavioral regressions for compatibility, storage 
isolation and access boundaries; update dependency reports.
   
   ### Tests
   
   - DataFusion library, all 55 integration-test targets and the example test 
target executed serially with `RUST_MIN_STACK=16777216 cargo test --locked -p 
paimon-datafusion --lib`, `--test <name>` and `--example datafusion_query` (the 
same targets as `--all-targets`), bounding local linker disk usage.
   - Python extension rebuilt with maturin; `pytest -q 
bindings/python/tests/test_read_files.py 
bindings/python/tests/test_sql_stream.py`
   - `cargo +1.98.0 clippy --locked --no-deps -p paimon-datafusion 
--all-targets -- -D warnings`
   - `cargo +1.98.0 fmt --all -- --check`
   - `cargo deny --locked --all-features check --warn unmaintained advisories 
licenses`
   - `python3 scripts/dependencies.py verify`
   - `python3 scripts/release_licenses.py --check`
   - Multiple review/fix passes covering Databricks behavior and independent 
authorization/isolation review.
   
   Final local results: ALL 57 TARGETS PASSED: 1272 passed, 0 failed, 0 
ignored. Python: 28 passed.
   
   ### API and Format
   
   Adds the `read_files` SQL table function, Rust 
`register_read_files_location` / `register_read_files_location_with_file_io`, 
and Python `register_read_files_location(prefix, storage_options=None)`. 
Registration is separate from catalog grants; HTTP(S) file URLs are 
unsupported. No Paimon storage-format changes.
   
   ### Documentation
   
   SQL guide, Python binding guide and DataFusion README document registration, 
supported formats/options, metadata and compatibility limits. This is a batch 
subset: generated `_metadata` is visible in `SELECT *`; partition inference, 
compressed text, rescued-data/schema-hint modes, streaming ingestion and 
additional formats are outside this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to