JingsongLi opened a new pull request, #1432: URL: https://github.com/apache/paimon-rust/pull/1432
### Purpose Allow SQL and Python callers to query files without first creating a catalog table, using a documented subset of Databricks `read_files(path, option => value, ...)`. File access requires an explicit application-authorized directory registration. ### Brief change log - Add CSV, Parquet, JSON/NDJSON, text and binaryFile reads, format inference, schemas, named options, directory globs and file metadata. - Match supported Databricks CSV defaults and escape behavior, preserve positional CSV columns, widen inferred scalar/JSON types, and enforce multiline options while streaming CSV, NDJSON and ordinary text. - Capture stores and credentials per SQLContext and query, including Parquet schema/statistics caches, without replacing the shared RuntimeEnv registry. - Resolve local opens and discovery through retained directory capabilities, reject special files without blocking, and prevent symlink replacement from escaping registered directories. - Add Rust and Python behavioral regressions for compatibility, storage isolation and access boundaries; update dependency reports. ### Tests - DataFusion library, all 55 integration-test targets and the example test target executed serially with `RUST_MIN_STACK=16777216 cargo test --locked -p paimon-datafusion --lib`, `--test <name>` and `--example datafusion_query` (the same targets as `--all-targets`), bounding local linker disk usage. - Python extension rebuilt with maturin; `pytest -q bindings/python/tests/test_read_files.py bindings/python/tests/test_sql_stream.py` - `cargo +1.98.0 clippy --locked --no-deps -p paimon-datafusion --all-targets -- -D warnings` - `cargo +1.98.0 fmt --all -- --check` - `cargo deny --locked --all-features check --warn unmaintained advisories licenses` - `python3 scripts/dependencies.py verify` - `python3 scripts/release_licenses.py --check` - Multiple review/fix passes covering Databricks behavior and independent authorization/isolation review. Final local results: ALL 57 TARGETS PASSED: 1272 passed, 0 failed, 0 ignored. Python: 28 passed. ### API and Format Adds the `read_files` SQL table function, Rust `register_read_files_location` / `register_read_files_location_with_file_io`, and Python `register_read_files_location(prefix, storage_options=None)`. Registration is separate from catalog grants; HTTP(S) file URLs are unsupported. No Paimon storage-format changes. ### Documentation SQL guide, Python binding guide and DataFusion README document registration, supported formats/options, metadata and compatibility limits. This is a batch subset: generated `_metadata` is visible in `SELECT *`; partition inference, compressed text, rescued-data/schema-hint modes, streaming ingestion and additional formats are outside this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
