jianguotian opened a new pull request, #81: URL: https://github.com/apache/paimon-mosaic/pull/81
## Purpose Replace the closed #69 with a review-converged, single-commit implementation that splits inferred JSON and CSV conversion into independently reviewable CLI commands, hardens conversion correctness and bounded replay behavior, and defers explicit-schema/Avro semantics plus global input byte/complexity limits to follow-up PRs. ## Changes - keep `convert` for inferred JSON and add `convert-csv` for inferred CSV - preserve the existing `--out` option as an alias of `--output` - support JSON projection and stats, with projection applied before schema inference - support multi-input CSV inference, dialect/header options, all-null fallback, and `--require` - merge headered CSV schemas by name, union shard-specific fields as nullable nulls, and preserve compatible numeric/timestamp semantics - reject lossy bare-integer `Float64` promotion, finite overflow, and explicit offsets in inferred local timestamps - disable CSV backslash escaping by default - reject non-regular inferred CSV paths before opening them; inputs must remain stable during the two-pass conversion - keep projection, required-field, and header mappings indexed for wide schemas - size CSV replay batches from the final union schema width to bound sparse multi-file alignment memory - replace recursive projected-JSON numeric traversal with a linear scan per selected subtree, with a fixed nesting guard ## Scope This PR adds user-visible CLI behavior but does not change the Mosaic file format or core library APIs. It intentionally does **not** add `--schema` or Avro semantics. Targeted replay/resource hardening is included; global input byte/complexity limits and replay-integrity verification remain deferred. ## Tests - `cargo fmt --all -- --check` - `cargo clippy --locked --all-targets --workspace -- -D warnings` - `cargo build --locked --all-targets` - regression RED→GREEN: `convert_csv_unions_missing_fields_by_name` - regression: `csv_replay_batch_size_uses_final_union_width` - regression: `projected_json_parser_does_not_rescan_large_nested_subtrees` - `cargo test --locked -p paimon-mosaic-cli` (25 unit + 82 E2E passed) - `cargo test --locked --workspace` - `cargo deny check licenses` - `python3 tools/dependencies.py check` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
