eldenmoon opened a new pull request, #67993:
URL: https://github.com/apache/doris/pull/67993
### What problem does this PR solve?
Issue Number: None
Related PR: #58711
Problem Summary:
Remove the Doris Native binary file format, which was added in #58711 for
OUTFILE/EXPORT and read back through load and TVF, and later got a format_v2
reader in #65046.
Removed:
- BE: V1 `NativeReader` (`be/src/format/native`), format_v2 `NativeReader`
(`be/src/format_v2/native`), `VNativeTransformer`, the `FORMAT_NATIVE` branches
in `FileScanner`, `FileScannerV2`, `VFileResultWriter` and fetch-table-schema
in `internal_service.cpp`, and `format::FileFormat::NATIVE`.
- Thrift: `TFileFormatType::FORMAT_NATIVE = 18` (the other values keep their
numbers).
- FE: `NativeFileFormatProperties` and the `"native"` format name handling
in `FileFormatConstants`, `FileFormatProperties`, `Util` and
`NereidsLoadScanProvider`.
- Tests: native reader/writer BE unit tests and test data, native cases in
`file_scanner_v2_test` / `table_reader_test` / `NereidsLoadScanProviderTest`,
and the `export_p0/outfile/native` regression suite. The p2 suite
`test_export_variant_10k_columns` now exports and loads back with parquet
instead of native.
Compatibility: during a rolling upgrade, an old FE sending `FORMAT_NATIVE`
to a new BE gets an "unsupported file format" error rather than a crash. Files
previously written in native format can no longer be read by this version.
### Release note
The `native` file format is no longer supported for OUTFILE, EXPORT, load
and TVF. Specifying `format = "native"` now fails with `format:native is not
supported.`, and files previously exported in native format can no longer be
read.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Ran `./build.sh --be --fe` (including FE checkstyle), FE UT
`NereidsLoadScanProviderTest` (8 passed), BE UT `FileScannerV2Test.*` and
`TableReaderTest.*` (143 passed), `build-support/check-build-hygiene.sh` and
`build-support/check-format.sh`. The p2 suite `test_export_variant_10k_columns`
(needs S3) was not rerun with parquet.
- Behavior changed:
- [ ] No.
- [x] Yes. The native file format is removed.
- Does this need documentation?
- [ ] No.
- [x] Yes. https://github.com/apache/doris-website/pull/4139
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
