tchivs opened a new pull request, #8292:
URL: https://github.com/apache/paimon/pull/8292
# Draft PR Plan
## Title
Introduce a file format provider boundary for no-Hadoop engines
## Motivation
Paimon already has a `FileIO` abstraction, but current ORC and Parquet
format construction still requires Hadoop-facing runtime classes in some paths.
This is a problem for engines such as Trino that provide native object-store IO
and native ORC/Parquet implementations, and do not expose Hadoop classes from
the connector runtime.
An earlier bundle-packaging experiment proved the runtime failures can be
hidden by carrying Hadoop classes inside the Paimon bundle, but that is a
packaging fallback rather than the target architecture. Paimon should be able
to support engines with native format implementations without requiring them to
ship Hadoop runtime classes.
## Scope
This first patch adds a small experimental provider boundary:
- `FileFormatProvider` lets an embedding engine supply a `FileFormat` for an
identifier such as `orc` or `parquet`.
- `FileFormat.fromIdentifier` uses a provider only when
`file.format.provider` explicitly selects one.
- `FileFormat.readerFromIdentifier`, `writerFromIdentifier`, and
`validationFromIdentifier` allow engines to select providers per operation.
- `file.format.provider` remains a backward-compatible global fallback.
- `file.format.read-provider`, `file.format.write-provider`, and
`file.format.validation-provider` let an engine avoid sending read paths
through a write-only native provider.
- If no provider is selected, or the selected provider returns
`Optional.empty()` for a format identifier, Paimon falls back to the existing
service-loaded `FileFormatFactory`.
- Duplicate provider identifiers fail with a clear and deterministically
ordered `FactoryException` instead of relying on classpath order.
- Catalog-level `table-runtime.*` options let embedding engines pass
runtime-only table options such as provider selection without persisting them
in table schemas.
- Runtime table options are used when creating tables and when loading
existing tables, so validation and write paths can consistently select the
provider boundary.
- Branch table loading and `FileStoreTable.switchToBranch` preserve catalog
runtime options, so branch operations keep the same provider boundary without
persisting engine-specific options or inheriting arbitrary user dynamic options.
The patch intentionally does not port Trino's native ORC/Parquet writers
into Paimon. It creates the reviewable boundary needed for that follow-up.
## Compatibility
- Existing Hadoop-backed ORC/Parquet implementations remain the default.
- Existing file format identifiers and table options are unchanged.
- Engines that do not explicitly select a `FileFormatProvider` should see no
behavior change.
- The new provider contract uses Paimon-owned types (`FileFormat`,
`FormatContext`, `Options`, `RowType`, `FormatReaderFactory`,
`FormatWriterFactory`) and does not expose Trino classes.
- Operation-specific provider keys let an engine provide only the operations
it owns; for example Trino can provide validation and writes while leaving
reads on Paimon's default reader discovery.
- Runtime options are not serialized into table schema files, so they do not
make table metadata engine-specific.
## Test Evidence
Provider behavior:
- explicit provider selection under a no-Hadoop classloader;
- provider is not used without `file.format.provider`;
- provider can return `Optional.empty()` to fall back to the default
`FileFormatFactory`;
- duplicate provider identifiers fail clearly with deterministically ordered
ambiguous class names.
- read/write/validation provider keys do not affect each other.
- catalog `table-runtime.*` options participate in table validation and
runtime table loading without being persisted in schema files.
- file-format and provider identifiers are normalized with `Locale.ROOT`,
including uppercase
format identifiers and JVM-default-locale edge cases.
- fallback branch switching preserves catalog runtime provider options,
keeps `table-runtime.path`
from replacing the table location or `options().path`, and does not
inherit unrelated
`copy(...)` dynamic options.
Focused provider tests:
```bash
mvn -pl paimon-common,paimon-core,paimon-format -am -Pfast-build
-DfailIfNoTests=false
-Dtest=FileFormatProviderTest,SchemaValidationTest,SchemaManagerTest,FileSystemCatalogTest,FormatProviderNoHadoopTest
test
```
Broader format regression:
```bash
mvn -pl paimon-format -am -Pfast-build -DfailIfNoTests=false
-Dtest=FormatProviderNoHadoopTest,ParquetReadWriteTest,OrcBulkWriterTest test
```
This passed with 45 tests in the latest local refresh.
Bundle packaging:
```bash
mvn -pl paimon-bundle -am -Pfast-build -DskipTests install
```
Fallback branch regression:
```bash
mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false
-Dtest=FallbackReadFileStoreTableTest test
```
License / whitespace:
```bash
git diff --check
git diff --check apache/master
mvn -N org.apache.rat:apache-rat-plugin:0.15:check
```
Current final verification:
```bash
mvn -pl paimon-common,paimon-core -am -Pfast-build -DfailIfNoTests=false
-Dtest=FileFormatProviderTest,FallbackReadFileStoreTableTest#testSwitchToBranch,FileSystemCatalogTest#testCreateTableWithRuntimeCatalogOptions,SchemaValidationTest#testSnapshotSequenceOrderingHonorsDynamicWriteOnlyValue+testDynamicOptionsCanRemoveSchemaOptionsDuringValidation,SchemaManagerTest#testCreateTableWithDynamicOptions+testCommitChangesWithDynamicOptions,FileMetaUtilsTest
test
mvn -pl paimon-format -am -Pfast-build -DfailIfNoTests=false
-Dtest=FormatProviderNoHadoopTest,ParquetReadWriteTest,OrcBulkWriterTest test
mvn -pl paimon-bundle -am -Pfast-build -DskipTests install
git diff --check
git diff --check apache/master
mvn -N org.apache.rat:apache-rat-plugin:0.15:check
```
The current Paimon hardening-focused command passed with 18 tests (11 in
`paimon-common`
and 7 in `paimon-core`), the format regression passed with 45 tests, and the
bundle install
refreshed the local `1.5-SNAPSHOT` artifact consumed by the Trino companion
branch.
Latest pre-commit review refresh added explicit `path` assertions for branch
and fallback
runtime-option isolation. A full PR-diff audit also fixed a missing `@Test`
annotation on
`SchemaValidationTest#testSnapshotSequenceOrderingHonorsDynamicWriteOnlyValue`,
so dynamic
`write-only` validation is now actually executed by Surefire. Current `git
diff --check`,
`git diff --check apache/master`, and RAT all pass.
The bundle install was refreshed after the latest runtime-path hardening.
Target and installed
Maven bundle inspection both report `paimon-bundle-1.5-SNAPSHOT.jar` at
`48,998,103` bytes
(`46.7 MiB`), with `0` ordinary Hadoop entries, `0` Hadoop runtime-prefix
entries, `0` Hadoop
`Configuration.class`, `0` `FileInputFormat.class`, and `1`
`FileFormatProvider.class`.
## Package Footprint
- `paimon-format-1.5-SNAPSHOT.jar` is `26,357,158` bytes (`25.1 MiB`) in the
latest local build.
- `paimon-bundle-1.5-SNAPSHOT.jar` is `48,998,103` bytes (`46.7 MiB`) in the
latest local build.
- The branch does not add embedded Hadoop core/runtime classes.
- Ordinary `org/apache/hadoop/` entries are `0`.
- Hadoop core/runtime classes in Paimon's dependency-isolation namespace are
`0`.
- Hadoop `Configuration.class` entries are `0`.
- Paimon's existing dependency-isolation namespace remains for non-Hadoop
format libraries, and
package checks verify that it contains no Hadoop core/runtime bytecode.
- Existing Hive storage API classes remain under Paimon's internal
dependency namespace; they are not Hadoop core/runtime classes.
## Runtime Option Boundaries
Catalog-level `table-runtime.*` options are runtime-only. Focused FileSystem
catalog coverage now
verifies that provider options are visible on loaded FileStore tables, while
`table-runtime.path`
does not redirect the loaded table away from the catalog location and is not
persisted to schema
files. The validation-provider test records the provider `FormatContext`
path and verifies it sees
the persisted schema path, never the catalog `table-runtime.path` value.
Branch table coverage
verifies the same runtime provider option is preserved when loading a
catalog branch table, while
runtime provider and path options remain absent from the branch schema file.
Fallback branch-switch
coverage additionally verifies that `switchToBranch` recovers catalog
runtime options from
`CatalogEnvironment.catalogContext()` rather than inheriting arbitrary user
dynamic options, and
that the switched table keeps the real table path in both `location()` and
`options()`.
## Companion Validation
A local Trino companion branch demonstrates the provider boundary with Trino
native ORC/Parquet writers:
- `table-runtime.file.format.validation-provider=trino` is set at catalog
creation time for schema validation;
- `file.format.write-provider=trino` is added only on supported write paths;
- normal reads do not use Trino's write-only provider;
- earlier focused connector suite passes (`370` tests);
- runtime dependency tree has no `org.apache.hadoop`, `io.trino.hadoop`, or
`org.apache.parquet:parquet-hadoop` entries;
- packaged Trino Paimon plugin has no standalone Hadoop or `parquet-hadoop`
jars;
- embedded Paimon bundle remains about `47M` with `0` ordinary Hadoop
entries and `0` Hadoop core/runtime classes under Paimon's internal dependency
namespace.
- the latest refreshed Trino plugin zip also reports
`connector_provider_class=1`,
`connector_provider_service_entries=1`, and
`connector_stats_extractor_class=1`
(`io.trino.plugin.paimon.format.TrinoPaimonSimpleStatsExtractor.class`).
- the latest post-hardening refresh passed the SQL provider regression,
runtime dependency tree,
plugin package, and zip inspection against the refreshed local Paimon
bundle.
- the current Trino plugin zip file size is `278,966,092` bytes (`266.0
MiB`); compressed ZIP
entries total `278,908,958` bytes and uncompressed entries total
`301,569,934` bytes. It contains
no standalone Hadoop, Trino Hadoop, or `parquet-hadoop` jars, embeds a
`48,998,103` byte
(`46.7 MiB`) Paimon bundle with no Hadoop runtime classes, and contains
the provider service,
provider class, and stats extractor class inside `trino-paimon-440.jar`.
Filesystem `du` may
report `512M` disk usage because of block allocation; use the ZIP file
size (`278,966,092` bytes)
for PR evidence.
Current companion commands:
```bash
JAVA_HOME=/root/.jdks/temurin-21.0.11
PATH=/root/.jdks/temurin-21.0.11/bin:$PATH ./mvnw -pl plugin/trino-paimon -nsu
-Dtest=TrinoITCase#testSqlCreateInsertReadUsesTrinoProviderForParquetAndOrc test
JAVA_HOME=/root/.jdks/temurin-21.0.11
PATH=/root/.jdks/temurin-21.0.11/bin:$PATH ./mvnw -pl plugin/trino-paimon -nsu
-DskipTests dependency:tree -Dscope=runtime
-Dincludes=org.apache.hadoop,io.trino.hadoop,org.apache.parquet:parquet-hadoop
JAVA_HOME=/root/.jdks/temurin-21.0.11
PATH=/root/.jdks/temurin-21.0.11/bin:$PATH ./mvnw -pl plugin/trino-paimon -nsu
-DskipTests package
```
The SQL provider regression passed with 1 test, the runtime dependency tree
emitted no matching
Hadoop or `parquet-hadoop` dependencies, and the packaged zip inspection
passed the no-Hadoop
artifact checks above.
## Follow-Ups
1. Decide whether `FileFormatProvider` and `table-runtime.*` should stay
`@Experimental` for one release or be promoted into documented extension points.
2. Decide whether `file.format.provider` should remain a runtime-only engine
selector or become a documented catalog/table option.
3. Move more ORC/Parquet implementation details behind no-Hadoop
reader/writer contracts once the boundary is accepted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
