JingsongLi opened a new pull request, #10423: URL: https://github.com/apache/paimon/pull/10423
### Purpose Register files from an absolute directory in a Paimon partition while their contents are unavailable, for example when the files are archived. Add `CALL sys.import_files(table, location, partition)` for Spark and Flink, backed by a shared metadata-only importer. There is no format argument: the target table's `file.format` selects matching file suffixes. The importer only lists directory entries and file sizes. It records original locations in `DataFileMeta.externalPath`, writes unavailable row counts (`-1`) and no column statistics, and appends all selected files in one commit. It never opens, copies, renames, or restores source files. UUID manifest identities allow the same source basename in different directories. Hidden files and subdirectories are skipped; an empty selection does not create a snapshot. Paths already referenced by the current snapshot are rejected; concurrent imports of the same directory must be avoided. Partitioned tables require the complete typed partition specification; unpartitioned tables omit it. Source files must match the target's format and non-partition columns, and must eventually become readable for queries. Registered external files participate in the normal Paimon file lifecycle, including deletion after compaction or partition removal and snapshot expiration. Supported targets are bucket-unaware append tables without row tracking, data evolution, or Iceberg metadata synchronization, using a filesystem/renaming snapshot commit backend (filesystem/Hive catalogs). Catalog-managed commits, including REST, reject unavailable counts. Depends on #10422 (https://github.com/apache/paimon/pull/10422); its base is that branch so the diff contains only the importer, engine procedures, tests, and procedure documentation. ### Tests Core importer tests use invalid source bytes and a FileIO spy that throws if any archived source file is opened. They verify metadata, typed partitions, filtering, duplicate rejection without partial commit, colliding basenames, mixed native/imported files, empty directories, and unsupported tables. Spark and Flink SQL tests register readable files and validate actual query results; Spark covers Parquet, ORC, Avro, cache refresh, nullable schema evolution, LIMIT, and ANALYZE. Validated with JDK 8, Spark 3, and Flink 1. Final Maven verification runs without `fast-build`, including the foundation regression suites and configured formatting/build checks: ```sh mvn -pl paimon-core,paimon-flink/paimon-flink-common,paimon-spark/paimon-spark-ut -am \ -Pflink1,spark3 -DfailIfNoTests=false \ -DwildcardSuites=org.apache.paimon.spark.sql.UnknownRowCountTest,org.apache.paimon.spark.sql.PushDownAggregatesTest,org.apache.paimon.spark.procedure.ImportFilesProcedureTest \ '-Dtest=ExternalFileImporterTest,UnknownRowCountTest,DataSplitCompatibleTest,FileIndexEvaluatorTest,PartitionStatisticsReporterTest,RemoveUnexistingManifestsTest,SnapshotTest,CatalogSnapshotCommitTest,RenamingSnapshotCommitTest,SplitWeightUtilsTest,FlinkTableSourceStatisticsTest,ImportFilesProcedureITCase,ProcedureTest,AppendOnlySimpleTableTest#testLimitPushDown' test ``` Result: **81 tests passed** (55 core, 9 Flink, 17 Spark). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
