Hi all,
Metadata-only converters such as Apache XTable write Hudi metadata over
Parquet files that another writer (Delta, Iceberg, plain Parquet) produced,
without renaming or copying them. Rewriting another format's data files is
out of scope for such a converter and would duplicate storage on every
sync. I'd like the community's view on whether Hudi should give these files
a supported path through every reader, and which mechanism is the right one
to extend.
Hudi already has part of this. ExternalFilePathUtil marks such a file with
a "_hudiext" suffix in the metadata table's files partition, and
AbstractTableFileSystemView.addFilesToView resolves that marker back to the
file's real partition, so a reader that builds its file view from the
metadata table sees these files.
Readers that list partition directories do not. They parse each base file
name as <fileId>_<writeToken>_<instantTime>.<ext> and never see the marker:
- Trino's Hudi connector: HudiReadOptimizedDirectoryLister builds a
fileListingBasedFileSystemView, on master as well. With Trino 483, a
directory holding one foreign-named Parquet file failed in
AbstractTableFileSystemView.buildFileGroups with "NullPointerException:
element cannot be mapped to a null key", and the query returned 0 rows
instead of the error; trinodb/trino#30864, still open, makes that error
surface.
- hudi-rs: FileLister lists the partition directory and
BaseFile::parse_file_name rejects the name ("Failed to parse file name ...
for base file."). I saw this with 0.4.0; the parsing code is unchanged in
0.5.0, and hudi-rs has no handling for the marker.
As a positive control, the same table with its one data file copied to a
Hudi-convention name, and the write stat updated to match, reads back the
exact row count through hudi-rs. The file name is the whole gap.
Nothing on the RFC list covers it, as far as I can tell. RFC-93 (pluggable
table formats) runs the other way: Hudi writes the data files, and its
Limitations section requires the table to be written only through Hudi.
METADATA_ONLY bootstrap still writes a skeleton file per source file, so it
is a smaller copy rather than an unmodified file registered in place.
Questions for the list:
1. Is the metadata-table route, with the "_hudiext" marker, the intended
contract for external files, so that the fix belongs in readers that list
directories?
2. RFC-105 would bring metadata-table-driven partition listing to the Trino
connector. Would its file view honour the marker too, or does Trino need
more than that?
3. Should hudi-rs follow the same route, building the file view from the
files partition when the metadata table is enabled?
I'm not proposing a design yet; I'd first like to know whether this is a
gap worth closing, and where.
Thanks,
Sylwester
