Gabriel39 opened a new issue, #70807: URL: https://github.com/apache/doris/issues/70807
### Background Avro object-container files are used for standalone data exchange and as data or delete files in lake tables. A file reader must preserve the surrounding table's schema, partition and deletion contracts. ### Current behavior Master's TVF format factory and BE scan-support validation reject Avro with `Avro file reads are no longer supported`. Hive input-format admission and Iceberg native planning also exclude Avro. Paimon already has an SDK path that must preserve its merge semantics. ### Expected behavior Read standalone Avro through existing TVFs and Avro files in Hive, Iceberg and Paimon tables, with correct projections, predicates, counts, NULLs and nested/logical values. Iceberg field IDs, schema evolution, snapshots and deletes must remain correct. ### Proposed design Add Apache Avro C++ 1.12.1 as a controlled static third-party dependency and a native FileScannerV2 physical reader. Use the existing filesystem, schema mapper, residual filters and table wrappers. Decode requested roots and retain absolute file row positions across byte ranges. Admit bulk decoder allocations and decompression workspace against the query memory budget and release retained scratch on close. Enable Hive Avro input-format recognition and name mapping. Preserve Avro file types in Iceberg planning and use the reader for data and delete files. Keep Paimon on its SDK merge path. Restoring the retired JNI scanner would recreate the old scan stack; exposing Paimon's raw files would bypass primary-key merge/delete semantics. ### Impact Users querying standalone Avro files and Hive/Iceberg lake tables on master gain native reads. Existing Paimon SDK behavior and other physical file formats retain their existing paths. The shared physical/table-reader layers serve both deployment modes. ### Reproduction or evidence The current format factory rejects `format='avro'`; BE scan validation rejects `FORMAT_AVRO`. Acceptance uses independent Java Avro/Iceberg producers and the Iceberg Java reader, plus Hive and Paimon writes followed by Doris SQL queries. ### Scope and non-goals Support Avro object containers with null, deflate, Snappy and Zstandard codecs, scalar/logical/nested values, and nullable unions with one non-null branch. Include Iceberg equality and position deletes and position-delete system tables. Recursive schemas, unions with several non-null alternatives, raw Avro messages and Avro output are outside the scope. ### Acceptance criteria - TVF values, projections, filters, counts and INSERT SELECT agree with the independent fixtures for all supported codecs and empty containers. - Hive reads partitions and maps reordered/added table columns against embedded file names. - Iceberg reads historical schemas, promoted/renamed fields, hidden delete keys and Avro data/delete combinations without changing surviving rows. - Paimon returns the SDK's final primary-key state after updates, NULLs and deletes. - Unit tests cover split boundaries, absolute/selected row IDs, logical/nested values, malformed input, memory-limit failures and cleanup. ### Risks and compatibility Keep `FORMAT_AVRO=12`; add no wire fields or persisted metadata. Reading Avro requires upgraded serving FE/BE nodes and the matching third-party library. Older nodes fail as unsupported. Avro remains a sequential row format without column/page indexes; projection avoids constructing unrequested roots but does not add Parquet-style pruning. ### Related Implementation PR follows this issue. No prior design agreement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
