Gabriel39 opened a new issue, #70807:
URL: https://github.com/apache/doris/issues/70807

   ### Background
   
   Avro object-container files are used for standalone data exchange and as 
data or delete files in lake tables. A file reader must preserve the 
surrounding table's schema, partition and deletion contracts.
   
   ### Current behavior
   
   Master's TVF format factory and BE scan-support validation reject Avro with 
`Avro file reads are no longer supported`. Hive input-format admission and 
Iceberg native planning also exclude Avro. Paimon already has an SDK path that 
must preserve its merge semantics.
   
   ### Expected behavior
   
   Read standalone Avro through existing TVFs and Avro files in Hive, Iceberg 
and Paimon tables, with correct projections, predicates, counts, NULLs and 
nested/logical values. Iceberg field IDs, schema evolution, snapshots and 
deletes must remain correct.
   
   ### Proposed design
   
   Add Apache Avro C++ 1.12.1 as a controlled static third-party dependency and 
a native FileScannerV2 physical reader. Use the existing filesystem, schema 
mapper, residual filters and table wrappers. Decode requested roots and retain 
absolute file row positions across byte ranges. Admit bulk decoder allocations 
and decompression workspace against the query memory budget and release 
retained scratch on close.
   
   Enable Hive Avro input-format recognition and name mapping. Preserve Avro 
file types in Iceberg planning and use the reader for data and delete files. 
Keep Paimon on its SDK merge path. Restoring the retired JNI scanner would 
recreate the old scan stack; exposing Paimon's raw files would bypass 
primary-key merge/delete semantics.
   
   ### Impact
   
   Users querying standalone Avro files and Hive/Iceberg lake tables on master 
gain native reads. Existing Paimon SDK behavior and other physical file formats 
retain their existing paths. The shared physical/table-reader layers serve both 
deployment modes.
   
   ### Reproduction or evidence
   
   The current format factory rejects `format='avro'`; BE scan validation 
rejects `FORMAT_AVRO`. Acceptance uses independent Java Avro/Iceberg producers 
and the Iceberg Java reader, plus Hive and Paimon writes followed by Doris SQL 
queries.
   
   ### Scope and non-goals
   
   Support Avro object containers with null, deflate, Snappy and Zstandard 
codecs, scalar/logical/nested values, and nullable unions with one non-null 
branch. Include Iceberg equality and position deletes and position-delete 
system tables. Recursive schemas, unions with several non-null alternatives, 
raw Avro messages and Avro output are outside the scope.
   
   ### Acceptance criteria
   
   - TVF values, projections, filters, counts and INSERT SELECT agree with the 
independent fixtures for all supported codecs and empty containers.
   - Hive reads partitions and maps reordered/added table columns against 
embedded file names.
   - Iceberg reads historical schemas, promoted/renamed fields, hidden delete 
keys and Avro data/delete combinations without changing surviving rows.
   - Paimon returns the SDK's final primary-key state after updates, NULLs and 
deletes.
   - Unit tests cover split boundaries, absolute/selected row IDs, 
logical/nested values, malformed input, memory-limit failures and cleanup.
   
   ### Risks and compatibility
   
   Keep `FORMAT_AVRO=12`; add no wire fields or persisted metadata. Reading 
Avro requires upgraded serving FE/BE nodes and the matching third-party 
library. Older nodes fail as unsupported. Avro remains a sequential row format 
without column/page indexes; projection avoids constructing unrequested roots 
but does not add Parquet-style pruning.
   
   ### Related
   
   Implementation PR follows this issue. No prior design agreement is claimed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to