Alexey Kudinkin created HUDI-3081:
-------------------------------------
Summary: [UMBRELLA] Revisiting Read Path Infra across Query Engines
Key: HUDI-3081
URL: https://issues.apache.org/jira/browse/HUDI-3081
Project: Apache Hudi
Issue Type: Bug
Reporter: Alexey Kudinkin
Currently, our Read-path infrastructure is mostly disparate for each individual
Query Engine having the same flow replicated multiple times:
* Hive leverages hierarchy based off `InputFormat` class
* Spark leverages hierarchy based off `SnapshotRelation`
This leads to substantial duplication of virtually the same flows being
replicated multiple times and unfortunately now diverging due to out of sync
lifecycle (bug-fixes, etc).
h3. Proposal
*Phase 1: Abstracting Common Functionality*
{_}T-shirt{_}: 1-1.5 weeks
{_}Goal{_}: Abstract following common items to avoid duplication of the complex
sequences across Engines
* Unify Hive’s RecordReaders (`RealtimeCompactedRecordReader`,
{{{}RealtimeUnmergedRecordReader{}}})
** _These Readers should only differ in the way they handle the payload,
everything else should remain constant_
* Abstract w/in common component (name TBD)
** Listing current file-slices at the requested instant (handling the timeline)
** Creating Record Iterator for the provided file-slice
*Phase 2: Revisiting Record Handling*
{_}T-shirt{_}: 1-1.5 weeks
{_}Goal{_}: Avoid tight coupling with particular record representation on the
Read Path (currently Avro) and enable
* Common record handling API for combining records (Merge API)
* Avoiding unnecessary serde by abstracting away standardized Record access
routines (getting key, merging, etc)
** Behind the interface we'd rely on engine-specific representation to carry
the payload (`InternalRow` for Spark, `ArrayWritable` for Hive, etc)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
