[
https://issues.apache.org/jira/browse/HUDI-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082773#comment-17082773
]
Yanjia Gary Li edited comment on HUDI-69 at 4/14/20, 10:11 PM:
---------------------------------------------------------------
After a closer look, I think Spark datasource support for realtime table needs:
* Support hadoop.mapreduce.xxx apis. We use hadoop.mapred.RecordReader, but
Spark sql use hadoop.mapreduce.RecordReader. We need to figure how to support
both apis, or upgrade to mapreduce.
* Implement the extension of ParquetInputFormat from Spark or a custom data
source reader to handle merge.
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala]
* Use Datasource V2 to be the default data source.
Please let me know what you guys think.
was (Author: garyli1019):
After a closer look, I think Spark datasource support for realtime table needs:
* Refactoring HoodieRealtimeFormat and (file split, record reader). Decouple
Hudi logic from the MapredParquetInputFormat. I think we can maintain the Hudi
file split and path filtering in a central place, and able to be adopted by
different query engines. With bootstrap support, the file format maintenance
could be more complicated. I think this is very essential.
* Implement the extension of ParquetInputFormat from Spark or a custom data
source reader to handle merge.
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala]
* Use Datasource V2 to be the default data source.
Please let me know what you guys think.
> Support realtime view in Spark datasource #136
> ----------------------------------------------
>
> Key: HUDI-69
> URL: https://issues.apache.org/jira/browse/HUDI-69
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: Spark Integration
> Reporter: Vinoth Chandar
> Assignee: Yanjia Gary Li
> Priority: Major
> Fix For: 0.6.0
>
>
> https://github.com/uber/hudi/issues/136
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
