[
https://issues.apache.org/jira/browse/HUDI-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980591#comment-16980591
]
Ethan Guo commented on HUDI-76:
-------------------------------
[~vinoth] [~vbalaji]
One thing to clarify, can we assume that the CSV files sit on DFS so we only
consider fetching the files on DFS based on the last modified time?
On a side note, I found that there are some duplicate code inside the
`org.apache.hudi.utilities.sources` package (some code can also be simplified):
(1) *DFSSource (AvroDFSSource, JsonDFSSource, ParquetDFSSource) classes: the
implementation of `fetchNewData()` is the same, while the only difference is
the `fromFiles()` API, which can be abstracted
(2) *KafkaSource (AvroKafkaSource, JsonKafkaSource): similar pattern as
mentioned above, with `fetchNewData()` and `toRDD()`
> CSV Source support for Hudi Delta Streamer
> ------------------------------------------
>
> Key: HUDI-76
> URL: https://issues.apache.org/jira/browse/HUDI-76
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: deltastreamer, Incremental Pull
> Reporter: Balaji Varadarajan
> Assignee: Ethan Guo
> Priority: Minor
>
> DeltaStreamer does not have support to pull CSV data from sources (hdfs log
> files/kafka). THis ticket is to provide support for csv sources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
