[
https://issues.apache.org/jira/browse/HUDI-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1619:
--------------------------------------
Description:
Add more tests to Multitable delta streamer for more sources and mix of diff
sources.
Current tests cover only JsonKafkaSource and ParquetSource
We need more coverage
* AvroSoruce
* AvroKafkaSource
* CsvDFSSource
* HiveIncrPullSource
* HoodieIncrSource
* JsonDFSSource
I am not sure how much is testable. but worth covering as much as possible.
Recently added tests for ParquetDFSSource:
[https://github.com/apache/hudi/pull/2577.] Tried my best to templatize tests.
Would be nice to re-use code as much as possible.
Apart from adding tests for these sources in a homogenous manner, we also need
tests in a heterogenous set up. Table1: source type1. Table2: source type2.
For eg, ParquetDFSSource and CsvDFSSource.
was:
Add more tests to Multitable delta streamer for more sources and mix of diff
sources.
Current tests cover only JsonKafkaSource and ParquetSource
We need more coverage
* AvroSoruce
* AvroKafkaSource
* CsvDFSSource
* HiveIncrPullSource
* HoodieIncrSource
* JsonDFSSource
I am not sure how much is testable. but worth covering as much as possible.
Apart from adding tests for these sources in a homogenous manner, we also need
tests in a heterogenous set up. Table1: source type1. Table2: source type2.
For eg, ParquetDFSSource and CsvDFSSource.
> Add tests to Multitable delta streamer for more sources and mix of diff
> sources
> -------------------------------------------------------------------------------
>
> Key: HUDI-1619
> URL: https://issues.apache.org/jira/browse/HUDI-1619
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: sivabalan narayanan
> Priority: Major
> Labels: newbie
>
> Add more tests to Multitable delta streamer for more sources and mix of diff
> sources.
> Current tests cover only JsonKafkaSource and ParquetSource
> We need more coverage
> * AvroSoruce
> * AvroKafkaSource
> * CsvDFSSource
> * HiveIncrPullSource
> * HoodieIncrSource
> * JsonDFSSource
> I am not sure how much is testable. but worth covering as much as possible.
> Recently added tests for ParquetDFSSource:
> [https://github.com/apache/hudi/pull/2577.] Tried my best to templatize
> tests. Would be nice to re-use code as much as possible.
> Apart from adding tests for these sources in a homogenous manner, we also
> need tests in a heterogenous set up. Table1: source type1. Table2: source
> type2.
> For eg, ParquetDFSSource and CsvDFSSource.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
