[
https://issues.apache.org/jira/browse/HUDI-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geser Dugarov updated HUDI-8788:
--------------------------------
Description:
Currently, there is a lack of end-to-end tests combining Flink and Spark (Hive,
etc.).
The most interesting scenario is stream writing by Flink and subsequent read by
Spark. Also, we could schedule compaction for MOR table using Flink, and run
compaction itself by Spark.
To implement it, I suppose, we need additional module, where we will place all
needed Hudi, Flink, Spark dependencies, and could implement complex test
scenarios with multiple engines.
was:
Currently, there is a lack of end-to-end tests combining Flink and Spark (Hive,
etc.).
The most interesting scenario is stream writing by Flink and subsequent read by
Spark.
To implement it, I suppose, we need additional module, where we will place all
needed Hudi, Flink, Spark dependencies, and could implement complex test
scenarios with multiple engines.
> e2e test module
> ---------------
>
> Key: HUDI-8788
> URL: https://issues.apache.org/jira/browse/HUDI-8788
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Geser Dugarov
> Assignee: Geser Dugarov
> Priority: Minor
> Fix For: 1.1.0
>
>
> Currently, there is a lack of end-to-end tests combining Flink and Spark
> (Hive, etc.).
> The most interesting scenario is stream writing by Flink and subsequent read
> by Spark. Also, we could schedule compaction for MOR table using Flink, and
> run compaction itself by Spark.
> To implement it, I suppose, we need additional module, where we will place
> all needed Hudi, Flink, Spark dependencies, and could implement complex test
> scenarios with multiple engines.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
