NetsanetGeb opened a new issue #714: Performance Comparison of HoodieDeltaStreamer and DataSourceAPI URL: https://github.com/apache/incubator-hudi/issues/714 I am trying to ingest data using the DatasourceAPI and HoodieDeltaStreamer. The ingestion latency of using DataSource API with the HoodieSparkSQLWriter is high compared to using delta streamer.Both datasource and deltastreamer use the same APIs. Why is it slow? Are there specific option where we could specify to minimize the ingestion latency. For example: when i run the delta streamer its talking about 1 minute to insert some data. If i use DataSource API with HoodieSparkSQLWriter, its taking 4 minutes. How can we optimize this? I am attaching the spark UI of HoodieDeltaStreamer and DataSourceAPI respectively. Spark UI of HoodieDeltaStreamer Note: The class Main is same as the HoodieDeltaStreamer  SparkUI of DataSourceAPI Note: am using the hoodie delta streamer first to get the dataframe and checkpoint and then later use the hoodie spark sql writer for ingestion. 
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
