[ https://issues.apache.org/jira/browse/SPARK-24095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-24095. ---------------------------------- Resolution: Incomplete > Spark Streaming performance drastically drops when when saving dataframes > with withColumn > ----------------------------------------------------------------------------------------- > > Key: SPARK-24095 > URL: https://issues.apache.org/jira/browse/SPARK-24095 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.3.0 > Reporter: brian wang > Priority: Major > Labels: bulk-closed > > We have a Spark Streaming application which is streaming data from Kafka and > ingesting the data in HDFS after a series of transformations. We are using > Spark SQL to do the transformations and storing the data into HDFS at two > stages. The ingestion to Spark which we do at the second stage is drastically > reducing the performance of the application. > There are close to 40 Million transactions per hour in the incoming data. WE > have observed a performance bottleneck in the write to hdfs. > Can you please help us optimize the application performance? > This is a critical issue since it is holding our deployment to production > cluster and we are running behind the schedule in production deployment. > > Answer: First Stage Save > test_Transformed_DOW.cache().withColumn("test_class_map", udf(test_class_map, > StringType())(array(test_class))).write.mode("append").option("header","true").csv("/hive/warehouse/test") > Second Stage Save > test_Data_Final=spark.sql("select test1,test2,test3...... when int(seats)>=2 > then 1 when int(seats) < 2 then 0 end as seats from > test_Data_Unpivoted").write.format("parquet").mode("append").saveAsTable("test_Data_Output") > It is the first save stage which is slowing our spark application's > performance if we enable it. If we disable it, the application seems to catch > up with the incoming data flow. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org