StephenZou created SPARK-31395: ---------------------------------- Summary: [SPARK][Core]preferred location causing single node be a hot spot Key: SPARK-31395 URL: https://issues.apache.org/jira/browse/SPARK-31395 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.5, 2.3.4 Reporter: StephenZou
my job is run as follows: # build model, model is saved in HDFS named prob1. prob2. probN # then load it to RDD from certain ProbInputformat # do some calculation The driver node which builds the model is scheduled more frequently than other nodes, because the HDFS block is firstly written to itself. The scheduling hot spot is unnecessary, and should better be flattened. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org