leanken edited a comment on pull request #29431: URL: https://github.com/apache/spark/pull/29431#issuecomment-674030541
Done some code investigation, come out with some information, FYI. 1. the effect of the `java.util.NoSuchElementException` will be SparkUI error like follow All NAAJ (EmptyHashedRelationWithAllNullKeys) Query which be rewritten into LocalRelation will cause following UI Error.  2. offline talked with @cloud-fan , he suggested that we should not keep those useless metrics just for the purpose of avoid `java.util.NoSuchElementException`, instead, we should filter out those outdated metrics. 3. Found some history code that share the consideration of Point No.2 ``` exec.driverAccumUpdates.foreach { case (id, value) => if (metricTypes.contains(id)) { // ************* ignore those outdated metrics by filter using mericsTypes val prev = allMetrics.getOrElse(id, null) val updated = if (prev != null) { // If the driver updates same metrics as tasks and has higher value then remove // that entry from maxMetricsFromAllStage. This would make stringValue function default // to "driver" that would be displayed on UI. if (maxMetricsFromAllStages.contains(id) && value > maxMetricsFromAllStages(id)(0)) { maxMetricsFromAllStages.remove(id) } val _copy = Arrays.copyOf(prev, prev.length + 1) _copy(prev.length) = value _copy } else { Array(value) } allMetrics(id) = updated } } ``` So, I supposed that I should also filter out outdated metrics from ``` val allMetrics = new mutable.HashMap[Long, Array[Long]]() val maxMetricsFromAllStages = new mutable.HashMap[Long, Array[Long]]() ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
