Reynold Xin created SPARK-20953: ----------------------------------- Summary: Add hash map metrics to aggregate and join Key: SPARK-20953 URL: https://issues.apache.org/jira/browse/SPARK-20953 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 2.2.0 Reporter: Reynold Xin
It would be useful if we can identify hash map collision issues early on. We should add avg hash map probe metric to aggregate operator and hash join operator and report them. If the avg probe is greater than a specific (configurable) threshold, we should log an error at runtime. The primary classes to look at are UnsafeFixedWidthAggregationMap, HashAggregateExec, HashedRelation, HashJoin. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org