Adam Roberts created SPARK-18197: ------------------------------------ Summary: Optimise AppendOnlyMap implementation Key: SPARK-18197 URL: https://issues.apache.org/jira/browse/SPARK-18197 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.0.1, 1.6.2 Reporter: Adam Roberts Priority: Minor
This improvement works by using the cheapest comparison test first and we observed a 1% performance on PageRank (HiBench large) with this change tprof output before the change follows, AppendOnlyMap.changeValue is where the optimisation occurs: {code} PID 256059 22.86 java_1fa29 MOD 81337 7.26 JITCODE SYM 11250 1.00 java/io/ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V_7fe098983af4 SYM 8053 0.72 org/apache/spark/util/collection/AppendOnlyMap.changeValue(Ljava/lang/Object;Lscala/Function2;)Ljava/lang/Object;_7fe098c211e8 SYM 5175 0.46 java/lang/String.equals(Ljava/lang/Object;)Z_7fe0989eb2e8 SYM 3616 0.32 org/apache/spark/util/SizeEstimator$.estimate(Ljava/lang/Object;Ljava/util/IdentityHashMap;)J_7fe098bc35a8 SYM 3235 0.29 org/apache/spark/util/collection/ExternalSorter$$anonfun$4$$anon$6.compare(Ljava/lang/Object;Ljava/lang/Object;)I_7fe098c855a8 SYM 3182 0.28 java/io/ObjectInputStream$BlockDataInputStream.readUTFBody(J)Ljava/lang/String;_7fe098980ec8 SYM 3111 0.28 org/apache/spark/util/SizeEstimator$SearchState.enqueue(Ljava/lang/Object;)V_7fe0989f2920 {code} tprof after the change {code} MOD 56804 5.07 JITCODE SYM 8766 0.78 java/io/ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V_7f0088bb2034 SYM 5746 0.51 java/io/ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;_7f0088944ae8 SYM 3378 0.30 java/io/ObjectInputStream.readObject0(Z)Ljava/lang/Object;_7f0088c5ed00 SYM 3121 0.28 java/io/ObjectInputStream$BlockDataInputStream.readUTFBody(J)Ljava/lang/String;_7f0088c6de08 SYM 2857 0.26 org/apache/spark/storage/BufferReleasingInputStream.read([BII)I_7f0088b3e3a8 SYM 2786 0.25 org/apache/spark/util/collection/AppendOnlyMap.changeValue(Ljava/lang/Object;Lscala/Function2;)Ljava/lang/Object;_7f008899b048 {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org