----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29063/#review65280 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java <https://reviews.apache.org/r/29063/#comment108358> To be safe, let's do this instead: result = hadoopRDD.mapToPair(new CopyFunction()); sparkPlan.addCachedRDDId(result.id()); result = result.persist(StorageLevel.MEMORY_AND_DISK()); Maybe rdd.persist() returns the same reference, but that's the implementation. This is just to be on the safe side. ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java <https://reviews.apache.org/r/29063/#comment108359> Same as above. - Xuefu Zhang On Dec. 16, 2014, 11:29 p.m., Jimmy Xiang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/29063/ > ----------------------------------------------------------- > > (Updated Dec. 16, 2014, 11:29 p.m.) > > > Review request for hive, chengxiang li and Xuefu Zhang. > > > Bugs: HIVE-8843 > https://issues.apache.org/jira/browse/HIVE-8843 > > > Repository: hive-git > > > Description > ------- > > Release cached RDD after a job is done, if any. > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java > 3102ab2 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java f748ee0 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java > e1946d5 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java 94c6c98 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java 53dd4a0 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java > 438efab > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/LocalSparkJobStatus.java > b8cc6e6 > spark-client/src/main/java/org/apache/hive/spark/client/JobContext.java > 8565bd8 > spark-client/src/main/java/org/apache/hive/spark/client/JobContextImpl.java > 137539b > > spark-client/src/main/java/org/apache/hive/spark/client/MonitorCallback.java > 3693f8f > spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java > 8c07ee1 > > spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java > 7f5ecd0 > > Diff: https://reviews.apache.org/r/29063/diff/ > > > Testing > ------- > > > Thanks, > > Jimmy Xiang > >
