> On Aug. 29, 2014, 5:30 p.m., Brock Noland wrote: > > Hi Na, > > > > Thank you very much for the patch! I have one high level question: > > > > It appears we created the union_remove_spark* files because we wanted to > > add an additional property to the union_remove .q file? Meaning what is the > > delta beween union_remove_spark_1.q and union_remove_? > > > > Cheers! > > Na Yang wrote: > Hi Brock, > > That is correct. the union_remove_spark* files include an extra config > property hive.merge.sparkfile comparing to the corresponding union_remove_* > files. Except that extra config property, all other queries in the > union_remove_spark* file are same as the queries in the union_remove_* file. > > The hive.merge.sparkfile value is set according to the hive.merge.mapfile > and hive.merge.mapredfile properity values in the orginal union_remove_* > file. Regarding to the test result, we expect to see the same data are > returned from the union_remove_spark* queries and the corresponding > union_remove_* queries. > > Thanks, > Na > > Brock Noland wrote: > Hi, > > Thank you very much for the information! I think instead of adding the > new union_remove_spark tests we should just add the hive.merge.sparkfile > property to the union_remove q files. The extra property won't impact the > existng tests other than an extra line of outpit. > > If instead we'd like to keep the hive_remove_spark* properties then we'd > need to add a check to QTestUtil that does not run spark files for MR: > > > https://github.com/apache/hive/blob/trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L431 > > as the tests are currently running for both spark and MR. As such, I > think the first solution (just add the property to the existing tests) makes > sense. > > Thoughts? > > Na Yang wrote: > Hi Brock, > > Thank you for your suggestion. I also prefer the first solution. Let me > modify the existing union_remove q files and re-genenrate the .q.out files > for both MR and Spark. > > Thanks, > Na > > Brock Noland wrote: > Awesome, thanks!!
Hi Brock, I have uploaded a new patch according to our previously discussion by modifying the existing union_remove q files. Can you please review the new changes? Thanks & Regards, Na - Na ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review51889 ----------------------------------------------------------- On Aug. 29, 2014, 8:59 p.m., Na Yang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25176/ > ----------------------------------------------------------- > > (Updated Aug. 29, 2014, 8:59 p.m.) > > > Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. > > > Bugs: HIVE-7870 > https://issues.apache.org/jira/browse/HIVE-7870 > > > Repository: hive-git > > > Description > ------- > > HIVE-7870: Insert overwrite table query does not generate correct task plan > [Spark Branch] > > The cause of this problem is during spark/tez task generation, the union file > sink operator are cloned to two new filesink operator. The linkedfilesinkdesc > info for those new filesink operators are missing. In addition, the two new > filesink operators also need to be linked together. > > > Diffs > ----- > > itests/src/test/resources/testconfiguration.properties 88ef4f8 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java > 5ddc16d > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > 379a39c > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 76fc290 > ql/src/test/queries/clientpositive/union_remove_1.q c87b3fe > ql/src/test/queries/clientpositive/union_remove_10.q 6701952 > ql/src/test/queries/clientpositive/union_remove_11.q 4b2fa42 > ql/src/test/queries/clientpositive/union_remove_12.q 69d0d0a > ql/src/test/queries/clientpositive/union_remove_13.q 7605f0e > ql/src/test/queries/clientpositive/union_remove_14.q a4fdfc8 > ql/src/test/queries/clientpositive/union_remove_15.q e3c937b > ql/src/test/queries/clientpositive/union_remove_16.q 537078b > ql/src/test/queries/clientpositive/union_remove_17.q d70f3d3 > ql/src/test/queries/clientpositive/union_remove_18.q 6352bc3 > ql/src/test/queries/clientpositive/union_remove_19.q 8c45953 > ql/src/test/queries/clientpositive/union_remove_2.q 83cd288 > ql/src/test/queries/clientpositive/union_remove_20.q f80f7c1 > ql/src/test/queries/clientpositive/union_remove_21.q 8963c25 > ql/src/test/queries/clientpositive/union_remove_22.q b0c1ccd > ql/src/test/queries/clientpositive/union_remove_23.q a1b989a > ql/src/test/queries/clientpositive/union_remove_24.q ec561e0 > ql/src/test/queries/clientpositive/union_remove_25.q 76c1ff5 > ql/src/test/queries/clientpositive/union_remove_3.q 9617f73 > ql/src/test/queries/clientpositive/union_remove_4.q cae323b > ql/src/test/queries/clientpositive/union_remove_5.q 5df84e1 > ql/src/test/queries/clientpositive/union_remove_6.q bfce26d > ql/src/test/queries/clientpositive/union_remove_7.q 3a95674 > ql/src/test/queries/clientpositive/union_remove_8.q a83a43e > ql/src/test/queries/clientpositive/union_remove_9.q e71f6dd > ql/src/test/results/clientpositive/spark/union10.q.out 20c681e > ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a > ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd > ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 > ql/src/test/results/clientpositive/spark/union29.q.out b9546ef > ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 > ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 > ql/src/test/results/clientpositive/spark/union33.q.out b89757f > ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 > ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 > ql/src/test/results/clientpositive/spark/union_remove_10.q.out 927a15d > ql/src/test/results/clientpositive/spark/union_remove_11.q.out 96651e1 > ql/src/test/results/clientpositive/spark/union_remove_16.q.out 0954ae4 > ql/src/test/results/clientpositive/spark/union_remove_4.q.out cc46dda > ql/src/test/results/clientpositive/spark/union_remove_5.q.out f6cdeb3 > ql/src/test/results/clientpositive/spark/union_remove_9.q.out 1f0260c > > Diff: https://reviews.apache.org/r/25176/diff/ > > > Testing > ------- > > > Thanks, > > Na Yang > >
