----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30264/#review69728 -----------------------------------------------------------
Looks good, glad we can mostly re-use miniYarnCluster. Some minor comments below. I also agree with Xuefu we dont need more golden files by having some output directory. I'm also ok with not running entire set of spark tests with every run, and just running the list we got from minimr.query.files, it is up to you guys. Only note is, once we check this in, committer will also make some edits to the build machine files to run these in proper batches, o/w they will run the entire TestMiniSparkOnYarnCliDriver suite with one batch. shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java <https://reviews.apache.org/r/30264/#comment114485> Need the apache license. shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java <https://reviews.apache.org/r/30264/#comment114486> Need a comment. - Szehon Ho On Jan. 26, 2015, 6:37 a.m., chengxiang li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30264/ > ----------------------------------------------------------- > > (Updated Jan. 26, 2015, 6:37 a.m.) > > > Review request for hive, Szehon Ho and Xuefu Zhang. > > > Bugs: HIVE-9211 > https://issues.apache.org/jira/browse/HIVE-9211 > > > Repository: hive-git > > > Description > ------- > > MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on > miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are > enabled in this unit test except 3 qfile: bucket_num_reducers.q, > bucket_num_reducers2.q, udf_using.q, which is not supported in HoS. > > > Diffs > ----- > > data/conf/spark/hive-site.xml 016f568 > data/conf/spark/standalone/hive-site.xml PRE-CREATION > data/conf/spark/yarn-client/hive-site.xml PRE-CREATION > itests/pom.xml e1e88f6 > itests/qtest-spark/pom.xml d12fad5 > itests/src/test/resources/testconfiguration.properties f583aaf > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java > 41a2ab7 > > ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out > PRE-CREATION > shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java > b17f465 > shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java > a61c3ac > > shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java > PRE-CREATION > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java > 064304c > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java > aea90db > > Diff: https://reviews.apache.org/r/30264/diff/ > > > Testing > ------- > > > Thanks, > > chengxiang li > >
