-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61716/
-----------------------------------------------------------
Review request for hive, Rui Li and Xuefu Zhang.
Bugs: HIVE-17292
https://issues.apache.org/jira/browse/HIVE-17292
Repository: hive-git
Description
-------
Currently MiniSparkShim will be able to provide only 1 executor, since the
FairScheduler is configured, and the allocation increment is not set so the
default 1024 is used. So only 1 executor could fit into the memory restrictions.
Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the
requested 2 executors.
Changing QTestUtil.setSparkSession, so we will wait until every executor is
available, not only the 1st.
Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided
paralellism in case of local spark.master too.
Regenerating golden files (numReducers, and number of files changed in the
explain plans)
The change contains 2 golden file changes
(spark_dynamic_partition_pruning_mapjoin_only.q.out,
spark_dynamic_partition_pruning.q.out) which are containing other neccessary
changes for a green run, so this patch should be regenerated after their
corresponding jiras are solved (HIVE-17347, HIVE-17346)
Diffs
-----
data/conf/spark/standalone/hive-site.xml 989e65e
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 825f826
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
8224ef9
ql/src/test/results/clientpositive/spark/constprog_semijoin.q.out be33189
ql/src/test/results/clientpositive/spark/index_bitmap3.q.out 31c921c
ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out e3b205c
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out
356f625
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
bf010e7
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out
d9da762
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out
8a794e7
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_mapjoin_only.q.out
564a6a2
ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out
9d162c9
ql/src/test/results/clientpositive/spark/spark_use_ts_stats_for_mapjoin.q.out
f92356e
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
ae6f542
Diff: https://reviews.apache.org/r/61716/diff/1/
Testing
-------
Thanks,
Peter Vary
