[
https://issues.apache.org/jira/browse/HIVE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494543#comment-15494543
]
Prasanth Jayachandran commented on HIVE-14767:
----------------------------------------------
Runtimes after the patch
||QFiles||TestMinimrCliDriver elapsed time (Build #1055)||TestMiniLlapCliDriver
elapsed time||
|bucket5.q|257.589|2.286|
|infer_bucket_sort_bucketed_table.q|251.557|2.174|
|bucketizedhiveinputformat.q|249.755|26.301|
|auto_sortmerge_join_16.q|171.094|6.137|
|schemeAuthority.q|170.839|1.345|
|leftsemijoin_mr.q|150.493|1.789|
|join_acid_non_acid.q|143.232|2.511|
|empty_dir_in_table.q|136.463|0.963|
|bucketmapjoin6.q|129.786|1.857|
|reduce_deduplicate.q|116.455|1.162|
|exchgpartition2lel.q|115.678|1.934|
|bucket_many.q|104.316|3.324|
|quotedid_smb.q|95.801|1.162|
|external_table_with_space_in_location_path.q|80.265|1.103|
|schemeAuthority2.q|72.64|0.869|
|disable_merge_for_bucketing.q|69.25|1.618|
|remote_script.q|67.029|7.773|
|groupby2.q|66.72|0.708|
|bucket4.q|66.02|1.616|
|file_with_header_footer.q|42.986|0.817|
|bucket6.q|34.963|0.961|
|non_native_window_udf.q|34.129|0.487|
|join1.q|33.831|1.213|
|bucketmapjoin7.q|33.71|1.85|
|list_bucket_dml_10.q|25.751|1.201|
|insert_dir_distcp.q|25.723|0.414|
|uber_reduce.q|16.481|0.528|
|load_hdfs_file_with_space_in_the_name.q|3.899|3.67|
|table_nonprintable.q|2.097|0.765|
|load_fs2.q|1.947|1.045|
|input16_cc.q|0.916|0.569|
|temp_table_external.q|0.751|0.23|
|import_exported_table.q|0.508|0.366|
Some tests are not migrate to minillap, mainly because it is not a supported
feature.
> Migrate slow MiniMr tests to faster options
> -------------------------------------------
>
> Key: HIVE-14767
> URL: https://issues.apache.org/jira/browse/HIVE-14767
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
> Affects Versions: 2.2.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Attachments: HIVE-14767.1.patch
>
>
> After analyzing the latest test results
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1200/testReport/org.apache.hadoop.hive.cli/
> there are many Minimr tests that are slow individually. There are also some
> tests that does not have to run in minimr, some tests are testing hdfs
> related features that can be migrated to MiniLlap (some are already in
> MiniSpark). Since we have removed hadoop20 now some MiniMr tests can be moved
> to TestCliDriver. We should keep the absolute minimum number of tests in
> minimr and move other slow test to minillap (spark already has most of these
> tests).
> These are the individual test runtimes for minimr tests
> {code}
> QFiles TestMinimrCliDriver elapsed time (Build #1055)
> infer_bucket_sort_reducers_power_two.q 356.211
> bucket5.q 257.589
> infer_bucket_sort_bucketed_table.q 251.557
> bucketizedhiveinputformat.q 249.755
> infer_bucket_sort_map_operators.q 249.484
> index_bitmap_auto.q 212.9
> infer_bucket_sort_dyn_part.q 204.401
> skewjoin_onesideskew.q 198.428
> index_bitmap3.q 194.221
> truncate_column_buckets.q 193.966
> auto_sortmerge_join_16.q 171.094
> schemeAuthority.q 170.839
> leftsemijoin_mr.q 150.493
> join_acid_non_acid.q 143.232
> empty_dir_in_table.q 136.463
> bucketmapjoin6.q 129.786
> reduce_deduplicate.q 116.455
> exchgpartition2lel.q 115.678
> bucket_many.q 104.316
> infer_bucket_sort_merge.q 100.976
> quotedid_smb.q 95.801
> external_table_with_space_in_location_path.q 80.265
> schemeAuthority2.q 72.64
> root_dir_external_table.q 72.171
> parallel_orderby.q 71.662
> disable_merge_for_bucketing.q 69.25
> bucket_num_reducers.q 67.618
> remote_script.q 67.029
> groupby2.q 66.72
> bucket4.q 66.02
> udf_using.q 64.985
> file_with_header_footer.q 42.986
> infer_bucket_sort_num_buckets.q 38.795
> scriptfile1.q 35.317
> bucket6.q 34.963
> non_native_window_udf.q 34.129
> join1.q 33.831
> bucketmapjoin7.q 33.71
> bucket_num_reducers2.q 30.645
> list_bucket_dml_10.q 25.751
> insert_dir_distcp.q 25.723
> uber_reduce.q 16.481
> load_hdfs_file_with_space_in_the_name.q 3.899
> table_nonprintable.q 2.097
> load_fs2.q 1.947
> input16_cc.q 0.916
> temp_table_external.q 0.751
> import_exported_table.q 0.508
> scriptfile1_win.q 0.119
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
