[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581381#comment-16581381 ]
Hive QA commented on HIVE-20321: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935649/HIVE-20321.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 121 failed/errored test(s), 14854 tests executed *Failed tests:* {noformat} TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=50) [fileformat_sequencefile.q,dynamic_semijoin_reduction.q,ppd_union.q,explain_ddl.q,vectorization_pushdown.q,udf_to_boolean.q,union_paren.q,join_cond_pushdown_unqual3.q,exim_hidden_files.q,udf_using.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,input_part2.q,udf_int.q,udf_find_in_set.q,insert_values_nonascii.q,vector_groupby_cube1.q,spark_use_ts_stats_for_mapjoin.q,symlink_text_input_format.q,annotate_stats_groupby2.q,inputddl7.q,multi_insert_gby4.q,erasure_simple.q,materialized_view_authorization_sqlstd.q,stats_part.q,overridden_confs.q,interval_arithmetic.q,udf_stddev_samp.q,groupby_multi_single_reducer3.q,vectorization_8.q] org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=256) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12] (batchId=264) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=70) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete] (batchId=35) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1] (batchId=34) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_3] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_8] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select] (batchId=18) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_date] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[diff_part_input_formats] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[except_all] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window] (batchId=34) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_2] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_expr_1] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_expr_2] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_pos] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partInit] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ptfgroupbyjoin] (batchId=91) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_and] (batchId=2) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_not] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_or] (batchId=40) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[setop_no_distinct] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_12] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=21) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert] (batchId=89) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr] (batchId=8) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_count] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_include_no_sel] (batchId=4) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join_no_keys] (batchId=46) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_parquet_projection] (batchId=45) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries] (batchId=103) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[bucket5] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_simple_select] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_in] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_not_in] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer4] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[count] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window_2] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_llap] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin6] (batchId=179) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin7] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_14] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_18] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_exists] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_remove_26] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_window] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_sort_11] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_reduce] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_nullsafe_join] (batchId=180) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_orc_nested_column_pruning] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_reference_windowed] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning] (batchId=165) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin] (batchId=189) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=187) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez] (batchId=108) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join16] (batchId=127) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_1] (batchId=139) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_2] (batchId=135) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select] (batchId=117) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_in] (batchId=123) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_not_in] (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[count] (batchId=121) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_nullsafe] (batchId=142) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[list_bucket_dml_2] (batchId=114) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[optimize_nullscan] (batchId=146) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark] (batchId=134) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_12] (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_18] (batchId=123) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_exists] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in] (batchId=138) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert] (batchId=147) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin] (batchId=141) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] (batchId=127) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select] (batchId=127) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_view] (batchId=115) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce] (batchId=146) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_parquet_projection] (batchId=129) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testInsertFromUnion (batchId=297) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testInsertToAcidWithUnionRemove (batchId=297) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=297) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testInsertFromUnion (batchId=299) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testInsertToAcidWithUnionRemove (batchId=299) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket (batchId=299) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication (batchId=243) org.apache.hive.jdbc.TestSchedulerQueue.testQueueMappingCheckDisabled (batchId=257) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13239/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13239/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13239/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 121 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12935649 - PreCommit-HIVE-Build > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > --------------------------------------------------------------------------------- > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug > Reporter: Gopal V > Assignee: Matt McCline > Priority: Major > Attachments: HIVE-20321.01.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > ***** 64-bit VM: ********************************************************** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 0 16 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcode N/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValues N/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 > int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 > int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)