[jira] [Commented] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288478#comment-15288478
]
Wei Zheng commented on HIVE-13691:
--
Nit: there're some unnecessary imports in TxnHandler.java and TestTxnCommands2.
Otherwise looks good. +1.
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288176#comment-15288176
]
Hive QA commented on HIVE-13562:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804365/HIVE-13562.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 9005 tests
executed
*Failed tests:*
{noformat}
TestCliDriverMethods - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_16.q-skewjoin.q-vectorization_div0.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin10.q-join_rc.q-skewjoinopt13.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2.q-custom_input_output_format.q-join41.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-join_casesensitive.q-filter_join_breaktask.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7.q-auto_join17.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-skewjoinopt16.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_noskew.q-sample2.q-skewjoinopt10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-runtime_skewjoin_mapjoin_spark.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-union_top_level.q-join11.q-auto_join1.q-and-12-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_columns
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_between_columns
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleTable
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTable
org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
[jira] [Commented] (HIVE-12643) For self describing InputFormat don't replicate schema information in partitions
[ https://issues.apache.org/jira/browse/HIVE-12643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288149#comment-15288149 ] Nita Dembla commented on HIVE-12643: I've tested a slightly modified version of the patch. Original changes to following files were rejected - ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java - ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java And following file needed modifications - ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java > For self describing InputFormat don't replicate schema information in > partitions > > > Key: HIVE-12643 > URL: https://issues.apache.org/jira/browse/HIVE-12643 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12643.1.patch, HIVE-12643.2.patch, > HIVE-12643.3.patch, HIVE-12643.3.patch, HIVE-12643.patch > > > Since self describing Input Formats don't use individual partition schemas > for schema resolution, there is no need to send that info to tasks. > Doing this should cut down plan size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288147#comment-15288147 ] Gopal V commented on HIVE-13029: Should go in this week, I'll rebase the patch and run another test run with a ~2x > RAM cache. > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288099#comment-15288099 ] Lefty Leverenz commented on HIVE-13643: --- Okay, thanks. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288086#comment-15288086 ] Lefty Leverenz commented on HIVE-13449: --- Doc note: This adds *hive.llap.create.token.locally* to HiveConf.java, so it will need to be documented in the wiki for release 2.1.0. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC2.1 label. > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.05.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8458) Potential null dereference in Utilities#clearWork()
[
https://issues.apache.org/jira/browse/HIVE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-8458:
-
Status: Patch Available (was: Open)
> Potential null dereference in Utilities#clearWork()
> ---
>
> Key: HIVE-8458
> URL: https://issues.apache.org/jira/browse/HIVE-8458
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Attachments: HIVE-8458.v2.patch, HIVE-8458_001.patch
>
>
> {code}
> Path mapPath = getPlanPath(conf, MAP_PLAN_NAME);
> Path reducePath = getPlanPath(conf, REDUCE_PLAN_NAME);
> // if the plan path hasn't been initialized just return, nothing to clean.
> if (mapPath == null && reducePath == null) {
> return;
> }
> try {
> FileSystem fs = mapPath.getFileSystem(conf);
> {code}
> If mapPath is null but reducePath is not null, getFileSystem() call would
> produce NPE
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8458) Potential null dereference in Utilities#clearWork()
[
https://issues.apache.org/jira/browse/HIVE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-8458:
-
Attachment: HIVE-8458.v2.patch
> Potential null dereference in Utilities#clearWork()
> ---
>
> Key: HIVE-8458
> URL: https://issues.apache.org/jira/browse/HIVE-8458
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Attachments: HIVE-8458.v2.patch, HIVE-8458_001.patch
>
>
> {code}
> Path mapPath = getPlanPath(conf, MAP_PLAN_NAME);
> Path reducePath = getPlanPath(conf, REDUCE_PLAN_NAME);
> // if the plan path hasn't been initialized just return, nothing to clean.
> if (mapPath == null && reducePath == null) {
> return;
> }
> try {
> FileSystem fs = mapPath.getFileSystem(conf);
> {code}
> If mapPath is null but reducePath is not null, getFileSystem() call would
> produce NPE
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13449: -- Labels: TODOC2.1 (was: ) > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.05.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9756) LLAP: use log4j 2 for llap (log to separate files, etc.)
[ https://issues.apache.org/jira/browse/HIVE-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9756: --- Attachment: HIVE-9756.4.patch The same patch > LLAP: use log4j 2 for llap (log to separate files, etc.) > > > Key: HIVE-9756 > URL: https://issues.apache.org/jira/browse/HIVE-9756 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gunther Hagleitner >Assignee: Sergey Shelukhin > Attachments: HIVE-9756.1.patch, HIVE-9756.2.patch, HIVE-9756.3.patch, > HIVE-9756.4.patch > > > For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get > throughput friendly logging. > http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9756) LLAP: use log4j 2 for llap (log to separate files, etc.)
[ https://issues.apache.org/jira/browse/HIVE-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9756: --- Assignee: Prasanth Jayachandran (was: Sergey Shelukhin) > LLAP: use log4j 2 for llap (log to separate files, etc.) > > > Key: HIVE-9756 > URL: https://issues.apache.org/jira/browse/HIVE-9756 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran > Attachments: HIVE-9756.1.patch, HIVE-9756.2.patch, HIVE-9756.3.patch, > HIVE-9756.4.patch > > > For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get > throughput friendly logging. > http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9756) LLAP: use log4j 2 for llap (log to separate files, etc.)
[ https://issues.apache.org/jira/browse/HIVE-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-9756: -- Assignee: Sergey Shelukhin (was: Prasanth Jayachandran) > LLAP: use log4j 2 for llap (log to separate files, etc.) > > > Key: HIVE-9756 > URL: https://issues.apache.org/jira/browse/HIVE-9756 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gunther Hagleitner >Assignee: Sergey Shelukhin > Attachments: HIVE-9756.1.patch, HIVE-9756.2.patch, HIVE-9756.3.patch > > > For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get > throughput friendly logging. > http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288047#comment-15288047 ] Sergey Shelukhin commented on HIVE-13029: - [~gopalv] ping? ;) > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12643) For self describing InputFormat don't replicate schema information in partitions
[ https://issues.apache.org/jira/browse/HIVE-12643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288034#comment-15288034 ] Ashutosh Chauhan commented on HIVE-12643: - [~mmccline] Can you also please take a look at this one? [~ndembla] tried it out and reported this gives us a speed-up in query compile time. > For self describing InputFormat don't replicate schema information in > partitions > > > Key: HIVE-12643 > URL: https://issues.apache.org/jira/browse/HIVE-12643 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12643.1.patch, HIVE-12643.2.patch, > HIVE-12643.3.patch, HIVE-12643.3.patch, HIVE-12643.patch > > > Since self describing Input Formats don't use individual partition schemas > for schema resolution, there is no need to send that info to tasks. > Doing this should cut down plan size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13736) View's input/output formats are TEXT by default
[
https://issues.apache.org/jira/browse/HIVE-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287997#comment-15287997
]
Hive QA commented on HIVE-13736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804243/HIVE-13736.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 10007 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkClient - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin_negative3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_escape_distributeby1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mergejoins_mixed
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_metadata_only_queries_with_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_left_outer_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_nested_mapjoin
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.05.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[ https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287965#comment-15287965 ] Ashutosh Chauhan commented on HIVE-13773: - [~pxiong] Can you describe the bug? > Stats state is not captured correctly in dynpart_sort_optimization_acid.q > - > > Key: HIVE-13773 > URL: https://issues.apache.org/jira/browse/HIVE-13773 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13773.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287961#comment-15287961 ] Ashutosh Chauhan commented on HIVE-13778: - How *many* files are needed to trigger this? cc: [~rajesh.balamohan] > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Attachment: HIVE-13443.WIP.nogen.patch > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.nogen.patch, HIVE-13443.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287935#comment-15287935 ] Sergey Shelukhin commented on HIVE-13443: - [~sseth] fyi. It's not ready for review but may be useful to skim > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Status: Patch Available (was: Open) > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Attachment: HIVE-13443.WIP.patch The WIP patch on top of million other patches. I did go the route of including the event with SubmitWork. That makes it much cleaner to verify, etc. It's nigh impossible to add something to TezEvent (comments in the patch) and adding extra layer of wrapping for splits and then extra wrapper input format is more complex, and probably a waste of effort anyway - we should rather have a cleaner API w/o depending on Tez. I am preserving the heartbeat-time logic for now - this event is given to the task only during the first heartbeat. > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287904#comment-15287904 ] Siddharth Seth commented on HIVE-13643: --- This doesn't require additional documentation, mostly bug fixes and minor enhancements. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13698) refactor TezSession-s to support shared state
[ https://issues.apache.org/jira/browse/HIVE-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13698: -- Target Version/s: 2.1.0 > refactor TezSession-s to support shared state > - > > Key: HIVE-13698 > URL: https://issues.apache.org/jira/browse/HIVE-13698 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > It would be nice to have shared state for the sessions, to reuse things like > clients to get tokens, etc. > Since the shared state is only useful in multi-session scenario, i.e. HS2, > the existing session creation paths thru the session pool can just be made > cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[ https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287806#comment-15287806 ] Pengcheng Xiong commented on HIVE-13773: the patch partially reverts "HIVE-6455: Scalable dynamic partitioning and bucketing optimization" by [~prasanth_j] and [~vikram.dixit]. Could you guys take a look? Thanks. > Stats state is not captured correctly in dynpart_sort_optimization_acid.q > - > > Key: HIVE-13773 > URL: https://issues.apache.org/jira/browse/HIVE-13773 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13773.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[ https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13773: --- Status: Patch Available (was: Open) > Stats state is not captured correctly in dynpart_sort_optimization_acid.q > - > > Key: HIVE-13773 > URL: https://issues.apache.org/jira/browse/HIVE-13773 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13773.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[ https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13773: --- Attachment: HIVE-13773.01.patch > Stats state is not captured correctly in dynpart_sort_optimization_acid.q > - > > Key: HIVE-13773 > URL: https://issues.apache.org/jira/browse/HIVE-13773 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13773.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287739#comment-15287739 ] Siddharth Seth commented on HIVE-13449: --- +1. > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.05.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil updated HIVE-13778: - Description: I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A that has many files, the files never get deleted. However, the Hive metastore logs do say that the path was deleted: "Not moving [path] to trash" "Deleted the diretory [path]" I initially thought that this was due to the eventually consistent nature of S3 for deletes, however, a week later, the files still exist. was: I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A that has many files, the files never get deleted. However, the Hive metastore logs do say that the path was deleted: "Not moving [path] to trash" "Deleted the diretory [path]" I initially thought that this was due to the eventually consistent nature of S3, however, a week later, the files still exist. > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.05.patch Minor change - some reuse and stopping the threads. > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.04.patch, > HIVE-13449.05.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13622: -- Status: Open (was: Patch Available) > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, > HIVE-13622.4.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287680#comment-15287680
]
Eugene Koifman commented on HIVE-13691:
---
Test results.
{noformat}
Test Name Duration Age
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
3.9 sec 1
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL 40 sec 1
org.apache.hive.spark.client.TestSparkClient.testJobSubmission 2.1 sec 1
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
5 sec 1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload 3.8 sec
2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
5 min 20 sec3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
51 sec 8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.1 sec 8
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 40 sec9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
1 min 21 sec9
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.7 sec 9
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.45 sec9
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 38 sec9
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
36 ms 29
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
33 sec 29
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
4 sec 42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate17 sec
61
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
12 sec 61
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 39 sec
61
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 31 sec61
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
7.4 sec 61
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
51 sec 61
{noformat}
Failures not related to the change.
[~wzheng] could you review please
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Status: Open (was: Patch Available)
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13768) Distinct functions don't work properly after hive.map.groupby.sorted is default to true
[
https://issues.apache.org/jira/browse/HIVE-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu resolved HIVE-13768.
-
Resolution: Not A Problem
Assignee: (was: Aihua Xu)
Seems I misunderstood the problem. It assumes that table is sorted by group by
key. So it works as expected.
> Distinct functions don't work properly after hive.map.groupby.sorted is
> default to true
> ---
>
> Key: HIVE-13768
> URL: https://issues.apache.org/jira/browse/HIVE-13768
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>
> HIVE-12325 changes hive.map.groupby.sorted default value to true. The
> following {{select count(distinct) from t1;}} will return incorrect result
> now. In the older version, if you set hive.map.groupby.sorted to true, seems
> it will also return incorrect result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13608:
-
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Metastore
>Affects Versions: 2.0.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13608.1.patch, HIVE-13608.2.patch,
> HIVE-13608.3.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287476#comment-15287476
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13608:
--
Tried all the failures locally, was not able to reproduce any.
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
> Components: Diagnosability, Metastore
>Affects Versions: 2.0.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch, HIVE-13608.2.patch,
> HIVE-13608.3.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Attachment: HIVE-13691.2.patch
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13622) WriteSet tracking optimizations
[
https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287249#comment-15287249
]
Hive QA commented on HIVE-13622:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804232/HIVE-13622.4.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 54 failed/errored test(s), 10026 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
[jira] [Updated] (HIVE-13776) MCSK repair fails for some tables with MetaException(message:Expected 1 components, got 3 (timestamp=2015-05-01/type=open/num=1))
[
https://issues.apache.org/jira/browse/HIVE-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13776:
Description:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table/timestamp=2015-05-01/type=open/num=1/file-1
/table/timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by timestamp (date), type (string) and num (integer).
Running MCSK repair succeeds for some files, but intermittently throws an
exception like this:
{code}
MetaException(message:Expected 1 components, got 3
(timestamp=2015-05-01/type=open/num=1))
{code}
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
was:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table/timestamp=2015-05-01/type=open/num=1/file-1
/table/timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by timestamp (date), type (string) and num (integer).
Running MCSK repair succeeds for some files, but intermittently throws an
exception like this:
{code}
MCSK repair fails for some tables with MetaException(message:Expected 1
components, got 3 (timestamp=2015-05-01/type=open/num=1))
{code}
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (timestamp=2015-05-01/type=open/num=1))
> -
>
> Key: HIVE-13776
> URL: https://issues.apache.org/jira/browse/HIVE-13776
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>
> We're seeing msck repair fail randomly for a directory layout like this:
> {code}
> /table/timestamp=2015-05-01
> /table/timestamp=2015-05-01/type=open
> /table/timestamp=2015-05-01/type=open/num=1/file-1
> /table/timestamp=2015-05-01/type=open/num=1/file-2
> {code}
> The table is partitioned by timestamp (date), type (string) and num
> (integer). Running MCSK repair succeeds for some files, but intermittently
> throws an exception like this:
> {code}
> MetaException(message:Expected 1 components, got 3
> (timestamp=2015-05-01/type=open/num=1))
> {code}
> Appears that there is a bug in DDLTask.mcsk
> (https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13776) MCSK repair fails for some tables with MetaException(message:Expected 1 components, got 3 (timestamp=2015-05-01/type=open/num=1))
[
https://issues.apache.org/jira/browse/HIVE-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13776:
Description:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table/timestamp=2015-05-01/type=open/num=1/file-1
/table/timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by timestamp (date), type (string) and num (integer).
Running MCSK repair succeeds for some files, but intermittently throws an
exception like this:
{code}
MCSK repair fails for some tables with MetaException(message:Expected 1
components, got 3 (timestamp=2015-05-01/type=open/num=1))
{code}
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
was:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table/timestamp=2015-05-01/type=open/num=1/file-1
/table/timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by timestamp (date), type (string) and num (integer).
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (timestamp=2015-05-01/type=open/num=1))
> -
>
> Key: HIVE-13776
> URL: https://issues.apache.org/jira/browse/HIVE-13776
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>
> We're seeing msck repair fail randomly for a directory layout like this:
> {code}
> /table/timestamp=2015-05-01
> /table/timestamp=2015-05-01/type=open
> /table/timestamp=2015-05-01/type=open/num=1/file-1
> /table/timestamp=2015-05-01/type=open/num=1/file-2
> {code}
> The table is partitioned by timestamp (date), type (string) and num
> (integer). Running MCSK repair succeeds for some files, but intermittently
> throws an exception like this:
> {code}
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (timestamp=2015-05-01/type=open/num=1))
> {code}
> Appears that there is a bug in DDLTask.mcsk
> (https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13776) MCSK repair fails for some tables with MetaException(message:Expected 1 components, got 3 (timestamp=2015-05-01/type=open/num=1))
[
https://issues.apache.org/jira/browse/HIVE-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13776:
Summary: MCSK repair fails for some tables with
MetaException(message:Expected 1 components, got 3
(timestamp=2015-05-01/type=open/num=1)) (was: MCSK repair fails for some
tables with MetaException(message:Expected 1 components, got 3
(part11=val11/part21=val21/part31=val31)))
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (timestamp=2015-05-01/type=open/num=1))
> -
>
> Key: HIVE-13776
> URL: https://issues.apache.org/jira/browse/HIVE-13776
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>
> We're seeing msck repair fail randomly for a directory layout like this:
> {code}
> /table/timestamp=2015-05-01
> /table/timestamp=2015-05-01/type=open
> /table/timestamp=2015-05-01/type=open/num=1/file-1
> /table/timestamp=2015-05-01/type=open/num=1/file-2
> {code}
> The table is partitioned by timestamp (date), type (string) and num (integer).
> Appears that there is a bug in DDLTask.mcsk
> (https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13776) MCSK repair fails for some tables with MetaException(message:Expected 1 components, got 3 (part11=val11/part21=val21/part31=val31))
[
https://issues.apache.org/jira/browse/HIVE-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13776:
Description:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table/timestamp=2015-05-01/type=open/num=1/file-1
/table/timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by timestamp (date), type (string) and num (integer).
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
was:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table//timestamp=2015-05-01/type=open/num=1/file-1
/table//timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by part11,
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (part11=val11/part21=val21/part31=val31))
> ---
>
> Key: HIVE-13776
> URL: https://issues.apache.org/jira/browse/HIVE-13776
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>
> We're seeing msck repair fail randomly for a directory layout like this:
> {code}
> /table/timestamp=2015-05-01
> /table/timestamp=2015-05-01/type=open
> /table/timestamp=2015-05-01/type=open/num=1/file-1
> /table/timestamp=2015-05-01/type=open/num=1/file-2
> {code}
> The table is partitioned by timestamp (date), type (string) and num (integer).
> Appears that there is a bug in DDLTask.mcsk
> (https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13776) MCSK repair fails for some tables with MetaException(message:Expected 1 components, got 3 (part11=val11/part21=val21/part31=val31))
[
https://issues.apache.org/jira/browse/HIVE-13776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13776:
Description:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/timestamp=2015-05-01
/table/timestamp=2015-05-01/type=open
/table//timestamp=2015-05-01/type=open/num=1/file-1
/table//timestamp=2015-05-01/type=open/num=1/file-2
{code}
The table is partitioned by part11,
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
was:
We're seeing msck repair fail randomly for a directory layout like this:
{code}
/table/part11=val11
/table/part11=val11/part21=val21
/table/part11=val11/part21=val21/part31=val31
/table/part11=val11/part21=val21/part32=val32
{code}
Appears that there is a bug in DDLTask.mcsk
(https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
> MCSK repair fails for some tables with MetaException(message:Expected 1
> components, got 3 (part11=val11/part21=val21/part31=val31))
> ---
>
> Key: HIVE-13776
> URL: https://issues.apache.org/jira/browse/HIVE-13776
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>
> We're seeing msck repair fail randomly for a directory layout like this:
> {code}
> /table/timestamp=2015-05-01
> /table/timestamp=2015-05-01/type=open
> /table//timestamp=2015-05-01/type=open/num=1/file-1
> /table//timestamp=2015-05-01/type=open/num=1/file-2
> {code}
> The table is partitioned by part11,
> Appears that there is a bug in DDLTask.mcsk
> (https://github.com/apache/hive/blob/6cb5dbe64968f052266f491492225e925e437dac/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L1832).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13774) add actual access checks to GenericUDTFGetSplits
[ https://issues.apache.org/jira/browse/HIVE-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-13774. - Resolution: Invalid Nm, didn't notice the code path already exists. GenericUDTFGetSplits.createPlanFragment -> Driver.compileAndRespond -> Driver.compileInternal -> Driver.compile -> Driver.doAuthorization > add actual access checks to GenericUDTFGetSplits > > > Key: HIVE-13774 > URL: https://issues.apache.org/jira/browse/HIVE-13774 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > GenericUDTFGetSplits generates splits without proper Hive compilation. We > need to actually perform security checks (Driver.doAuthorization) before we > sign and return the splits -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287121#comment-15287121 ] Sergey Shelukhin commented on HIVE-10815: - [~thejas] any comments? This will also result in some load balancing :) Should it be done once per process though, to minimize the number of connections? > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287111#comment-15287111
]
Matt McCline commented on HIVE-13562:
-
[~ashutoshc] Thank you for your review!
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287084#comment-15287084
]
Ashutosh Chauhan commented on HIVE-13562:
-
LGTM +1
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287051#comment-15287051 ] Ashutosh Chauhan commented on HIVE-13269: - +1 > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13269: --- Attachment: (was: HIVE-13269.04.patch) > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13269: --- Attachment: HIVE-13269.04.patch > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286958#comment-15286958 ] Wei Zheng commented on HIVE-13730: -- Test failures are not related. [~vikram.dixit] Can you review please? > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13767:
---
Fix Version/s: 1.3.0
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.calcite.rex.RexUtil$FixNullabilityShuttle.visitInputRef(RexUtil.java:2042)
> ~[calcite-core-1.6.0.2.5.0.0-248.jar:1.6.0.2.5.0.0-248]
> at
>
[jira] [Updated] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13767:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Fails are unrelated, except semijoin5.q for which I needed to regenerate the q
file as I had generated originally on top of HIVE-13068.
Pushed to master, thanks for reviewing [~ashutoshc]!
> Wrong type inferred in Semijoin condition leads to AssertionError
> -
>
> Key: HIVE-13767
> URL: https://issues.apache.org/jira/browse/HIVE-13767
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.1.0
>
> Attachments: HIVE-13767.patch
>
>
> Following query fails to run:
> {noformat}
> SELECT
> COALESCE(498, LEAD(COALESCE(-973, -684, 515)) OVER (PARTITION BY
> (t2.int_col_10 + t1.smallint_col_50) ORDER BY (t2.int_col_10 +
> t1.smallint_col_50), FLOOR(t1.double_col_16) DESC), 524) AS int_col,
> (t2.int_col_10) + (t1.smallint_col_50) AS int_col_1,
> FLOOR(t1.double_col_16) AS float_col,
> COALESCE(SUM(COALESCE(62, -380, -435)) OVER (PARTITION BY (t2.int_col_10
> + t1.smallint_col_50) ORDER BY (t2.int_col_10 + t1.smallint_col_50) DESC,
> FLOOR(t1.double_col_16) DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 48
> FOLLOWING), 704) AS int_col_2
> FROM table_1 t1
> INNER JOIN table_18 t2 ON (((t2.tinyint_col_15) = (t1.bigint_col_7)) AND
>((t2.decimal2709_col_9) =
> (t1.decimal2016_col_26))) AND
>((t2.tinyint_col_20) = (t1.tinyint_col_3))
> WHERE (t2.smallint_col_19) IN (SELECT
> COALESCE(-92, -994) AS int_col
> FROM table_1 tt1
> INNER JOIN table_18 tt2 ON (tt2.decimal1911_col_16) =
> (tt1.decimal2612_col_77)
> WHERE (t1.timestamp_col_9) = (tt2.timestamp_col_18));
> {noformat}
> Following error is seen in the logs:
> {noformat}
> 2016-04-27T04:32:09,605 WARN [...2a24 HiveServer2-Handler-Pool: Thread-211]:
> thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(501)) - Error
> executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.AssertionError: mismatched type $8 TIMESTAMP(9)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:327)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:458)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:435)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0.2.5.0.0-248.jar:2.1.0.2.5.0.0-248]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.AssertionError: mismatched type $8
[jira] [Commented] (HIVE-13767) Wrong type inferred in Semijoin condition leads to AssertionError
[
https://issues.apache.org/jira/browse/HIVE-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286791#comment-15286791
]
Hive QA commented on HIVE-13767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804213/HIVE-13767.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 90 failed/errored test(s), 9919 tests
executed
*Failed tests:*
{noformat}
TestColumn - did not produce a TEST-*.xml file
TestCookieSigner - did not produce a TEST-*.xml file
TestHS2HttpServer - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestHiveSQLException - did not produce a TEST-*.xml file
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file
TestLdapAuthenticationProviderImpl - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more
- did not produce a TEST-*.xml file
TestPlainSaslHelper - did not produce a TEST-*.xml file
TestPluggableHiveSessionImpl - did not produce a TEST-*.xml file
TestRetryingThriftCLIServiceClient - did not produce a TEST-*.xml file
TestServerOptionsProcessor - did not produce a TEST-*.xml file
TestSessionGlobalInitFile - did not produce a TEST-*.xml file
TestSessionHooks - did not produce a TEST-*.xml file
TestSessionManagerMetrics - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin5
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucketpruning1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_column_names_with_leading_and_trailing_spaces
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_without_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_trailing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_not
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_case
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_windowing_gby
[jira] [Commented] (HIVE-13721) HPL/SQL COPY FROM FTP Statement: lack of DIR option leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-13721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286787#comment-15286787
]
Dmitry Tolpeko commented on HIVE-13721:
---
Fixing this.
> HPL/SQL COPY FROM FTP Statement: lack of DIR option leads to NPE
>
>
> Key: HIVE-13721
> URL: https://issues.apache.org/jira/browse/HIVE-13721
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Carter Shanklin
>
> The docs (http://www.hplsql.org/copy-from-ftp) suggest DIR is optional. When
> I left it out in:
> {code}
> copy from ftp hdp250.example.com user 'vagrant' pwd 'vagrant' files
> 'sampledata.csv' to /tmp overwrite
> {code}
> I got:
> {code}
> Ln:2 Connected to ftp: hdp250.example.com (29 ms)
> Ln:2 Retrieving directory listing
> Listing the current working FTP directory
> Ln:2 Files to copy: 45 bytes, 1 file, 0 subdirectories scanned (27 ms)
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.hplsql.Ftp.getTargetFileName(Ftp.java:342)
> at org.apache.hive.hplsql.Ftp.run(Ftp.java:149)
> at org.apache.hive.hplsql.Ftp.copyFiles(Ftp.java:121)
> at org.apache.hive.hplsql.Ftp.run(Ftp.java:91)
> at org.apache.hive.hplsql.Exec.visitCopy_from_ftp_stmt(Exec.java:1292)
> at org.apache.hive.hplsql.Exec.visitCopy_from_ftp_stmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$Copy_from_ftp_stmtContext.accept(HplsqlParser.java:11956)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:994)
> at org.apache.hive.hplsql.Exec.visitStmt(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$StmtContext.accept(HplsqlParser.java:1012)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at
> org.apache.hive.hplsql.HplsqlBaseVisitor.visitBlock(HplsqlBaseVisitor.java:28)
> at
> org.apache.hive.hplsql.HplsqlParser$BlockContext.accept(HplsqlParser.java:446)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visitChildren(AbstractParseTreeVisitor.java:70)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:901)
> at org.apache.hive.hplsql.Exec.visitProgram(Exec.java:52)
> at
> org.apache.hive.hplsql.HplsqlParser$ProgramContext.accept(HplsqlParser.java:389)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
> at org.apache.hive.hplsql.Exec.run(Exec.java:760)
> at org.apache.hive.hplsql.Exec.run(Exec.java:736)
> at org.apache.hive.hplsql.Hplsql.main(Hplsql.java:23)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Traceback leads to:
> {code}
> /**
>* Get the target file relative path and name
>*/
> String getTargetFileName(String file) {
> int len = dir.length();
> return targetDir + file.substring(len);
> }
> {code}
> in Ftp.java
> When I added DIR '/' this worked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13616) Investigate renaming a table without invalidating the column stats
[ https://issues.apache.org/jira/browse/HIVE-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13616: Attachment: HIVE-13616.2.patch Attached patch-2: address comments. > Investigate renaming a table without invalidating the column stats > -- > > Key: HIVE-13616 > URL: https://issues.apache.org/jira/browse/HIVE-13616 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13616.1.patch, HIVE-13616.2.patch > > > Right now when we rename a table, we clear the column stats rather than > updating it (HIVE-9720) since ObjectStore uses DN to talk to DB. Investigate > the possibility that if we can achieve updating the stats without rescanning > the whole table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13269: --- Status: Open (was: Patch Available) > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13269: --- Attachment: HIVE-13269.04.patch > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13269: --- Status: Patch Available (was: In Progress) > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13269 started by Jesus Camacho Rodriguez. -- > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[
https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286393#comment-15286393
]
Hive QA commented on HIVE-13730:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804208/HIVE-13730.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 82 failed/errored test(s), 10042 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby5_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_innerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input1_limit
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_vc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_mixed
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.llap.daemon.impl.TestLlapDaemonProtocolServerImpl.test
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13750:
---
Attachment: HIVE-13750.01.patch
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work stopped] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13750 stopped by Jesus Camacho Rodriguez.
--
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13750:
---
Status: Patch Available (was: Open)
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13750 started by Jesus Camacho Rodriguez.
--
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13197) Add adapted constprog2.q and constprog_partitioner.q tests back
[ https://issues.apache.org/jira/browse/HIVE-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286233#comment-15286233 ] Jesus Camacho Rodriguez commented on HIVE-13197: +1 > Add adapted constprog2.q and constprog_partitioner.q tests back > --- > > Key: HIVE-13197 > URL: https://issues.apache.org/jira/browse/HIVE-13197 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Jesus Camacho Rodriguez >Assignee: Ashutosh Chauhan > Attachments: HIVE-13197.patch > > > HIVE-12749 removes constprog2.q and constprog_partitioner.q tests, as they > did not test constant propagation anymore. Ideally, we should create them > again with compatible types to test constant propagation and constant > propagation in the presence of partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8285) Reference equality is used on boolean values in PartitionPruner#removeTruePredciates()
[
https://issues.apache.org/jira/browse/HIVE-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-8285:
-
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
This is no longer a problem.
> Reference equality is used on boolean values in
> PartitionPruner#removeTruePredciates()
> --
>
> Key: HIVE-8285
> URL: https://issues.apache.org/jira/browse/HIVE-8285
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HIVE-8285.patch
>
>
> {code}
> if (e.getTypeInfo() == TypeInfoFactory.booleanTypeInfo
> && eC.getValue() == Boolean.TRUE) {
> {code}
> equals() should be used in the above comparison.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13614) Implementation for PreparedStatement#setBigDecimal
[
https://issues.apache.org/jira/browse/HIVE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286227#comment-15286227
]
Hive QA commented on HIVE-13614:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804202/HIVE-13614.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 104 failed/errored test(s), 10003 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join1.q-and-12-more - did not produce
a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join37
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_rc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_reduce_deduplicate_exclude_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_script_env_var2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_div0
[jira] [Updated] (HIVE-8285) Reference equality is used on boolean values in PartitionPruner#removeTruePredciates()
[
https://issues.apache.org/jira/browse/HIVE-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-8285:
-
Description:
{code}
if (e.getTypeInfo() == TypeInfoFactory.booleanTypeInfo
&& eC.getValue() == Boolean.TRUE) {
{code}
equals() should be used in the above comparison.
was:
{code}
if (e.getTypeInfo() == TypeInfoFactory.booleanTypeInfo
&& eC.getValue() == Boolean.TRUE) {
{code}
equals() should be used in the above comparison.
> Reference equality is used on boolean values in
> PartitionPruner#removeTruePredciates()
> --
>
> Key: HIVE-8285
> URL: https://issues.apache.org/jira/browse/HIVE-8285
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HIVE-8285.patch
>
>
> {code}
> if (e.getTypeInfo() == TypeInfoFactory.booleanTypeInfo
> && eC.getValue() == Boolean.TRUE) {
> {code}
> equals() should be used in the above comparison.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks to Xuefu for the review. > Query occurs performance degradation after enabling parallel order by for > Hive on Spark > --- > > Key: HIVE-13293 > URL: https://issues.apache.org/jira/browse/HIVE-13293 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0 >Reporter: Lifeng Wang >Assignee: Rui Li > Fix For: 2.1.0 > > Attachments: HIVE-13293.1.patch, HIVE-13293.2.patch, > HIVE-13293.3.patch, HIVE-13293.3.patch, HIVE-13293.3.patch > > > I use TPCx-BB to do some performance test on Hive on Spark engine. And found > query 10 has performance degradation when enabling parallel order by. > It seems that sampling cost much time before running the real query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286198#comment-15286198
]
Matt McCline commented on HIVE-13562:
-
[~ashutoshc] I have a patch queued for Hive QA and a review board created. Can
you review the change? Thanks
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: HIVE-13562.02.patch
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: (was: HIVE-13562.02.patch)
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: HIVE-13562.02.patch
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: (was: HIVE-13562.02.patch)
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: proposal.pdf > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: (was: proposal.pdf) > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-10815: - Status: Patch Available (was: Open) Patch rebased to master. > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-10815: - Attachment: HIVE-10815.1.patch > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-10815: - Status: Open (was: Patch Available) > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
