[jira] [Commented] (HIVE-14178) Hive::needsToCopy should reuse FileUtils::equalsFileSystem
[
https://issues.apache.org/jira/browse/HIVE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368946#comment-15368946
]
Hive QA commented on HIVE-14178:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816880/HIVE-14178.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/434/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/434/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-434/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-434/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7a91bbf HIVE-14114 Ensure RecordWriter in streaming API is using
the same UserGroupInformation as StreamingConnection (Eugene Koifman, reviewed
by Wei Zheng)
+ git clean -f -d
Removing
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7a91bbf HIVE-14114 Ensure RecordWriter in streaming API is using
the same UserGroupInformation as StreamingConnection (Eugene Koifman, reviewed
by Wei Zheng)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816880 - PreCommit-HIVE-MASTER-Build
> Hive::needsToCopy should reuse FileUtils::equalsFileSystem
> --
>
> Key: HIVE-14178
> URL: https://issues.apache.org/jira/browse/HIVE-14178
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1, 2.1.0, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14178.1.patch, HIVE-14178.2.patch
>
>
> Clear bug triggered from missing FS checks in Hive.java
> {code}
> //Check if different FileSystems
> if (!srcFs.getClass().equals(destFs.getClass()))
> {
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368945#comment-15368945
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816878/HIVE-13901.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10294 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_external2
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/433/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/433/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-433/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816878 - PreCommit-HIVE-MASTER-Build
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch, HIVE-13901.9.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8079) CTAS_PARCOL_COEXISTENCE should mention bucketing
[
https://issues.apache.org/jira/browse/HIVE-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-8079:
-
Assignee: (was: Eugene Koifman)
> CTAS_PARCOL_COEXISTENCE should mention bucketing
>
>
> Key: HIVE-8079
> URL: https://issues.apache.org/jira/browse/HIVE-8079
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.13.1
>Reporter: Eugene Koifman
>
> {noformat}
> create table FOO clustered by (i) into 8 buckets stored as orc AS select *
> from BAR;
> {noformat}
> this statement throws a SemanticException with
> ErrorMsg.CTAS_PARCOL_COEXISTENCE. The actual message text is
> "CREATE-TABLE-AS-SELECT does not support partitioning in the target table"
> which is confusing since target table is not partitioned.
> The real issue is that bucketed table is not supported
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableAsSelect(CTAS)
> Need to update the message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14114: -- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 1.3.0 Target Version/s: 2.1.0, 1.3.0, 2.2.0 (was: 1.3.0, 2.1.0, 2.2.0) Status: Resolved (was: Patch Available) Committed to branch-1, branch-2.1 and master Thanks Wei for the review > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14114.2.patch, HIVE-14114.3.patch, > HIVE-14114.4.patch, HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-14192:
--
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
1.3.0
Status: Resolved (was: Patch Available)
Committed to branch-1, branch-2.1 and master
Thanks Wei for the review
> False positive error due to thrift
> --
>
> Key: HIVE-14192
> URL: https://issues.apache.org/jira/browse/HIVE-14192
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-14192.2.patch, HIVE-14192.patch
>
>
> Given Thrift definition like this
> {noformat}
> struct LockComponent {
> 1: required LockType type,
> 2: required LockLevel level,
> 3: required string dbname,
> 4: optional string tablename,
> 5: optional string partitionname,
> 6: optional DataOperationType operationType = DataOperationType.UNSET,
> 7: optional bool isAcid = false
> }
> {noformat}
> The generated LockComponent has
> {noformat}
> public LockComponent() {
> this.operationType =
> org.apache.hadoop.hive.metastore.api.DataOperationType.UNSET;
> this.isAcid = false;
> }
> public boolean isSetOperationType() {
> return this.operationType != null;
> }
> public boolean isSetIsAcid() {
> return EncodingUtils.testBit(__isset_bitfield, __ISACID_ISSET_ID);
> }
> {noformat}
> So bottom line is even if LockComponent is created by old version of the
> client which doesn't have operationType filed, isSetOperationType() will
> still return true on the server.
> This causes a false positive exception in TxnHandler.enqueueLockWithRetry()
> during Rolling Upgrade scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[
https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368908#comment-15368908
]
Hive QA commented on HIVE-13934:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816931/HIVE-13934.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 84 failed/errored test(s), 10279 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vectorization_16.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_acid3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_include_no_sel
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1
[jira] [Comment Edited] (HIVE-11402) HS2 - add an option to disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368906#comment-15368906 ] Thejas M Nair edited comment on HIVE-11402 at 7/9/16 3:08 AM: -- Maybe pass the lock object to operation async execution and have it release it when its done ? ie, send it all way to SQLOperation.runInternal .. was (Author: thejas): Maybe pass the lock object to operation async execution and have it release it when its done ? > HS2 - add an option to disallow parallel query execution within a single > Session > > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.02.patch, > HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - add an option to disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368906#comment-15368906 ] Thejas M Nair commented on HIVE-11402: -- Maybe pass the lock object to operation async execution and have it release it when its done ? > HS2 - add an option to disallow parallel query execution within a single > Session > > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.02.patch, > HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14201) Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
[
https://issues.apache.org/jira/browse/HIVE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14201:
---
Attachment: HIVE-14201.1.patch
> Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
>
>
> Key: HIVE-14201
> URL: https://issues.apache.org/jira/browse/HIVE-14201
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: Performance
> Attachments: HIVE-14201.1.patch
>
>
> {code}
> public PrimitiveCategory getPrimitiveCategory() {
> return getPrimitiveTypeEntry().primitiveCategory;
> }
> public PrimitiveTypeEntry getPrimitiveTypeEntry() {
> return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
> }
> /**
>* Get the TypeEntry for the given base type name (int, varchar, etc).
>*/
> public static PrimitiveTypeEntry getTypeEntryFromTypeName(String typeName) {
> return typeNameToTypeEntry.get(typeName);
> }
> // Base type name to PrimitiveTypeEntry map.
> private static final Map typeNameToTypeEntry =
> new HashMap();
> {code}
> Not even a single intern() call there. SMH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14201) Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
[
https://issues.apache.org/jira/browse/HIVE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14201:
---
Affects Version/s: 2.2.0
> Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
>
>
> Key: HIVE-14201
> URL: https://issues.apache.org/jira/browse/HIVE-14201
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: Performance
> Attachments: HIVE-14201.1.patch
>
>
> {code}
> public PrimitiveCategory getPrimitiveCategory() {
> return getPrimitiveTypeEntry().primitiveCategory;
> }
> public PrimitiveTypeEntry getPrimitiveTypeEntry() {
> return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
> }
> /**
>* Get the TypeEntry for the given base type name (int, varchar, etc).
>*/
> public static PrimitiveTypeEntry getTypeEntryFromTypeName(String typeName) {
> return typeNameToTypeEntry.get(typeName);
> }
> // Base type name to PrimitiveTypeEntry map.
> private static final Map typeNameToTypeEntry =
> new HashMap();
> {code}
> Not even a single intern() call there. SMH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14201) Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
[
https://issues.apache.org/jira/browse/HIVE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14201:
---
Component/s: Types
> Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
>
>
> Key: HIVE-14201
> URL: https://issues.apache.org/jira/browse/HIVE-14201
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: Performance
> Attachments: HIVE-14201.1.patch
>
>
> {code}
> public PrimitiveCategory getPrimitiveCategory() {
> return getPrimitiveTypeEntry().primitiveCategory;
> }
> public PrimitiveTypeEntry getPrimitiveTypeEntry() {
> return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
> }
> /**
>* Get the TypeEntry for the given base type name (int, varchar, etc).
>*/
> public static PrimitiveTypeEntry getTypeEntryFromTypeName(String typeName) {
> return typeNameToTypeEntry.get(typeName);
> }
> // Base type name to PrimitiveTypeEntry map.
> private static final Map typeNameToTypeEntry =
> new HashMap();
> {code}
> Not even a single intern() call there. SMH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14201) Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
[
https://issues.apache.org/jira/browse/HIVE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14201:
---
Labels: Performance (was: )
> Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
>
>
> Key: HIVE-14201
> URL: https://issues.apache.org/jira/browse/HIVE-14201
> Project: Hive
> Issue Type: Bug
> Components: Types
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: Performance
> Attachments: HIVE-14201.1.patch
>
>
> {code}
> public PrimitiveCategory getPrimitiveCategory() {
> return getPrimitiveTypeEntry().primitiveCategory;
> }
> public PrimitiveTypeEntry getPrimitiveTypeEntry() {
> return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
> }
> /**
>* Get the TypeEntry for the given base type name (int, varchar, etc).
>*/
> public static PrimitiveTypeEntry getTypeEntryFromTypeName(String typeName) {
> return typeNameToTypeEntry.get(typeName);
> }
> // Base type name to PrimitiveTypeEntry map.
> private static final Map typeNameToTypeEntry =
> new HashMap();
> {code}
> Not even a single intern() call there. SMH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - add an option to disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368895#comment-15368895 ] Thejas M Nair commented on HIVE-11402: -- Sorry about the delay in reviewing. This will however not disallow query execution in parallel when the query execution is done asynchronously (the default mode with our jdbc driver). > HS2 - add an option to disallow parallel query execution within a single > Session > > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.02.patch, > HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14201) Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
[
https://issues.apache.org/jira/browse/HIVE-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-14201:
--
Assignee: Gopal V
> Types: Remove hash-lookup from PrimitiveTypeInfo::getPrimitiveCategory()
>
>
> Key: HIVE-14201
> URL: https://issues.apache.org/jira/browse/HIVE-14201
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>
> {code}
> public PrimitiveCategory getPrimitiveCategory() {
> return getPrimitiveTypeEntry().primitiveCategory;
> }
> public PrimitiveTypeEntry getPrimitiveTypeEntry() {
> return PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(typeName);
> }
> /**
>* Get the TypeEntry for the given base type name (int, varchar, etc).
>*/
> public static PrimitiveTypeEntry getTypeEntryFromTypeName(String typeName) {
> return typeNameToTypeEntry.get(typeName);
> }
> // Base type name to PrimitiveTypeEntry map.
> private static final Map typeNameToTypeEntry =
> new HashMap();
> {code}
> Not even a single intern() call there. SMH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14199) Enable Bucket Pruning for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368883#comment-15368883 ] Saket Saurabh commented on HIVE-14199: -- Thanks [~gopalv] for the comment. I have updated the patch with these changes. Currently, to disable the codepath for legacy layouts, I do not consider the case of matching the bucketName against the AcidUtils.LEGACY_BUCKET_DIGIT_PATTERN. So, I am thinking these legacy layouts will be ignored then. > Enable Bucket Pruning for ACID tables > - > > Key: HIVE-14199 > URL: https://issues.apache.org/jira/browse/HIVE-14199 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14199.01.patch, HIVE-14199.02.patch > > > Currently, ACID tables do not benefit from the bucket pruning feature > introduced in HIVE-11525. The reason for this has been the fact that bucket > pruning happens at split generation level and for ACID, traditionally the > delta files were never split. The parallelism for ACID was then restricted to > the number of buckets. There would be as many splits as the number of buckets > and each worker processing one split would inevitably read all the delta > files for that bucket, even when the query may have originally required only > one of the buckets to be read. > However, HIVE-14035 now enables even the delta files to be also split. What > this means is that now we have enough information at the split generation > level to determine appropriate buckets to process for the delta files. This > can efficiently allow us to prune unnecessary buckets for delta files and > will lead to good performance gain for a large number of selective queries on > ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14199) Enable Bucket Pruning for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14199: - Attachment: HIVE-14199.02.patch > Enable Bucket Pruning for ACID tables > - > > Key: HIVE-14199 > URL: https://issues.apache.org/jira/browse/HIVE-14199 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14199.01.patch, HIVE-14199.02.patch > > > Currently, ACID tables do not benefit from the bucket pruning feature > introduced in HIVE-11525. The reason for this has been the fact that bucket > pruning happens at split generation level and for ACID, traditionally the > delta files were never split. The parallelism for ACID was then restricted to > the number of buckets. There would be as many splits as the number of buckets > and each worker processing one split would inevitably read all the delta > files for that bucket, even when the query may have originally required only > one of the buckets to be read. > However, HIVE-14035 now enables even the delta files to be also split. What > this means is that now we have enough information at the split generation > level to determine appropriate buckets to process for the delta files. This > can efficiently allow us to prune unnecessary buckets for delta files and > will lead to good performance gain for a large number of selective queries on > ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Status: Patch Available (was: Open) > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch, > HIVE-14200.3.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Attachment: HIVE-14200.3.patch Reuse the max factor as the total # of reducers, if auto-reducer parallelism is on, but the min is below a single reducer. > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch, > HIVE-14200.3.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Status: Open (was: Patch Available) > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368867#comment-15368867 ] Gunther Hagleitner commented on HIVE-14200: --- Can we keep the inflation in these cases though? I know ppl using a large max factor esp to over provision if there are only few reducers. > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368861#comment-15368861 ] Eugene Koifman commented on HIVE-14114: --- failures have age > 1 except testCliDriver_list_bucket_dml_13 which had the same failure in https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/427/testReport/ > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14114.2.patch, HIVE-14114.3.patch, > HIVE-14114.4.patch, HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[
https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368856#comment-15368856
]
Hive QA commented on HIVE-14114:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816857/HIVE-14114.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10294 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/431/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/431/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-431/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816857 - PreCommit-HIVE-MASTER-Build
> Ensure RecordWriter in streaming API is using the same UserGroupInformation
> as StreamingConnection
> --
>
> Key: HIVE-14114
> URL: https://issues.apache.org/jira/browse/HIVE-14114
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14114.2.patch, HIVE-14114.3.patch,
> HIVE-14114.4.patch, HIVE-14114.patch
>
>
> currently both DelimitedInputWriter and StrictJsonWriter perform some
> Metastore access operations but without using UGI created by the caller for
> Metastore operations made by matching StreamingConnection & TransactionBatch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368848#comment-15368848
]
Sergey Shelukhin edited comment on HIVE-14089 at 7/9/16 1:08 AM:
-
The latest error is too ORC for me to figure out immediately... will look next
week unless you know what it could be cc [~prasanth_j]
Looks like position provider is missing stuff for nested reader?
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_45]
at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_45]
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
~[?:1.8.0_45]
at
org.apache.orc.OrcProto$RowIndexEntry.getPositions(OrcProto.java:6867)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.orc.impl.RecordReaderImpl$PositionProviderImpl.getNext(RecordReaderImpl.java:230)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$StringStreamReader.seek(EncodedTreeReaderFactory.java:228)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$MapStreamReader.seek(EncodedTreeReaderFactory.java:1810)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.positionInStreams(OrcEncodedDataConsumer.java:147)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.decodeBatch(OrcEncodedDataConsumer.java:103)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.decodeBatch(OrcEncodedDataConsumer.java:45)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:76)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:30)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:420)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:399)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:210)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:207)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_45]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_45]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
~[hadoop-common-2.6.1.jar:?]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:207)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:91)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
~[tez-common-0.8.3.jar:0.8.3]
... 4 more
{noformat}
was (Author: sershe):
The latest error is too ORC for me to figure out immediately... will look next
week unless you know what it could be cc [~pjayachandran]
Looks like position provider is missing stuff for nested reader?
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_45]
at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_45]
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
~[?:1.8.0_45]
at
org.apache.orc.OrcProto$RowIndexEntry.getPositions(OrcProto.java:6867)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.orc.impl.RecordReaderImpl$PositionProviderImpl.getNext(RecordReaderImpl.java:230)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$StringStreamReader.seek(EncodedTreeReaderFactory.java:228)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$MapStreamReader.seek(EncodedTreeReaderFactory.java:1810)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368848#comment-15368848
]
Sergey Shelukhin commented on HIVE-14089:
-
The latest error is too ORC for me to figure out immediately... will look next
week unless you know what it could be cc [~pjayachandran]
Looks like position provider is missing stuff for nested reader?
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_45]
at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_45]
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
~[?:1.8.0_45]
at
org.apache.orc.OrcProto$RowIndexEntry.getPositions(OrcProto.java:6867)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.orc.impl.RecordReaderImpl$PositionProviderImpl.getNext(RecordReaderImpl.java:230)
~[hive-orc-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$StringStreamReader.seek(EncodedTreeReaderFactory.java:228)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedTreeReaderFactory$MapStreamReader.seek(EncodedTreeReaderFactory.java:1810)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.positionInStreams(OrcEncodedDataConsumer.java:147)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.decodeBatch(OrcEncodedDataConsumer.java:103)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.decodeBatch(OrcEncodedDataConsumer.java:45)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:76)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:30)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:420)
~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:399)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:210)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:207)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_45]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_45]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
~[hadoop-common-2.6.1.jar:?]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:207)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:91)
~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
~[tez-common-0.8.3.jar:0.8.3]
... 4 more
{noformat}
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.WIP.3.patch
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: (was: HIVE-14069.WIP.3.patch)
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14027 started by Jesus Camacho Rodriguez.
--
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.01.patch, HIVE-14027.02.patch,
> HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Attachment: HIVE-14027.02.patch
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.01.patch, HIVE-14027.02.patch,
> HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Status: Open (was: Patch Available)
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.01.patch, HIVE-14027.02.patch,
> HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Status: Patch Available (was: In Progress)
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.01.patch, HIVE-14027.02.patch,
> HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14128) Parallelize jobClose phases
[ https://issues.apache.org/jira/browse/HIVE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14128: Status: Patch Available (was: Open) > Parallelize jobClose phases > --- > > Key: HIVE-14128 > URL: https://issues.apache.org/jira/browse/HIVE-14128 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.1.0, 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14128.1.patch, HIVE-14128.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14128) Parallelize jobClose phases
[ https://issues.apache.org/jira/browse/HIVE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14128: Assignee: Ashutosh Chauhan Status: Open (was: Patch Available) > Parallelize jobClose phases > --- > > Key: HIVE-14128 > URL: https://issues.apache.org/jira/browse/HIVE-14128 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.1.0, 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14128.1.patch, HIVE-14128.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14128) Parallelize jobClose phases
[ https://issues.apache.org/jira/browse/HIVE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14128: Attachment: HIVE-14128.1.patch > Parallelize jobClose phases > --- > > Key: HIVE-14128 > URL: https://issues.apache.org/jira/browse/HIVE-14128 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14128.1.patch, HIVE-14128.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Status: Patch Available (was: Open) > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Attachment: HIVE-14200.2.patch > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch, HIVE-14200.2.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Description: The min/max factors offer no real improvement when the fractions are meaningless, for example when 0.25 * 2 is applied as the min. > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368806#comment-15368806
]
Sergey Shelukhin edited comment on HIVE-14089 at 7/9/16 12:27 AM:
--
Patch backup... gets as far as positionInStreams :) I am still looking at it
today. Fixed some issues with new schema logic and various indexing issues
caused by nesting (probably a similar problem in positionInStreams...). Please
do not remove existing logic changes if making another WIP patch on top
was (Author: sershe):
Patch backup... gets as far as positionInStreams :) I am still looking at it
today.
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14069.WIP.3.patch, HIVE-14089.WIP.2.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Attachment: HIVE-14200.1.patch > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14200.1.patch > > > The min/max factors offer no real improvement when the fractions are > meaningless, for example when 0.25 * 2 is applied as the min. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14069.WIP.3.patch
Patch backup... gets as far as positionInStreams :) I am still looking at it
today.
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14069.WIP.3.patch, HIVE-14089.WIP.2.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14199) Enable Bucket Pruning for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368799#comment-15368799 ] Gopal V edited comment on HIVE-14199 at 7/9/16 12:23 AM: - [~saketj]: I recommend reusing the AcidUtils.BUCKET_DIGIT_PATTERN instead of a new regex for this case. And disable the codepath for legacy layouts (ACID streaming had bucket-id issues historically). was (Author: gopalv): [~saketj]: I recommend reusing the AcidUtils.BUCKET_DIGIT_PATTERN instead of a new regex for this case. > Enable Bucket Pruning for ACID tables > - > > Key: HIVE-14199 > URL: https://issues.apache.org/jira/browse/HIVE-14199 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14199.01.patch > > > Currently, ACID tables do not benefit from the bucket pruning feature > introduced in HIVE-11525. The reason for this has been the fact that bucket > pruning happens at split generation level and for ACID, traditionally the > delta files were never split. The parallelism for ACID was then restricted to > the number of buckets. There would be as many splits as the number of buckets > and each worker processing one split would inevitably read all the delta > files for that bucket, even when the query may have originally required only > one of the buckets to be read. > However, HIVE-14035 now enables even the delta files to be also split. What > this means is that now we have enough information at the split generation > level to determine appropriate buckets to process for the delta files. This > can efficiently allow us to prune unnecessary buckets for delta files and > will lead to good performance gain for a large number of selective queries on > ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14199) Enable Bucket Pruning for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368799#comment-15368799 ] Gopal V commented on HIVE-14199: [~saketj]: I recommend reusing the AcidUtils.BUCKET_DIGIT_PATTERN instead of a new regex for this case. > Enable Bucket Pruning for ACID tables > - > > Key: HIVE-14199 > URL: https://issues.apache.org/jira/browse/HIVE-14199 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14199.01.patch > > > Currently, ACID tables do not benefit from the bucket pruning feature > introduced in HIVE-11525. The reason for this has been the fact that bucket > pruning happens at split generation level and for ACID, traditionally the > delta files were never split. The parallelism for ACID was then restricted to > the number of buckets. There would be as many splits as the number of buckets > and each worker processing one split would inevitably read all the delta > files for that bucket, even when the query may have originally required only > one of the buckets to be read. > However, HIVE-14035 now enables even the delta files to be also split. What > this means is that now we have enough information at the split generation > level to determine appropriate buckets to process for the delta files. This > can efficiently allow us to prune unnecessary buckets for delta files and > will lead to good performance gain for a large number of selective queries on > ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-14200: -- Assignee: Gopal V > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14200) Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
[ https://issues.apache.org/jira/browse/HIVE-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14200: --- Summary: Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0 (was: Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor ) > Tez: disable auto-reducer parallelism when reducer-count * > min.partition.factor < 1.0 > - > > Key: HIVE-14200 > URL: https://issues.apache.org/jira/browse/HIVE-14200 > Project: Hive > Issue Type: Bug >Reporter: Gopal V > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14168) Avoid serializing all parameters from HiveConf.java into in-memory HiveConf instances
[
https://issues.apache.org/jira/browse/HIVE-14168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368780#comment-15368780
]
Sergey Shelukhin commented on HIVE-14168:
-
I was recently looking at that code. That was rather surprising and sometimes
actually results in unexpected behavior (e.g. schematool tries to get
connection settings ensuring they are set, without the default values, but with
this magic map the default values are returned to it anyway since they are
explicitly added to configuration).
I think it would be a good idea to remove this, but I didn't have time then to
investigate in detail. Looking at the history of this feature may shed some
light on why this is done.
Also there may be code that relies on this behavior unwittingly, but I think we
should fix it after removing the map rather than looking for it in advance.
> Avoid serializing all parameters from HiveConf.java into in-memory HiveConf
> instances
> -
>
> Key: HIVE-14168
> URL: https://issues.apache.org/jira/browse/HIVE-14168
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Priority: Critical
>
> All non-null parameters from HiveConf.java are explicitly set in each
> HiveConf instance.
> {code}
> // Overlay the ConfVars. Note that this ignores ConfVars with null values
> addResource(getConfVarInputStream());
> {code}
> This unnecessarily bloats each Configuration object - 400+ conf variables
> being set instead of probably <30 which would exist in hive-site.xml.
> Looking at a HS2 heapdump - HiveConf is almost always the largest component
> by a long way. Conf objects are also serialized very often - transmitting
> lots of unneeded variables (serialized Hive conf is typically 1000+ variables
> - due to Hadoop injecting it's configs into every config instance).
> As long as HiveConf.get() is the approach used to read from a config - this
> is avoidable. Hive code itself should be doing this.
> This would be a potentially incompatible change for UDFs and other plugins
> which have access to a Configuration object.
> I'd suggest turning off the insert by default, and adding a flag to control
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14199) Enable Bucket Pruning for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14199: - Attachment: HIVE-14199.01.patch Initial commit for this feature. Please note it is dependent on HIVE-14035. > Enable Bucket Pruning for ACID tables > - > > Key: HIVE-14199 > URL: https://issues.apache.org/jira/browse/HIVE-14199 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14199.01.patch > > > Currently, ACID tables do not benefit from the bucket pruning feature > introduced in HIVE-11525. The reason for this has been the fact that bucket > pruning happens at split generation level and for ACID, traditionally the > delta files were never split. The parallelism for ACID was then restricted to > the number of buckets. There would be as many splits as the number of buckets > and each worker processing one split would inevitably read all the delta > files for that bucket, even when the query may have originally required only > one of the buckets to be read. > However, HIVE-14035 now enables even the delta files to be also split. What > this means is that now we have enough information at the split generation > level to determine appropriate buckets to process for the delta files. This > can efficiently allow us to prune unnecessary buckets for delta files and > will lead to good performance gain for a large number of selective queries on > ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368758#comment-15368758 ] Ashutosh Chauhan commented on HIVE-14129: - Motivation is executing movetask in parallel is huge perf gain when executing on FS which are slow. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan > Attachments: HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368756#comment-15368756 ] Ashutosh Chauhan commented on HIVE-14129: - [~thejas] You pointed out couple of issues on HIVE-9665 Can you comment if they are resolved and thus enabling this will be safe now or those are still unresolved. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan > Attachments: HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368704#comment-15368704
]
Eugene Koifman commented on HIVE-14192:
---
all failures have age > 1
> False positive error due to thrift
> --
>
> Key: HIVE-14192
> URL: https://issues.apache.org/jira/browse/HIVE-14192
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14192.2.patch, HIVE-14192.patch
>
>
> Given Thrift definition like this
> {noformat}
> struct LockComponent {
> 1: required LockType type,
> 2: required LockLevel level,
> 3: required string dbname,
> 4: optional string tablename,
> 5: optional string partitionname,
> 6: optional DataOperationType operationType = DataOperationType.UNSET,
> 7: optional bool isAcid = false
> }
> {noformat}
> The generated LockComponent has
> {noformat}
> public LockComponent() {
> this.operationType =
> org.apache.hadoop.hive.metastore.api.DataOperationType.UNSET;
> this.isAcid = false;
> }
> public boolean isSetOperationType() {
> return this.operationType != null;
> }
> public boolean isSetIsAcid() {
> return EncodingUtils.testBit(__isset_bitfield, __ISACID_ISSET_ID);
> }
> {noformat}
> So bottom line is even if LockComponent is created by old version of the
> client which doesn't have operationType filed, isSetOperationType() will
> still return true on the server.
> This causes a false positive exception in TxnHandler.enqueueLockWithRetry()
> during Rolling Upgrade scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14178) Hive::needsToCopy should reuse FileUtils::equalsFileSystem
[
https://issues.apache.org/jira/browse/HIVE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14178:
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Pushed to master & branch-2.1 Thanks, Gopal!
> Hive::needsToCopy should reuse FileUtils::equalsFileSystem
> --
>
> Key: HIVE-14178
> URL: https://issues.apache.org/jira/browse/HIVE-14178
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1, 2.1.0, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14178.1.patch, HIVE-14178.2.patch
>
>
> Clear bug triggered from missing FS checks in Hive.java
> {code}
> //Check if different FileSystems
> if (!srcFs.getClass().equals(destFs.getClass()))
> {
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368677#comment-15368677
]
Hive QA commented on HIVE-14192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816856/HIVE-14192.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10293 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/430/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/430/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-430/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816856 - PreCommit-HIVE-MASTER-Build
> False positive error due to thrift
> --
>
> Key: HIVE-14192
> URL: https://issues.apache.org/jira/browse/HIVE-14192
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14192.2.patch, HIVE-14192.patch
>
>
> Given Thrift definition like this
> {noformat}
> struct LockComponent {
> 1: required LockType type,
> 2: required LockLevel level,
> 3: required string dbname,
> 4: optional string tablename,
> 5: optional string partitionname,
> 6: optional DataOperationType operationType = DataOperationType.UNSET,
> 7: optional bool isAcid = false
> }
> {noformat}
> The generated LockComponent has
> {noformat}
> public LockComponent() {
> this.operationType =
> org.apache.hadoop.hive.metastore.api.DataOperationType.UNSET;
> this.isAcid = false;
> }
> public boolean isSetOperationType() {
> return this.operationType != null;
> }
> public boolean isSetIsAcid() {
> return EncodingUtils.testBit(__isset_bitfield, __ISACID_ISSET_ID);
> }
> {noformat}
> So bottom line is even if LockComponent is created by old version of the
> client which doesn't have operationType filed, isSetOperationType() will
> still return true on the server.
> This causes a false positive exception in TxnHandler.enqueueLockWithRetry()
> during Rolling Upgrade scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14168) Avoid serializing all parameters from HiveConf.java into in-memory HiveConf instances
[
https://issues.apache.org/jira/browse/HIVE-14168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368644#comment-15368644
]
Siddharth Seth commented on HIVE-14168:
---
Any thoughts on this ?
> Avoid serializing all parameters from HiveConf.java into in-memory HiveConf
> instances
> -
>
> Key: HIVE-14168
> URL: https://issues.apache.org/jira/browse/HIVE-14168
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Priority: Critical
>
> All non-null parameters from HiveConf.java are explicitly set in each
> HiveConf instance.
> {code}
> // Overlay the ConfVars. Note that this ignores ConfVars with null values
> addResource(getConfVarInputStream());
> {code}
> This unnecessarily bloats each Configuration object - 400+ conf variables
> being set instead of probably <30 which would exist in hive-site.xml.
> Looking at a HS2 heapdump - HiveConf is almost always the largest component
> by a long way. Conf objects are also serialized very often - transmitting
> lots of unneeded variables (serialized Hive conf is typically 1000+ variables
> - due to Hadoop injecting it's configs into every config instance).
> As long as HiveConf.get() is the approach used to read from a config - this
> is avoidable. Hive code itself should be doing this.
> This would be a potentially incompatible change for UDFs and other plugins
> which have access to a Configuration object.
> I'd suggest turning off the insert by default, and adding a flag to control
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: (was: HIVE-13934.7.patch) > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch, HIVE-13934.7.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: HIVE-13934.7.patch > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch, HIVE-13934.7.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13392) disable speculative execution for ACID Compactor
[
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-13392.
---
Resolution: Fixed
Fix Version/s: 2.1.1
committed to 2.1 as well
https://github.com/apache/hive/commit/39ecc205e64cd1808bebec3ae1dc448e01c48680
> disable speculative execution for ACID Compactor
>
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch,
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative
> execution for the mappers and reducers by setting the
> mapred.map.tasks.speculative.execution and
> mapred.reduce.tasks.speculative.execution JobConf options to false,
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
> Failed to CREATE_FILE
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
> for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on
> 172.18.129.12 because this file lease is currently owned by
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on
> 172.18.129.18
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14195) HiveMetaStoreClient getFunction() does not throw NoSuchObjectException
[
https://issues.apache.org/jira/browse/HIVE-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368514#comment-15368514
]
Hive QA commented on HIVE-14195:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816848/HIVE-14195.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10293 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testSimpleFunction
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testSimpleFunction
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testSimpleFunction
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/429/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/429/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-429/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816848 - PreCommit-HIVE-MASTER-Build
> HiveMetaStoreClient getFunction() does not throw NoSuchObjectException
> --
>
> Key: HIVE-14195
> URL: https://issues.apache.org/jira/browse/HIVE-14195
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14195.patch
>
>
> HiveMetaStoreClient getFunction(dbName, funcName) does not throw
> NoSuchObjectException when no function with funcName exists in the db.
> Instead, I need to search the MetaException message for
> 'NoSuchObjectException'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14197) LLAP service driver precondition failure should include the values
[
https://issues.apache.org/jira/browse/HIVE-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14197:
-
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Committed to branch-2.1 and master. Thanks [~sseth] for the review!
> LLAP service driver precondition failure should include the values
> --
>
> Key: HIVE-14197
> URL: https://issues.apache.org/jira/browse/HIVE-14197
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14197.1.patch
>
>
> LLAP service driver's precondition failure message are like below
> {code}
> Working memory + cache has to be smaller than the container sizing
> {code}
> It will be better to include the actual values for the sizes in the
> precondition failure message.
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14197) LLAP service driver precondition failure should include the values
[
https://issues.apache.org/jira/browse/HIVE-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368460#comment-15368460
]
Siddharth Seth commented on HIVE-14197:
---
+1
> LLAP service driver precondition failure should include the values
> --
>
> Key: HIVE-14197
> URL: https://issues.apache.org/jira/browse/HIVE-14197
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14197.1.patch
>
>
> LLAP service driver's precondition failure message are like below
> {code}
> Working memory + cache has to be smaller than the container sizing
> {code}
> It will be better to include the actual values for the sizes in the
> precondition failure message.
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: HIVE-13934.7.patch > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch, HIVE-13934.7.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14197) LLAP service driver precondition failure should include the values
[
https://issues.apache.org/jira/browse/HIVE-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14197:
-
Status: Patch Available (was: Open)
> LLAP service driver precondition failure should include the values
> --
>
> Key: HIVE-14197
> URL: https://issues.apache.org/jira/browse/HIVE-14197
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14197.1.patch
>
>
> LLAP service driver's precondition failure message are like below
> {code}
> Working memory + cache has to be smaller than the container sizing
> {code}
> It will be better to include the actual values for the sizes in the
> precondition failure message.
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14197) LLAP service driver precondition failure should include the values
[
https://issues.apache.org/jira/browse/HIVE-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14197:
-
Attachment: HIVE-14197.1.patch
> LLAP service driver precondition failure should include the values
> --
>
> Key: HIVE-14197
> URL: https://issues.apache.org/jira/browse/HIVE-14197
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14197.1.patch
>
>
> LLAP service driver's precondition failure message are like below
> {code}
> Working memory + cache has to be smaller than the container sizing
> {code}
> It will be better to include the actual values for the sizes in the
> precondition failure message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14197) LLAP service driver precondition failure should include the values
[
https://issues.apache.org/jira/browse/HIVE-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14197:
-
Description:
LLAP service driver's precondition failure message are like below
{code}
Working memory + cache has to be smaller than the container sizing
{code}
It will be better to include the actual values for the sizes in the
precondition failure message.
NO PRECOMMIT TESTS
was:
LLAP service driver's precondition failure message are like below
{code}
Working memory + cache has to be smaller than the container sizing
{code}
It will be better to include the actual values for the sizes in the
precondition failure message.
> LLAP service driver precondition failure should include the values
> --
>
> Key: HIVE-14197
> URL: https://issues.apache.org/jira/browse/HIVE-14197
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14197.1.patch
>
>
> LLAP service driver's precondition failure message are like below
> {code}
> Working memory + cache has to be smaller than the container sizing
> {code}
> It will be better to include the actual values for the sizes in the
> precondition failure message.
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14198) Refactor aux jar related code to make them more consistent
[ https://issues.apache.org/jira/browse/HIVE-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14198: Affects Version/s: 2.2.0 Status: Patch Available (was: Open) > Refactor aux jar related code to make them more consistent > -- > > Key: HIVE-14198 > URL: https://issues.apache.org/jira/browse/HIVE-14198 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14198.1.patch > > > There are some redundancy and inconsistency between hive.aux.jar.paths and > hive.reloadable.aux.jar.paths and also between MR and spark. > Refactor the code to reuse the same code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14169:
Description:
* When Beeline prints out a {{ResultSet}} to stdout it uses the
{{BeeLine.print}} method
* This method takes the {{ResultSet}} from the completed query and uses a
specified {{OutputFormat}} to print the rows (by default it uses
{{TableOutputFormat}})
* The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
(either a {{IncrementalRows}} or a {{BufferedRows}} class)
The advantage of {{BufferedRows}} is that it can do a global calculation of the
column width, however, this is only useful for {{TableOutputFormat}}. So there
is no need to buffer all the rows if a different {{OutputFormat}} is used. This
JIRA will change the behavior of the {{--incremental}} flag so that it is only
honored if {{TableOutputFormat}} is used.
was:
* When Beeline prints out a {{ResultSet}} to stdout it uses the
{{BeeLine.print}} method
* This method takes the {{ResultSet}} from the completed query and uses a
specified {{OutputFormat}} to print the rows (by default it uses
{{TableOutputFormat}})
* The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
(either a {{IncrementalRows}} or a {{BufferedRows}} class)
* The {{Rows}} class will calculate the optimal width that each row in the
{{ResultSet}} should be displayed with
* However, this width is only relevant / used by {{TableOutputFormat}}
We should modify the logic so that the width is only calculated if
{{TableOutputFormat}} is used. This will save CPU cycles when printing records
out to the user.
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14198) Refactor aux jar related code to make them more consistent
[ https://issues.apache.org/jira/browse/HIVE-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368454#comment-15368454 ] Aihua Xu commented on HIVE-14198: - Attached patch-1: refactor the code so hive.aux.jar.paths will be initialized with the same function call as hive.reloadable.aux.jar.paths. So both will support folder and files. Also made the change to share the same code between MR and spark. > Refactor aux jar related code to make them more consistent > -- > > Key: HIVE-14198 > URL: https://issues.apache.org/jira/browse/HIVE-14198 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14198.1.patch > > > There are some redundancy and inconsistency between hive.aux.jar.paths and > hive.reloadable.aux.jar.paths and also between MR and spark. > Refactor the code to reuse the same code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14198) Refactor aux jar related code to make them more consistent
[ https://issues.apache.org/jira/browse/HIVE-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14198: Summary: Refactor aux jar related code to make them more consistent (was: Refactor aux jar related code to make them more consistency) > Refactor aux jar related code to make them more consistent > -- > > Key: HIVE-14198 > URL: https://issues.apache.org/jira/browse/HIVE-14198 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14198.1.patch > > > There are some redundancy and inconsistency between hive.aux.jar.paths and > hive.reloadable.aux.jar.paths and also between MR and spark. > Refactor the code to reuse the same code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Beeline Row printing should only calculate the width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368452#comment-15368452
]
Sahil Takiar commented on HIVE-14169:
-
[~thejas] I checked and it looks like the column width is only calculated if
{{TableOutputFormat}} is used. {{BufferedRows}} has a method called
{{normalizeWidths}} that is only invoked in {{TableOutputFormat}}. Thus, I am
changing the goal of this JIRA, the code will now only honor the
{{--incremental}} flag if {{TableOutputFormat}} is used. If a different
{{OutputFormat}} is used then {{IncrementalRows}} is always used.
> Beeline Row printing should only calculate the width if TableOutputFormat is
> used
> -
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> * The {{Rows}} class will calculate the optimal width that each row in the
> {{ResultSet}} should be displayed with
> * However, this width is only relevant / used by {{TableOutputFormat}}
> We should modify the logic so that the width is only calculated if
> {{TableOutputFormat}} is used. This will save CPU cycles when printing
> records out to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14169:
Summary: Honor --incremental flag only if TableOutputFormat is used (was:
Beeline Row printing should only calculate the width if TableOutputFormat is
used)
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> * The {{Rows}} class will calculate the optimal width that each row in the
> {{ResultSet}} should be displayed with
> * However, this width is only relevant / used by {{TableOutputFormat}}
> We should modify the logic so that the width is only calculated if
> {{TableOutputFormat}} is used. This will save CPU cycles when printing
> records out to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14198) Refactor aux jar related code to make them more consistency
[ https://issues.apache.org/jira/browse/HIVE-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14198: Attachment: HIVE-14198.1.patch > Refactor aux jar related code to make them more consistency > --- > > Key: HIVE-14198 > URL: https://issues.apache.org/jira/browse/HIVE-14198 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14198.1.patch > > > There are some redundancy and inconsistency between hive.aux.jar.paths and > hive.reloadable.aux.jar.paths and also between MR and spark. > Refactor the code to reuse the same code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14169) Beeline Row printing should only calculate the width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14169:
Status: Patch Available (was: In Progress)
> Beeline Row printing should only calculate the width if TableOutputFormat is
> used
> -
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> * The {{Rows}} class will calculate the optimal width that each row in the
> {{ResultSet}} should be displayed with
> * However, this width is only relevant / used by {{TableOutputFormat}}
> We should modify the logic so that the width is only calculated if
> {{TableOutputFormat}} is used. This will save CPU cycles when printing
> records out to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14169) Beeline Row printing should only calculate the width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14169 started by Sahil Takiar.
---
> Beeline Row printing should only calculate the width if TableOutputFormat is
> used
> -
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> * The {{Rows}} class will calculate the optimal width that each row in the
> {{ResultSet}} should be displayed with
> * However, this width is only relevant / used by {{TableOutputFormat}}
> We should modify the logic so that the width is only calculated if
> {{TableOutputFormat}} is used. This will save CPU cycles when printing
> records out to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14169) Beeline Row printing should only calculate the width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14169:
Attachment: HIVE-14169.1.patch
> Beeline Row printing should only calculate the width if TableOutputFormat is
> used
> -
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> * The {{Rows}} class will calculate the optimal width that each row in the
> {{ResultSet}} should be displayed with
> * However, this width is only relevant / used by {{TableOutputFormat}}
> We should modify the logic so that the width is only calculated if
> {{TableOutputFormat}} is used. This will save CPU cycles when printing
> records out to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-12393) Simplify ColumnPruner when CBO optimizes the query
[ https://issues.apache.org/jira/browse/HIVE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-12393. Resolution: Duplicate > Simplify ColumnPruner when CBO optimizes the query > -- > > Key: HIVE-12393 > URL: https://issues.apache.org/jira/browse/HIVE-12393 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > The plan for any given query optimized by CBO will always contain a Project > operator on top of the TS that prunes that columns that are not needed. > Thus, there is no need for Hive optimizer to traverse the whole plan to check > which columns can be pruned. In fact, Hive ColumnPruner optimizer only needs > to match TS operators when CBO optimized the plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-13392) disable speculative execution for ACID Compactor
[
https://issues.apache.org/jira/browse/HIVE-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reopened HIVE-13392:
---
this needs to go into 2.1.1 as well
> disable speculative execution for ACID Compactor
>
>
> Key: HIVE-13392
> URL: https://issues.apache.org/jira/browse/HIVE-13392
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.2.0
>
> Attachments: HIVE-13392.2.patch, HIVE-13392.3.patch,
> HIVE-13392.4.patch, HIVE-13392.patch
>
>
> https://developer.yahoo.com/hadoop/tutorial/module4.html
> Speculative execution is enabled by default. You can disable speculative
> execution for the mappers and reducers by setting the
> mapred.map.tasks.speculative.execution and
> mapred.reduce.tasks.speculative.execution JobConf options to false,
> respectively.
> CompactorMR is currently not set up to handle speculative execution and may
> lead to something like
> {code}
> 2016-02-08 22:56:38,256 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
> Failed to CREATE_FILE
> /apps/hive/warehouse/service_logs_v2/ds=2016-01-20/_tmp_6cf08b9f-c2e2-4182-bc81-e032801b147f/base_13858600/bucket_4
> for DFSClient_attempt_1454628390210_27756_m_01_1_131224698_1 on
> 172.18.129.12 because this file lease is currently owned by
> DFSClient_attempt_1454628390210_27756_m_01_0_-2027182532_1 on
> 172.18.129.18
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2451)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:688)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> {code}
> Short term: disable speculative execution for this job
> Longer term perhaps make each task write to dir with UUID...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple expr e.g. WHERE 'a'
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Fix Version/s: 1.3.0 > Hive PPD might remove predicates when they are defined as a simple expr e.g. > WHERE 'a' > -- > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple expr e.g. WHERE 'a'
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Pushed to master, branch-2.1. Thanks for reviewing [~ashutoshc]! > Hive PPD might remove predicates when they are defined as a simple expr e.g. > WHERE 'a' > -- > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. WHERE pred
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Summary: Hive PPD might remove predicates when they are defined as a simple node e.g. WHERE pred (was: Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred") > Hive PPD might remove predicates when they are defined as a simple node e.g. > WHERE pred > --- > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple expr e.g. WHERE 'a'
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14147: --- Summary: Hive PPD might remove predicates when they are defined as a simple expr e.g. WHERE 'a' (was: Hive PPD might remove predicates when they are defined as a simple node e.g. WHERE pred) > Hive PPD might remove predicates when they are defined as a simple expr e.g. > WHERE 'a' > -- > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11402) HS2 - add an option to disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11402: Summary: HS2 - add an option to disallow parallel query execution within a single Session (was: HS2 - disallow parallel query execution within a single Session) > HS2 - add an option to disallow parallel query execution within a single > Session > > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.02.patch, > HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14188) LLAPIF: wrong user field is used from the token
[ https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14188: Attachment: HIVE-14188.patch Same patch... > LLAPIF: wrong user field is used from the token > --- > > Key: HIVE-14188 > URL: https://issues.apache.org/jira/browse/HIVE-14188 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14188.patch, HIVE-14188.patch > > > realUser is not usually set in all cases for delegation tokens; we should use > the owner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14189) backport HIVE-13945 to branch-1
[ https://issues.apache.org/jira/browse/HIVE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14189: Attachment: HIVE-14189.02-branch-1.patch Same patch... > backport HIVE-13945 to branch-1 > --- > > Key: HIVE-14189 > URL: https://issues.apache.org/jira/browse/HIVE-14189 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14189.01-branch-1.patch, > HIVE-14189.02-branch-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: HIVE-14111.05.patch The same patch for HiveQA > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14172) LLAP: force evict blocks by size to handle memory fragmentation
[ https://issues.apache.org/jira/browse/HIVE-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14172: Attachment: HIVE-14172.01.patch The same patch for HiveQA > LLAP: force evict blocks by size to handle memory fragmentation > --- > > Key: HIVE-14172 > URL: https://issues.apache.org/jira/browse/HIVE-14172 > Project: Hive > Issue Type: Bug >Reporter: Nita Dembla >Assignee: Sergey Shelukhin > Attachments: HIVE-14172.01.patch, HIVE-14172.patch > > > In the long run, we should replace buddy allocator with a better scheme. For > now do a workaround for fragmentation that cannot be easily resolved. It's > still not perfect but works for practical ORC cases, where we have the > default size and smaller blocks, rather than large allocations having trouble. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-14089:
---
Assignee: Sergey Shelukhin (was: Prasanth Jayachandran)
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.WIP.2.patch, HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[
https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368241#comment-15368241
]
Hive QA commented on HIVE-14146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816835/HIVE-14146.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10293 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/428/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/428/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-428/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816835 - PreCommit-HIVE-MASTER-Build
> Column comments with "\n" character "corrupts" table metadata
> -
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch,
> HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.patch
>
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an
> individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +---++---+--+
> | col_name | data_type |comment|
> +---++---+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +---++---+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14173) NPE was thrown after enabling directsql in the middle of session
[
https://issues.apache.org/jira/browse/HIVE-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-14173:
---
Attachment: HIVE-14173.patch
To kickoff a new precommit build
> NPE was thrown after enabling directsql in the middle of session
>
>
> Key: HIVE-14173
> URL: https://issues.apache.org/jira/browse/HIVE-14173
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-14173.patch, HIVE-14173.patch, HIVE-14173.patch
>
>
> hive.metastore.try.direct.sql is initially set to false in HMS hive-site.xml,
> then changed to true using set metaconf command in the middle of a session,
> running a query will be thrown NPE with error message is as following:
> {code}
> 2016-07-06T17:44:41,489 ERROR [pool-5-thread-2]: metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invokeInternal(192)) -
> MetaException(message:java.lang.NullPointerException)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5741)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rethrowException(HiveMetaStore.java:4771)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy18.get_partitions_by_expr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12048)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12032)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.(ObjectStore.java:2667)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetListHelper.(ObjectStore.java:2825)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$4.(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:2400)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy17.getPartitionsByExpr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4749)
> ... 20 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368188#comment-15368188 ] Aihua Xu commented on HIVE-11402: - Just worried how useful that configuration will be since mostly the users will use mixed HUE, beeline, etc, I guess. So we may have to keep the default. But seems this is safe to add. +1. > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.02.patch, > HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14147) Hive PPD might remove predicates when they are defined as a simple node e.g. "WHERE pred"
[ https://issues.apache.org/jira/browse/HIVE-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368178#comment-15368178 ] Ashutosh Chauhan commented on HIVE-14147: - +1 > Hive PPD might remove predicates when they are defined as a simple node e.g. > "WHERE pred" > - > > Key: HIVE-14147 > URL: https://issues.apache.org/jira/browse/HIVE-14147 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Attachments: HIVE-14147.01.patch, HIVE-14147.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368177#comment-15368177 ] Sergio Peña commented on HIVE-13930: I am investigating about the jar and how to build it. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14184) Adding test for limit pushdown in presence of grouping sets
[ https://issues.apache.org/jira/browse/HIVE-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368166#comment-15368166 ] Ashutosh Chauhan commented on HIVE-14184: - +1 > Adding test for limit pushdown in presence of grouping sets > --- > > Key: HIVE-14184 > URL: https://issues.apache.org/jira/browse/HIVE-14184 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14184.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14137) Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty tables
[
https://issues.apache.org/jira/browse/HIVE-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14137:
Attachment: HIVE-14137.4.patch
> Hive on Spark throws FileAlreadyExistsException for jobs with multiple empty
> tables
> ---
>
> Key: HIVE-14137
> URL: https://issues.apache.org/jira/browse/HIVE-14137
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14137.1.patch, HIVE-14137.2.patch,
> HIVE-14137.3.patch, HIVE-14137.4.patch, HIVE-14137.patch
>
>
> The following queries:
> {code}
> -- Setup
> drop table if exists empty1;
> create table empty1 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> drop table if exists empty2;
> create table empty2 (col1 bigint, col2 bigint) stored as parquet
> tblproperties ('parquet.compress'='snappy');
> drop table if exists empty3;
> create table empty3 (col1 bigint) stored as parquet tblproperties
> ('parquet.compress'='snappy');
> -- All empty HDFS directories.
> -- Fails with [08S01]: Error while processing statement: FAILED: Execution
> Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- Two empty HDFS directories.
> -- Create an empty file in HDFS.
> insert into empty1 select * from empty1 where false;
> -- Same query fails with [08S01]: Error while processing statement: FAILED:
> Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> -- One empty HDFS directory.
> -- Create an empty file in HDFS.
> insert into empty2 select * from empty2 where false;
> -- Same query succeeds.
> select empty1.col1
> from empty1
> inner join empty2
> on empty2.col1 = empty1.col1
> inner join empty3
> on empty3.col1 = empty2.col2;
> {code}
> Will result in the following exception:
> {code}
> org.apache.hadoop.fs.FileAlreadyExistsException:
> /tmp/hive/hive/1f3837aa-9407-4780-92b1-42a66d205139/hive_2016-06-24_15-45-23_206_79177714958655528-2/-mr-10004/0/emptyFile
> for client 172.26.14.151 already exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2561)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:593)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1902)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1738)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1663)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:405)
>
[jira] [Work started] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14170 started by Sahil Takiar.
---
> Beeline IncrementalRows should buffer rows and incrementally re-calculate
> width if TableOutputFormat is used
>
>
> Key: HIVE-14170
> URL: https://issues.apache.org/jira/browse/HIVE-14170
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14170.1.patch
>
>
> If {{--incremental}} is specified in Beeline, rows are meant to be printed
> out immediately. However, if {{TableOutputFormat}} is used with this option
> the formatting can look really off.
> The reason is that {{IncrementalRows}} does not do a global calculation of
> the optimal width size for {{TableOutputFormat}} (it can't because it only
> sees one row at a time). The output of {{BufferedRows}} looks much better
> because it can do this global calculation.
> If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width
> should be re-calculated every "x" rows ("x" can be configurable and by
> default it can be 1000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14170:
Status: Patch Available (was: In Progress)
> Beeline IncrementalRows should buffer rows and incrementally re-calculate
> width if TableOutputFormat is used
>
>
> Key: HIVE-14170
> URL: https://issues.apache.org/jira/browse/HIVE-14170
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14170.1.patch
>
>
> If {{--incremental}} is specified in Beeline, rows are meant to be printed
> out immediately. However, if {{TableOutputFormat}} is used with this option
> the formatting can look really off.
> The reason is that {{IncrementalRows}} does not do a global calculation of
> the optimal width size for {{TableOutputFormat}} (it can't because it only
> sees one row at a time). The output of {{BufferedRows}} looks much better
> because it can do this global calculation.
> If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width
> should be re-calculated every "x" rows ("x" can be configurable and by
> default it can be 1000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368157#comment-15368157 ] Ashutosh Chauhan commented on HIVE-13930: - [~sershe] TestEncrytedHDFSCliDriver failures do look legit. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14196) Exclude LLAP IO complex types test
[ https://issues.apache.org/jira/browse/HIVE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368151#comment-15368151 ] Prasanth Jayachandran commented on HIVE-14196: -- Yeah. That's better. Will put up patch. > Exclude LLAP IO complex types test > -- > > Key: HIVE-14196 > URL: https://issues.apache.org/jira/browse/HIVE-14196 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Let's exclude vector_complex_* tests added for llap which is currently broken > and fails in all test runs. We can re-enable it with HIVE-14089 patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14158) deal with derived column names
[ https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14158: --- Attachment: (was: HIVE-14158.03.patch) > deal with derived column names > -- > > Key: HIVE-14158 > URL: https://issues.apache.org/jira/browse/HIVE-14158 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch, > HIVE-14158.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14195) HiveMetaStoreClient getFunction() does not throw NoSuchObjectException
[ https://issues.apache.org/jira/browse/HIVE-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368139#comment-15368139 ] Sergey Shelukhin commented on HIVE-14195: - +1 pending tests > HiveMetaStoreClient getFunction() does not throw NoSuchObjectException > -- > > Key: HIVE-14195 > URL: https://issues.apache.org/jira/browse/HIVE-14195 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14195.patch > > > HiveMetaStoreClient getFunction(dbName, funcName) does not throw > NoSuchObjectException when no function with funcName exists in the db. > Instead, I need to search the MetaException message for > 'NoSuchObjectException'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14196) Exclude LLAP IO complex types test
[ https://issues.apache.org/jira/browse/HIVE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368135#comment-15368135 ] Sergey Shelukhin commented on HIVE-14196: - Rather we should disable IO for complex type until it's fixed. > Exclude LLAP IO complex types test > -- > > Key: HIVE-14196 > URL: https://issues.apache.org/jira/browse/HIVE-14196 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Let's exclude vector_complex_* tests added for llap which is currently broken > and fails in all test runs. We can re-enable it with HIVE-14089 patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14178) Hive::needsToCopy should reuse FileUtils::equalsFileSystem
[
https://issues.apache.org/jira/browse/HIVE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14178:
Status: Patch Available (was: Open)
> Hive::needsToCopy should reuse FileUtils::equalsFileSystem
> --
>
> Key: HIVE-14178
> URL: https://issues.apache.org/jira/browse/HIVE-14178
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 1.2.1, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14178.1.patch, HIVE-14178.2.patch
>
>
> Clear bug triggered from missing FS checks in Hive.java
> {code}
> //Check if different FileSystems
> if (!srcFs.getClass().equals(destFs.getClass()))
> {
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14178) Hive::needsToCopy should reuse FileUtils::equalsFileSystem
[
https://issues.apache.org/jira/browse/HIVE-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14178:
Attachment: HIVE-14178.2.patch
> Hive::needsToCopy should reuse FileUtils::equalsFileSystem
> --
>
> Key: HIVE-14178
> URL: https://issues.apache.org/jira/browse/HIVE-14178
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1, 2.1.0, 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14178.1.patch, HIVE-14178.2.patch
>
>
> Clear bug triggered from missing FS checks in Hive.java
> {code}
> //Check if different FileSystems
> if (!srcFs.getClass().equals(destFs.getClass()))
> {
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14158) deal with derived column names
[ https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14158: --- Status: Patch Available (was: Open) > deal with derived column names > -- > > Key: HIVE-14158 > URL: https://issues.apache.org/jira/browse/HIVE-14158 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch, > HIVE-14158.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
