[jira] [Commented] (HIVE-14092) Kryo exception when deserializing VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349056#comment-15349056
]
Hive QA commented on HIVE-14092:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813127/HIVE-14092.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/259/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/259/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813127 - PreCommit-HIVE-MASTER-Build
> Kryo exception when deserializing VectorFileSinkOperator
>
>
> Key: HIVE-14092
> URL: https://issues.apache.org/jira/browse/HIVE-14092
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14092.1.patch
>
>
> Following exception is thrown for queries using VectorFileSinkOperator
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ... 46 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
> ... 62 more
> Caused by: java.lang.StackOverflowError
> at
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Affects Version/s: 2.0.0
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Fix Version/s: 2.2.0
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13725: -- Assignee: Vaibhav Gumashta (was: Eugene Koifman) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch, addendum.txt > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13725: -- Attachment: addendum.txt [~vgumashta] please look at addendum.txt. Could you incorporate it into your next patch? It adds a test and fixes exception propagation in Heartbeater. > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch, addendum.txt > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-13725: - Assignee: Eugene Koifman (was: Vaibhav Gumashta) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Attachment: HIVE-13982.7.patch
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Attachment: (was: HIVE-13982.7.patch)
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349033#comment-15349033
]
Hive QA commented on HIVE-14070:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813126/HIVE-14070.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10263 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/258/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/258/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813126 - PreCommit-HIVE-MASTER-Build
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349031#comment-15349031
]
Sergey Shelukhin commented on HIVE-14089:
-
Seems like column ID handling is incorrect; it doesn't account for nested
types, but also conflates table vs orc columns. I'll fix next week
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[
https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349028#comment-15349028
]
Eugene Koifman commented on HIVE-13725:
---
TransactionBatchImpl has "this.heartbeaterMSClient = msClient;" this looks
wrong
In DbTxnManager, instead of
bq. private static int heartbeaterMSClientCount = 0;
bq. private static Object heartBeaterClientCountLock = new Object();
can heartbeaterMSClientCount be AtomicInteger?
I also think that doing "LOG.warn("The number of hearbeater metastore client
has ex..." from synchronized block is suboptimal.
I would also modify the message to include both current client count and thread
count. I think it would be more useful.
> ACID: Streaming API should synchronize calls when multiple threads use the
> same endpoint
>
>
> Key: HIVE-13725
> URL: https://issues.apache.org/jira/browse/HIVE-13725
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Metastore, Transactions
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
> Labels: ACID, Streaming
> Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch,
> HIVE-13725.3.patch, HIVE-13725.4.patch
>
>
> Currently, the streaming endpoint creates a metastore client which gets used
> for RPC. The client itself is not internally thread safe. Therefore, the API
> methods should provide the relevant synchronization so that the methods can
> be called from different threads. A sample use case is as follows:
> 1. Thread 1 creates a streaming endpoint and opens a txn batch.
> 2. Thread 2 heartbeats the txn batch.
> With the current impl, this can result in an "out of sequence response",
> since the response of the calls in thread1 might end up going to thread2 and
> vice-versa.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349023#comment-15349023
]
Ashutosh Chauhan commented on HIVE-13982:
-
+1 pending tests.
Lets create follow-up jira for 2 identified issues.
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-13725: --- Assignee: Vaibhav Gumashta (was: Eugene Koifman) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13725: Attachment: HIVE-13725.4.patch > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch, HIVE-13725.4.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348973#comment-15348973
]
Hive QA commented on HIVE-14085:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813125/HIVE-14085.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/257/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/257/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813125 - PreCommit-HIVE-MASTER-Build
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348967#comment-15348967
]
Jesus Camacho Rodriguez edited comment on HIVE-14027 at 6/25/16 12:56 AM:
--
It seems to be a problem in the execution side, not in the planner.
If we set hive.auto.convert.join to false, we execute a MergeJoin and we get
correct results.
was (Author: jcamachorodriguez):
It seems to be a problem in the execution side, not in the planner.
If we set hive.auto.convert.join to false, we get correct results.
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348967#comment-15348967
]
Jesus Camacho Rodriguez commented on HIVE-14027:
It seems to be a problem in the execution side, not in the planner.
If we set hive.auto.convert.join to false, we get correct results.
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348941#comment-15348941 ] Jason Dere commented on HIVE-14093: --- Not too sure how to get that to work here - thing is we would need to setup the CountdownLatch with the correct count of pending writes, but one or more of these writes could finish while we are in the process of setting up the CountdownLatch. Would also need extra logic in the writeListener to see if the stream is currently waiting to close, and if so decrement the latch. Would still probably need some kind of synchronization around both of these parts. I think the solution in the patch is relatively simple, especially since we can re-use it for waiting when there are too many pending writes. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348937#comment-15348937 ] Vaibhav Gumashta commented on HIVE-7224: [~stakiar] Sure, thanks for taking it up. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348904#comment-15348904 ] Prasanth Jayachandran commented on HIVE-14093: -- Will using CountDownLatch make this easier since you already know how many writes are pending? > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348886#comment-15348886 ] Sahil Takiar commented on HIVE-7224: [~vgumashta] I wanted to help moving this ticket forward, it seems there were some test failures related to this change. Is it ok if I assign this ticket to myself and start working on resolving the test failures? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-7224: --- Attachment: HIVE-7224.3.patch > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348880#comment-15348880 ] Sahil Takiar commented on HIVE-7224: Attaching a new, re-based patch in order to trigger build and see if the test failures are still present. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Status: Patch Available (was: In Progress)
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Attachment: HIVE-13982.7.patch
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13982 started by Jesus Camacho Rodriguez.
--
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Status: Open (was: Patch Available)
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch,
> HIVE-13982.7.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14078) LLAP input split should get task attempt number from conf if available
[ https://issues.apache.org/jira/browse/HIVE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348874#comment-15348874 ] Prasanth Jayachandran commented on HIVE-14078: -- You can use MRInputHelpers.getTaskAttemptIndex() to get attempt id. Other than that the patch looks good to me, +1 > LLAP input split should get task attempt number from conf if available > -- > > Key: HIVE-14078 > URL: https://issues.apache.org/jira/browse/HIVE-14078 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch > > > Currently the attempt number is hard-coded to 0. If the split is being > fetched as part of a hadoop job we can get the task attempt ID from the conf > if it has been set, and use the attempt number from that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348873#comment-15348873 ] Prasanth Jayachandran commented on HIVE-14093: -- Will rever this comment and +1. Will move it over to HIVE-14078 > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Attachment: HIVE-14093.1.patch Attaching correct patch. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-7224: -- Assignee: Sahil Takiar (was: Vaibhav Gumashta) > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348871#comment-15348871 ] Jason Dere commented on HIVE-14093: --- Oops, attached the wrong patch to this Jira. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Attachment: (was: HIVE-14078.1.patch) > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348861#comment-15348861 ] Rahul Sharma edited comment on HIVE-13966 at 6/24/16 11:25 PM: --- Repost: Can a listener be registered with multiple properties, for ex: can DBNotification listener be part of hive.metastore.synchronous.event.listeners(proposed above) and hive.metastore.event.listeners. Should this be a check from the code to prevent duplicated entries in the notification log or this be checked by the user ? was (Author: rahul9269): Repost: Can a listener be registered with multiple properties, for ex: can DBNotification listener be part of hive.metastore.synchronous.event.listeners(proposed above) and hive.metastore.event.listeners. Should this be a check from the code to prevent duplicated entries in the notification log or this be checked by the user/CM ? > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348864#comment-15348864 ] Prasanth Jayachandran commented on HIVE-14093: -- You can use MRInputHelpers.getTaskAttemptIndex() to get attempt id. Other than that the patch looks good to me, +1 > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348861#comment-15348861 ] Rahul Sharma commented on HIVE-13966: - Repost: Can a listener be registered with multiple properties, for ex: can DBNotification listener be part of hive.metastore.synchronous.event.listeners(proposed above) and hive.metastore.event.listeners. Should this be a check from the code to prevent duplicated entries in the notification log or this be checked by the user/CM ? > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Status: Patch Available (was: Open) > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Attachment: HIVE-14078.1.patch Keeping a count of the pending writes, and setting the Future listener for the write calls notify the close() method when it is ok to finally close the channel. I've also reworked the changes from HIVE-13956 to use the same waiting mechanism. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14091) some errors are not propagated to LLAP external clients
[
https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348837#comment-15348837
]
Hive QA commented on HIVE-14091:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813121/HIVE-14091.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/256/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/256/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-256/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813121 - PreCommit-HIVE-MASTER-Build
> some errors are not propagated to LLAP external clients
> ---
>
> Key: HIVE-14091
> URL: https://issues.apache.org/jira/browse/HIVE-14091
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14091.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[
https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HIVE-13966:
Comment: was deleted
(was: Interesting idea. I suppose we could check at runtime which of the two
interfaces each configured extension class implemented (either
{{TransactionalMetastoreEventListener}}, or the existing
{{DBNotificationListener}} interface) and treat it appropriately.)
> DbNotificationListener: can loose DDL operation notifications
> -
>
> Key: HIVE-13966
> URL: https://issues.apache.org/jira/browse/HIVE-13966
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Nachiket Vaidya
>Assignee: Rahul Sharma
>Priority: Critical
>
> The code for each API in HiveMetaStore.java is like this:
> 1. openTransaction()
> 2. -- operation--
> 3. commit() or rollback() based on result of the operation.
> 4. add entry to notification log (unconditionally)
> If the operation is failed (in step 2), we still add entry to notification
> log. Found this issue in testing.
> It is still ok as this is the case of false positive.
> If the operation is successful and adding to notification log failed, the
> user will get an MetaException. It will not rollback the operation, as it is
> already committed. We need to handle this case so that we will not have false
> negatives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14092) Kryo exception when deserializing VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348770#comment-15348770
]
Ashutosh Chauhan commented on HIVE-14092:
-
At some point we need to determine a way to auto register classes.
+1
> Kryo exception when deserializing VectorFileSinkOperator
>
>
> Key: HIVE-14092
> URL: https://issues.apache.org/jira/browse/HIVE-14092
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14092.1.patch
>
>
> Following exception is thrown for queries using VectorFileSinkOperator
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ... 46 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
> ... 62 more
> Caused by: java.lang.StackOverflowError
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.get(HashMap.java:556)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:61)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348681#comment-15348681
]
Vihang Karajgaonkar commented on HIVE-14085:
Review board link : https://reviews.apache.org/r/49215
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14092) Kryo exception when deserializing VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14092:
-
Attachment: HIVE-14092.1.patch
> Kryo exception when deserializing VectorFileSinkOperator
>
>
> Key: HIVE-14092
> URL: https://issues.apache.org/jira/browse/HIVE-14092
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14092.1.patch
>
>
> Following exception is thrown for queries using VectorFileSinkOperator
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ... 46 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
> ... 62 more
> Caused by: java.lang.StackOverflowError
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.get(HashMap.java:556)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:61)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14092) Kryo exception when deserializing VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14092:
-
Status: Patch Available (was: Open)
> Kryo exception when deserializing VectorFileSinkOperator
>
>
> Key: HIVE-14092
> URL: https://issues.apache.org/jira/browse/HIVE-14092
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14092.1.patch
>
>
> Following exception is thrown for queries using VectorFileSinkOperator
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ... 46 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
> ... 62 more
> Caused by: java.lang.StackOverflowError
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.get(HashMap.java:556)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:61)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348665#comment-15348665
]
Vihang Karajgaonkar commented on HIVE-14085:
[~spena] Can you take a look at the patch? Thanks!
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-14085:
---
Affects Version/s: (was: 2.1.0)
Status: Patch Available (was: Open)
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Attachment: (was: HIVE-14070.03.patch)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Open (was: Patch Available)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Attachment: HIVE-14070.03.patch
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-14085:
---
Attachment: HIVE-14085.1.patch
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Patch Available (was: Open)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348649#comment-15348649 ] Abdullah Yousufi commented on HIVE-13964: - Would something like this work? - --property-file The file to read connection properties (url, driver, user, password) from. Usage: beeline --property-file props > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Labels: Docs > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, > HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348633#comment-15348633
]
Sergey Shelukhin commented on HIVE-14089:
-
It appears that the patch allows the complex types (previously blocked by
Vectorizer) to enter LLAP IO; previously, those paths were never executed. Now
they are and hence the errors
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Summary: complex type support in LLAP IO is broken (was: Fix MiniLlap
test failures in master)
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14091) some errors are not propagated to LLAP external clients
[
https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348611#comment-15348611
]
Sergey Shelukhin edited comment on HIVE-14091 at 6/24/16 9:00 PM:
--
I added an exception to the end-to-end test. Without the patch, the test times
out. With the patch, the error looks like this:
{noformat}
ests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.07 sec <<<
FAILURE! - in org.apache.hive.jdbc.TestJdbcWithMiniLlap
testLlapInputFormatEndToEnd(org.apache.hive.jdbc.TestJdbcWithMiniLlap) Time
elapsed: 5.089 sec <<< ERROR!
java.io.IOException: Received reader event error: Received an error for task ID
attempt_8772768970312654090_0001_0_00_00_0: Error while running task (
failure ) :
attempt_8772768970312654090_0001_0_00_00_0:java.lang.RuntimeException:
java.lang.Exception: boom!
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: boom!
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:497)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
... 14 more
at
org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:163)
at
org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:47)
at
org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:107)
at
org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:56)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlap.getLlapIFRowCount(TestJdbcWithMiniLlap.java:201)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd(TestJdbcWithMiniLlap.java:223)
{noformat}
was (Author: sershe):
I added some exception to the end-to-end test. Without the patch, the test
times out. With the patch, the error looks like this:
{noformat}
ests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.07 sec <<<
FAILURE! - in org.apache.hive.jdbc.TestJdbcWithMiniLlap
testLlapInputFormatEndToEnd(org.apache.hive.jdbc.TestJdbcWithMiniLlap) Time
elapsed: 5.089 sec <<< ERROR!
java.io.IOException: Received reader event error: Received an error for task ID
attempt_8772768970312654090_0001_0_00_00_0: Error while running task (
failure ) :
attempt_8772768970312654090_0001_0_00_00_0:java.lang.RuntimeException:
java.lang.Exception: boom!
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[jira] [Commented] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348613#comment-15348613 ] Sergey Shelukhin commented on HIVE-14091: - [~jdere] [~sseth] can you take a look? > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14091) some errors are not propagated to LLAP external clients
[
https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14091:
Attachment: HIVE-14091.patch
I added some exception to the end-to-end test. Without the patch, the test
times out. With the patch, the error looks like this:
{noformat}
ests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.07 sec <<<
FAILURE! - in org.apache.hive.jdbc.TestJdbcWithMiniLlap
testLlapInputFormatEndToEnd(org.apache.hive.jdbc.TestJdbcWithMiniLlap) Time
elapsed: 5.089 sec <<< ERROR!
java.io.IOException: Received reader event error: Received an error for task ID
attempt_8772768970312654090_0001_0_00_00_0: Error while running task (
failure ) :
attempt_8772768970312654090_0001_0_00_00_0:java.lang.RuntimeException:
java.lang.Exception: boom!
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: boom!
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:497)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:171)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
... 14 more
at
org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:163)
at
org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:47)
at
org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:107)
at
org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:56)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlap.getLlapIFRowCount(TestJdbcWithMiniLlap.java:201)
at
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd(TestJdbcWithMiniLlap.java:223)
{noformat}
> some errors are not propagated to LLAP external clients
> ---
>
> Key: HIVE-14091
> URL: https://issues.apache.org/jira/browse/HIVE-14091
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14091.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14091: Status: Patch Available (was: Open) > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14089) Fix MiniLlap test failures in master
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-14089:
---
Assignee: Sergey Shelukhin
> Fix MiniLlap test failures in master
>
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14068) make more effort to find hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14068: Resolution: Implemented Status: Resolved (was: Patch Available) This was accidentally committed as part of HIVE-14055 (I had too many branches/patches in the same git clone...). Tests have passed before (and since the commit) > make more effort to find hive-site.xml > -- > > Key: HIVE-14068 > URL: https://issues.apache.org/jira/browse/HIVE-14068 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14068.01.patch, HIVE-14068.02.patch, > HIVE-14068.patch > > > It pretty much doesn't make sense to run Hive w/o the config, so we should > make more effort to find one if it's missing on the classpath, or the > classloader does not return it for some reason (e.g. classloader ignores some > permission issues; explicitly looking for the file may expose them better) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14068) make more effort to find hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14068: Fix Version/s: 2.2.0 > make more effort to find hive-site.xml > -- > > Key: HIVE-14068 > URL: https://issues.apache.org/jira/browse/HIVE-14068 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-14068.01.patch, HIVE-14068.02.patch, > HIVE-14068.patch > > > It pretty much doesn't make sense to run Hive w/o the config, so we should > make more effort to find one if it's missing on the classpath, or the > classloader does not return it for some reason (e.g. classloader ignores some > permission issues; explicitly looking for the file may expose them better) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-13278: --- Assignee: Sahil Takiar > Many redundant 'File not found' messages appeared in container log during > query execution with Hive on Spark > > > Key: HIVE-13278 > URL: https://issues.apache.org/jira/browse/HIVE-13278 > Project: Hive > Issue Type: Bug > Environment: Hive on Spark engine > Found based on : > Apache Hive 2.0.0 > Apache Spark 1.6.0 >Reporter: Xin Hao >Assignee: Sahil Takiar >Priority: Minor > > Many redundant 'File not found' messages appeared in container log during > query execution with Hive on Spark. > Certainly, it doesn't prevent the query from running successfully. So mark it > as Minor currently. > Error message example: > 16/03/14 01:45:06 INFO exec.Utilities: File not found: File does not exist: > /tmp/hive/hadoop/2d378538-f5d3-493c-9276-c62dd6634fb4/hive_2016-03-14_01-44-16_835_623058724409492515-6/-mr-10010/0a6d0cae-1eb3-448c-883b-590b3b198a73/reduce.xml > at > org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) > at > org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:565) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13278 started by Sahil Takiar. --- > Many redundant 'File not found' messages appeared in container log during > query execution with Hive on Spark > > > Key: HIVE-13278 > URL: https://issues.apache.org/jira/browse/HIVE-13278 > Project: Hive > Issue Type: Bug > Environment: Hive on Spark engine > Found based on : > Apache Hive 2.0.0 > Apache Spark 1.6.0 >Reporter: Xin Hao >Assignee: Sahil Takiar >Priority: Minor > > Many redundant 'File not found' messages appeared in container log during > query execution with Hive on Spark. > Certainly, it doesn't prevent the query from running successfully. So mark it > as Minor currently. > Error message example: > 16/03/14 01:45:06 INFO exec.Utilities: File not found: File does not exist: > /tmp/hive/hadoop/2d378538-f5d3-493c-9276-c62dd6634fb4/hive_2016-03-14_01-44-16_835_623058724409492515-6/-mr-10010/0a6d0cae-1eb3-448c-883b-590b3b198a73/reduce.xml > at > org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) > at > org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:565) > at > org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14090 started by Sahil Takiar.
---
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14090.patch
>
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:707)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:702)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:702)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: Attempt
> to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> org.datanucleus.exceptions.NucleusUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum
[jira] [Updated] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14090:
Attachment: HIVE-14090.patch
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14090.patch
>
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:707)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:702)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:702)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: Attempt
> to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> org.datanucleus.exceptions.NucleusUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that
[jira] [Commented] (HIVE-14088) Fix list_bucket test failures in master
[
https://issues.apache.org/jira/browse/HIVE-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348527#comment-15348527
]
Sergio Peña commented on HIVE-14088:
Hey [~prasanth_j].
The ptest environment is running with JDK8. Search for {{java1.8.0_25}} here:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/240/consoleText
What you saw on the Jenkins environment variables is the version used to build
PTest client only. Ptest should not affect the execution of tests. The
configuration for the JDK used for tests are in the PTest server properties
file:
{noformat}
$ cat trunk-mr2.properties
...
javaHome = /usr/java/jdk1.8.0_25
javaHomeForTests = /usr/java/jdk1.8.0_25
...
{noformat}
> Fix list_bucket test failures in master
> ---
>
> Key: HIVE-14088
> URL: https://issues.apache.org/jira/browse/HIVE-14088
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>
> The failures are related to diff showing jdk7 vs jdk8 hashmap ordering
> differences. Since we have already moved our test infra to jdk8 it will be
> safe to regen the golden file with jdk8.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14072) QueryIds reused across different queries
[
https://issues.apache.org/jira/browse/HIVE-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348518#comment-15348518
]
Hive QA commented on HIVE-14072:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812914/HIVE-14072.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/249/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/249/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-249/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812914 - PreCommit-HIVE-MASTER-Build
> QueryIds reused across different queries
>
>
> Key: HIVE-14072
> URL: https://issues.apache.org/jira/browse/HIVE-14072
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-14072.patch
>
>
> While testing HIVE-14023, and running TestMiniLlapCluster - query ids were
> re-uesd for the entire init scripts. 30+ different queries - same queryId,
> new Tez dag submission, for different queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14090:
Summary: JDOExceptions thrown by the Metastore have their full stack trace
returned to clients (was: JDOException's thrown by the Metastore have their
full stack trace returned to clients)
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:707)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:702)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:702)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: Attempt
> to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> org.datanucleus.exceptions.NucleusUserException: Attempt to store value
>
[jira] [Comment Edited] (HIVE-14090) JDOException's thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348497#comment-15348497
]
Sahil Takiar edited comment on HIVE-14090 at 6/24/16 4:17 PM:
--
The root cause of this issue is in {{RetryingHMSHandler.invokeInternal}}
method, specifically the following lines:
{code}
if (retryCount >= retryLimit) {
LOG.error("HMSHandler Fatal error: " +
ExceptionUtils.getStackTrace(caughtException));
// Since returning exceptions with a nested "cause" can be a problem in
// Thrift, we are stuffing the stack trace into the message itself.
throw new MetaException(ExceptionUtils.getStackTrace(caughtException));
}
{code}
Some context: this code is run on the Hive Metastore server (clients
communicate with the remote Hive Metastore instance via a Thrift API).
The {{caughtException}} variable is a {{JDOFatalUserException}}. So the entire
stack-trace of {{JDOFatalUserException}} gets stuffed into the message of a
{{MetaException}}, which then gets returned to the client (HiveServer2
usually), via the Hive Metastore Thrift API. This only happens for
{{JDOException}}s due to some other logic in the class, which is why it is not
a very common complaint. So when HiveServer2 sees an exception is thrown by the
Metastore, it displays the exception message to the user, which in this case
contains the full stack-trace of the exception.
Note that the comments in the code mention that the stack-trace is stuffed into
the message due to some problems with Thrift not sending the entire stack-trace
back to the client. This comment was added in HIVE-3400 - there is a
Phabricator review linked to the Hive JIRA (https://reviews.facebook.net/D4791)
which contains a relevant conversation about this choice, I copied it below:
{code}
Actually the RetryingHMSHandler is currently putting the stacktrace into the
"message" of the MetaException and not its "cause". In the Thrift generated
MetaException.java, there is no constructor taking the cause so I stuffed it
into the message. Now that I think of it, I can call initCause(jdoexception)
after constructing the MetaException. Then I can make the change you suggested.
Do you want me to do that?
...
Yes, please do that. Type comparison is much better than regex matching.
...
Unfortunately, things break with that change. In direct db mode things are
fine. But when we go through Thrift, the MetaException received by Hive client
from the Thrift server has null as its cause. Even though the cause is being
set properly on the Thrift side. This might be happening because Thrift does
not "know" about JDOExceptions. Or it may even be a limitation of Thrift
exception handling. Not sure. I'm satisfied with the way it was. The two key
requirements of letting the client know about the JDOException and shipping the
entire stack trace back were being met. What do you think?
...
Yeah, lets keep it this way then. If, thrift can't transport exception trace
correctly. I will suggest to leave a comment there saying why we have to do
regex matching of exception message instead of checking type of exception.
{code}
I modified the code to stop stuffing the stack-trace into the exception
message, and I tested the change locally. The change works. The full exception
is still shipped to the client (HiveServer2), who prints the full exception
stack trace in its log files, and only the exception message is displayed to
the user. I was sure to run the Hive Metastore as a Thrift Server in order to
check if Thrift had any issues transporting the stack trace of the exception,
and I saw no issues.
was (Author: stakiar):
The root cause of this issue is in {{RetryingHMSHandler.invokeInternal}}
method, specifically the following lines:
{code}
if (retryCount >= retryLimit) {
LOG.error("HMSHandler Fatal error: " +
ExceptionUtils.getStackTrace(caughtException));
// Since returning exceptions with a nested "cause" can be a problem in
// Thrift, we are stuffing the stack trace into the message itself.
throw new MetaException(ExceptionUtils.getStackTrace(caughtException));
}
{code}
Some context: this code is run on the Hive Metastore server (clients
communicate with the remote Hive Metastore instance via a Thrift API).
The {{caughtException}} variable is a {{JDOFatalUserException}}. So the entire
stack-trace of {{JDOFatalUserException}} gets stuffed into the message of a
{{MetaException}}, which then gets returned to the client (HiveServer2
usually), via the Hive Metastore Thrift API. This only happens for
{{JDOException}}s due to some other logic in the class, which is why it is not
a very common complaint. So when HiveServer2 sees an exception is thrown by the
Metastore, it displays the exception message to the user, which in this case
contains the full stack-trace of the exception.
Note that the comments in the code
[jira] [Commented] (HIVE-14090) JDOException's thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348497#comment-15348497
]
Sahil Takiar commented on HIVE-14090:
-
The root cause of this issue is in {{RetryingHMSHandler.invokeInternal}}
method, specifically the following lines:
{code}
if (retryCount >= retryLimit) {
LOG.error("HMSHandler Fatal error: " +
ExceptionUtils.getStackTrace(caughtException));
// Since returning exceptions with a nested "cause" can be a problem in
// Thrift, we are stuffing the stack trace into the message itself.
throw new MetaException(ExceptionUtils.getStackTrace(caughtException));
}
{code}
Some context: this code is run on the Hive Metastore server (clients
communicate with the remote Hive Metastore instance via a Thrift API).
The {{caughtException}} variable is a {{JDOFatalUserException}}. So the entire
stack-trace of {{JDOFatalUserException}} gets stuffed into the message of a
{{MetaException}}, which then gets returned to the client (HiveServer2
usually), via the Hive Metastore Thrift API. This only happens for
{{JDOException}}s due to some other logic in the class, which is why it is not
a very common complaint. So when HiveServer2 sees an exception is thrown by the
Metastore, it displays the exception message to the user, which in this case
contains the full stack-trace of the exception.
Note that the comments in the code mention that the stack-trace is stuffed into
the message due to some problems with Thrift not sending the entire stack-trace
back to the client. This comment was added in
https://issues.apache.org/jira/browse/HIVE-3400 - there is a Phabricator review
linked to the Hive JIRA (https://reviews.facebook.net/D4791) which contains a
relevant conversation about this choice, I copied it below:
{code}
Actually the RetryingHMSHandler is currently putting the stacktrace into the
"message" of the MetaException and not its "cause". In the Thrift generated
MetaException.java, there is no constructor taking the cause so I stuffed it
into the message. Now that I think of it, I can call initCause(jdoexception)
after constructing the MetaException. Then I can make the change you suggested.
Do you want me to do that?
...
Yes, please do that. Type comparison is much better than regex matching.
...
Unfortunately, things break with that change. In direct db mode things are
fine. But when we go through Thrift, the MetaException received by Hive client
from the Thrift server has null as its cause. Even though the cause is being
set properly on the Thrift side. This might be happening because Thrift does
not "know" about JDOExceptions. Or it may even be a limitation of Thrift
exception handling. Not sure. I'm satisfied with the way it was. The two key
requirements of letting the client know about the JDOException and shipping the
entire stack trace back were being met. What do you think?
...
Yeah, lets keep it this way then. If, thrift can't transport exception trace
correctly. I will suggest to leave a comment there saying why we have to do
regex matching of exception message instead of checking type of exception.
{code}
I modified the code to stop stuffing the stack-trace into the exception
message, and I tested the change locally. The change works. The full exception
is still shipped to the client (HiveServer2), who prints the full exception
stack trace in its log files, and only the exception message is displayed to
the user. I was sure to run the Hive Metastore as a Thrift Server in order to
check if Thrift had any issues transporting the stack trace of the exception,
and I saw no issues.
> JDOException's thrown by the Metastore have their full stack trace returned
> to clients
> --
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
>
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Open (was: Patch Available)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Patch Available (was: Open)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch,
> HIVE-14070.03.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14083) ALTER INDEX in Tez causes NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348344#comment-15348344
]
Hive QA commented on HIVE-14083:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812911/HIVE-14083.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/248/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/248/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-248/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812911 - PreCommit-HIVE-MASTER-Build
> ALTER INDEX in Tez causes NullPointerException
> --
>
> Key: HIVE-14083
> URL: https://issues.apache.org/jira/browse/HIVE-14083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-14083.patch
>
>
> ALTER INDEX causes a NullPointerException when run under TEZ execution
> engine. Query runs without issue when submitted using MR execution mode.
> To reproduce:
> 1. CREATE INDEX sample_08_index ON TABLE sample_08 (code) AS 'COMPACT' WITH
> DEFERRED REBUILD;
> 2. ALTER INDEX sample_08_index ON sample_08 REBUILD;
> *Stacktrace from Hive 1.2.1*
> {code:java}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1460577396252_0005_1_00, diagnostics=[Task failed,
> taskId=task_1460577396252_0005_1_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:196)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:135)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:101)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:650)
> at
>
[jira] [Commented] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348284#comment-15348284
]
Aihua Xu commented on HIVE-13991:
-
Thanks for the explanation. I can't think of a better way. Maybe add a little
comments when you commit on that line to set mergeIsDirect.
+1.
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13991.1.patch, HIVE-13991.2.patch
>
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348278#comment-15348278
]
Yongzhi Chen commented on HIVE-13991:
-
init(true); combined with genResolvedParseTree() do the step one work, so
assign mergeIsDirect in init(boolean) is fine. You can think it is the same
reason as clear pruned partitions or not.
There are only two places in our code that calls genResolvedParseTree() , list
it may help you understand the code:
{noformat}
if (reAnalyzeAST) {
init(true);
prunedPartitions.clear();
// Assumption: At this point Parse Tree gen & resolution will always
// be true (since we started out that way).
super.genResolvedParseTree(ast, new PlannerContext());
{noformat}
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13991.1.patch, HIVE-13991.2.patch
>
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13631) Support index in HBase Metastore
[
https://issues.apache.org/jira/browse/HIVE-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348197#comment-15348197
]
Hive QA commented on HIVE-13631:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812899/HIVE-13631.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/247/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/247/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-247/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812899 - PreCommit-HIVE-MASTER-Build
> Support index in HBase Metastore
>
>
> Key: HIVE-13631
> URL: https://issues.apache.org/jira/browse/HIVE-13631
> Project: Hive
> Issue Type: Improvement
> Components: HBase Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13631.1-nogen.patch, HIVE-13631.1.patch,
> HIVE-13631.2-nogen.patch, HIVE-13631.2.patch, HIVE-13631.3.patch
>
>
> Currently all index related methods in HBaseStore is not implemented. We need
> to add those missing methods and index support in hbaseimport tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[
https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348117#comment-15348117
]
Hive QA commented on HIVE-14074:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812883/HIVE-14074.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 233 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_windowing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Patch Available (was: Open) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HIVE-11527: Status: Open (was: Patch Available) > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.10.patch, HIVE-11527.11.patch, > HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12656) Turn hive.compute.query.using.stats on by default
[
https://issues.apache.org/jira/browse/HIVE-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348009#comment-15348009
]
Hive QA commented on HIVE-12656:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812879/HIVE-12656.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 10265 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_query_tbl_in_locked_db
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_noscan_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hive.beeline.TestBeeLineWithArgs.testEmbeddedBeelineOutputs
[jira] [Commented] (HIVE-14088) Fix list_bucket test failures in master
[ https://issues.apache.org/jira/browse/HIVE-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347991#comment-15347991 ] Prasanth Jayachandran commented on HIVE-14088: -- This bug is invalid if we have already moved test infra to jdk8. > Fix list_bucket test failures in master > --- > > Key: HIVE-14088 > URL: https://issues.apache.org/jira/browse/HIVE-14088 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran > > The failures are related to diff showing jdk7 vs jdk8 hashmap ordering > differences. Since we have already moved our test infra to jdk8 it will be > safe to regen the golden file with jdk8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14088) Fix list_bucket test failures in master
[ https://issues.apache.org/jira/browse/HIVE-14088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347990#comment-15347990 ] Prasanth Jayachandran commented on HIVE-14088: -- Looking at the test failures from here https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/240/#showFailuresLink I initially thought the golden files are not updated for jdk8. But it looks like the test ran with jdk7 https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/240/injectedEnvVars/ I thought we moved our tests entirely to jdk8. [~ashutoshc]/[~spena] Any idea why the test environment still using jdk7? > Fix list_bucket test failures in master > --- > > Key: HIVE-14088 > URL: https://issues.apache.org/jira/browse/HIVE-14088 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran > > The failures are related to diff showing jdk7 vs jdk8 hashmap ordering > differences. Since we have already moved our test infra to jdk8 it will be > safe to regen the golden file with jdk8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13617) LLAP: support non-vectorized execution in IO
[
https://issues.apache.org/jira/browse/HIVE-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347973#comment-15347973
]
Prasanth Jayachandran commented on HIVE-13617:
--
[~sershe] This patch is causing test failures in master. Following 2 failures
are related
{code}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{code}
Reverting this patch makes these tests pass.
Following exception is thrown
{code}
Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 4
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:346)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:302)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:227)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:148)
at
org.apache.hadoop.hive.ql.io.BatchToRowReader.ensureBatch(BatchToRowReader.java:167)
at
org.apache.hadoop.hive.ql.io.BatchToRowReader.next(BatchToRowReader.java:140)
at
org.apache.hadoop.hive.ql.io.BatchToRowReader.next(BatchToRowReader.java:78)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
at
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:240)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:417)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:209)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:206)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:90)
... 5 more
{code}
Created HIVE-14089 for tracking.
> LLAP: support non-vectorized execution in IO
>
>
> Key: HIVE-13617
> URL: https://issues.apache.org/jira/browse/HIVE-13617
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-13617-wo-11417.patch, HIVE-13617-wo-11417.patch,
> HIVE-13617.01.patch, HIVE-13617.03.patch, HIVE-13617.04.patch,
> HIVE-13617.05.patch, HIVE-13617.06.patch, HIVE-13617.patch, HIVE-13617.patch,
> HIVE-15396-with-oi.patch
>
>
> Two approaches - a separate decoding path, into rows instead of VRBs; or
> decoding VRBs into rows on a higher level (the original LlapInputFormat). I
> think the latter might be better - it's not a hugely important path, and perf
> in non-vectorized case is not the best anyway, so it's better to make do with
> much less new code and architectural disruption.
> Some ORC patches in progress introduce an easy to reuse (or so I hope,
> anyway) VRB-to-row conversion, so we should just use that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14079) Remove file, method and line number from pattern layout
[
https://issues.apache.org/jira/browse/HIVE-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14079:
-
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Test failures are unrelated. Committed to branch-2.1 and master.
> Remove file, method and line number from pattern layout
> ---
>
> Key: HIVE-14079
> URL: https://issues.apache.org/jira/browse/HIVE-14079
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14079.1.patch
>
>
> Using %F%M and %L in pattern layouts need location information which is
> expensive to get and is disabled by default. We should remove them from the
> default layouts. This will avoid creating empty brackets like below
> {code}
> lockmgr.DbTxnManager (:())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13950) Beeline: infinite loop during the connection to as remote hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-13950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347907#comment-15347907
]
Alexandre Linte commented on HIVE-13950:
I think this JIRA is partially solved with Hive 2.1.0. Its also related to the
following ticket:
- [HIVE-12834|https://issues.apache.org/jira/browse/HIVE-12834]
> Beeline: infinite loop during the connection to as remote hiveserver2
> -
>
> Key: HIVE-13950
> URL: https://issues.apache.org/jira/browse/HIVE-13950
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.0, 2.0.1
> Environment: Hadoop 2.7.2, Hive 2.0.1, Tez 0.8.3, Kerberos V
>Reporter: Alexandre Linte
>
> From a hive client machine, I can use beeline to connect to a remote
> Hiveserver2. The connection is secured with Kerberos.
> During the connection process, I have an infinite loop when a username is
> entered. The loop prints "Enter username for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:"
> and only stops if the user pushes down "enter" on its keyboard.
> {noformat}
> [shfs3453@hive-cli01 workspace]$ beeline
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.1-bin/lib/hive-jdbc-2.0.1-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Spark/spark-1.6.1/hive/assembly/spark-assembly-1.4.1-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Beeline version 2.0.1 by Apache Hive
> beeline> !connect
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS
> Connecting to
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS
> Enter username for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
> jdbc:hive2://hiveserver2.bigdata.fr:1/shfs3453;principal=hiveserver2/hiveserver2.bigdata.fr@REALM.KERBEROS:
>
> Enter password for
>
[jira] [Updated] (HIVE-14087) ALTER TABLE table PARTITION requires write permissions
[
https://issues.apache.org/jira/browse/HIVE-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexandre Linte updated HIVE-14087:
---
Component/s: CLI
Beeline
> ALTER TABLE table PARTITION requires write permissions
> --
>
> Key: HIVE-14087
> URL: https://issues.apache.org/jira/browse/HIVE-14087
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI, Hive
>Affects Versions: 2.0.1
> Environment: Hadoop 2.7.2, Hive 2.0.1, Kerberos
>Reporter: Alexandre Linte
>
> I discovered that altering a table will require write permissions when a
> partition is created.
> {noformat}
> hive (shfs3453)> ALTER TABLE external_table ADD IF NOT EXISTS
> PARTITION(address='Idaho') LOCATION
> "hdfs://sandbox/User/shfs3453/WORK/HIVE_TEST";
> ALTER TABLE external_table ADD IF NOT EXISTS PARTITION(address='Idaho')
> LOCATION "hdfs://sandbox/User/shfs3453/WORK/HIVE_TEST"
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.security.AccessControlException: Permission
> denied: user=shfs3453, access=WRITE,
> inode="/User/shfs3453/WORK/HIVE_TEST":shfs3453:shfs3453:dr-xr-x---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1704)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1678)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:8178)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:1911)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1443)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> )
> {noformat}
> This is a strange behavior because nothing is written in
> "/User/shfs3453/WORK/HIVE_TEST".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14068) make more effort to find hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347859#comment-15347859
]
Hive QA commented on HIVE-14068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812691/HIVE-14068.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/242/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/242/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-242/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-242/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3bc615f HIVE-14028: Insert overwrite does not work in HBase
tables: stats is not updated(Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 3bc615f HIVE-14028: Insert overwrite does not work in HBase
tables: stats is not updated(Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812691 - PreCommit-HIVE-MASTER-Build
> make more effort to find hive-site.xml
> --
>
> Key: HIVE-14068
> URL: https://issues.apache.org/jira/browse/HIVE-14068
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14068.01.patch, HIVE-14068.02.patch,
> HIVE-14068.patch
>
>
> It pretty much doesn't make sense to run Hive w/o the config, so we should
> make more effort to find one if it's missing on the classpath, or the
> classloader does not return it for some reason (e.g. classloader ignores some
> permission issues; explicitly looking for the file may expose them better)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13725: Attachment: HIVE-13725.3.patch > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13725: Assignee: Eugene Koifman (was: Vaibhav Gumashta) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Eugene Koifman >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-13725: --- Assignee: Vaibhav Gumashta (was: Eugene Koifman) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: ACID, Streaming > Attachments: HIVE-13725.1.patch, HIVE-13725.2.patch, > HIVE-13725.3.patch > > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
