Re: HIVE building on ARM
Hi Zoltan, Thanks alot for the information, so looks like one possible solution is as you suggest, move the current ARM2 and ARM3 (those two were donate to builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs just as what has been done in current jenkins. I will also ask our team member works on other projects to find out what the status of other projects is. BR, On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich wrote: > Hey, > > There is an effort by the Apache Infra to change the way Jenkins stuff is > organized; a couple months ago Gavin wrote an email about it: > > http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E > The resources for running these jobs are coming from the H0~H21 slaves > which will be migrated to the new jenkins master eventually. > > >> So please > >> suggest a way which direction we can move and can you share some > details > >> about the new ci-hadoop instance. > > Since Hadoop testing is also happening on ARM - I think the best would be > to also migrate the armN slaves and the Hive arm nightly over to the new > ci-hadoop instance. > > On 6/16/20 8:40 AM, Zhenyu Zheng wrote: > > Thanks for the info, I wonder if where does the resource of ci-hadoop and > > hive-test-kube come from? Do they include ARM resources? > > Interesting question; the resources for Hive testing are donated by > Cloudera. > About the ARM workers I think Chinna could provide more details. > ...I've no idea don't know who sponsors the Hxx slaves > > > Can you provide some more information about how the new hive-test-kube is > > running? > It's basically a Jenkins instance which is using kubernetes pods to run > things. > The whole thing is running on a GKE cluster. > While I was working on it I collected stuff needed for it in this repo: > https://github.com/kgyrtkirk/hive-test-kube/ > it should be possible to start a new deployment using that stuff > > cheers, > Zoltan > > > > > BR, > > Kevin Zheng > > > > On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < > > lalamchinnara...@gmail.com> wrote: > > > >> Hi Zoltan, > >> > >> Thanks for the update. > >> > >> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is > >> targeting to run hive tests daily on "arm" slaves, it is using 2 arm > >> slaves. > >> To find any potential issues with "arm" and fix the issues. So please > >> suggest a way which direction we can move and can you share some details > >> about the new ci-hadoop instance. > >> > >> Thanks, > >> Chinna > >> > >> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich wrote: > >> > >>> Hey all, > >>> > >>> In an ticket (INFRA-20416) Gavin asked me if we are completely off > >>> builds.apache.org - when I went over the jobs I've saw that > >>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there > >>> once a day. > >>> > >>> Since builds.apache.org will be shut down in sometime in the future - > we > >>> should move this job to the new ci-hadoop instance or to > hive-test-kube. > >>> The key feature of the job is that it runs the test on the "armX" > slaves; > >>> which are statically configured on b.a.o. > >>> Not sure which way to go - but we will have to move in some direction. > >>> > >>> cheers, > >>> Zoltan > >>> > >>> > >>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: > Hi Chinna, > > Thanks alot for the reply, I uploaded a patch and also a github PR for > https://issues.apache.org/jira/browse/HIVE-21939 . > In the patch, I bumped the protobuf used in standalone-metadata to > 2.6.1 > and added a new profile, this profile will identify > the hardware architecture and if it is Aarch64, it will override the > protobuf group.id and package to com.github.os72 which > includes ARM support. For X86 platform, Hive will still download the > protobuf packages from org.google repo. I think with > this method, we can keep the influence to existing x86 users to the > minimum. I hope this could be a acceptable short-term > solution. > > I've manually tested on my machine and the github PR travis CI test > has > already passed, so the build process is OK, so let's > wait for the full test result from builds.apache.org. > > BR, > > Zhenyu > > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < > >>> lalamchinnara...@gmail.com> > wrote: > > > Hi Zhenyu, > > > > Until HBase dependency resolved, without effecting the existing code > >>> on X86 > > i suggest create a separate profile with "os72" repo. > > > > Down the line we should have common version for both X86 and ARM. > > > > Hope It Helps, > > Chinna > > > > On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < > >>> zhengzhenyul...@gmail.com> > > wrote: > > > >> Hi Chinna, David and others might interested, > >> > >> Thanks for bring this up, we are c
[jira] [Created] (HIVE-23708) MergeFileTask.execute() need to close jobclient
YulongZ created HIVE-23708: -- Summary: MergeFileTask.execute() need to close jobclient Key: HIVE-23708 URL: https://issues.apache.org/jira/browse/HIVE-23708 Project: Hive Issue Type: Bug Environment: Hadoop 3.1 YARN 3.1 （with timelineserver enabled，https enabled) Hive 3.1 Reporter: YulongZ So when YARN useuseHttps, MergeFileTask causes more and more Threads named “ReloadingX509TrustManager” in HiveServer2。The threads named “ReloadingX509TrustManager” does not interrupt，and HS2 becomes abnormal。 In MergeFileTask.execute(DriverContext driverContext) ,a Jobclient created but not closed。The issue cause above。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23707) Unable to create materialized views with transactions enabled with MySQL metastore
Dustin Koupal created HIVE-23707:
Summary: Unable to create materialized views with transactions
enabled with MySQL metastore
Key: HIVE-23707
URL: https://issues.apache.org/jira/browse/HIVE-23707
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 3.1.2
Reporter: Dustin Koupal
When attempting to create a materialized view with transactions enabled, we get
the following exception:
{code:java}
ERROR : FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Failed to
generate new Mapping of type
org.datanucleus.store.rdbms.mapping.java.StringMapping, exception : JDBC type
CLOB declared for field
"org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type
java.lang.String cant be mapped for this datastore.ERROR : FAILED: Execution
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:Failed to generate new Mapping of type
org.datanucleus.store.rdbms.mapping.java.StringMapping, exception : JDBC type
CLOB declared for field
"org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type
java.lang.String cant be mapped for this datastore.JDBC type CLOB declared for
field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of
java type java.lang.String cant be mapped for this
datastore.org.datanucleus.exceptions.NucleusException: JDBC type CLOB declared
for field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of
java type java.lang.String cant be mapped for this datastore. at
org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getDatastoreMappingClass(RDBMSMappingManager.java:1386)
at
org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.createDatastoreMapping(RDBMSMappingManager.java:1616)
at
org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.prepareDatastoreMapping(SingleFieldMapping.java:59)
at
org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.initialize(SingleFieldMapping.java:48)
at
org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getMapping(RDBMSMappingManager.java:482)
at
org.datanucleus.store.rdbms.table.ClassTable.manageMembers(ClassTable.java:536)
at
org.datanucleus.store.rdbms.table.ClassTable.manageClass(ClassTable.java:442)
at
org.datanucleus.store.rdbms.table.ClassTable.initializeForClass(ClassTable.java:1270)
at
org.datanucleus.store.rdbms.table.ClassTable.initialize(ClassTable.java:276) at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3279)
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2889)
at
org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2088)
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1271)
at
org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3760)
at
org.datanucleus.state.StateManagerImpl.setIdentity(StateManagerImpl.java:2267)
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:484)
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:120)
at
org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:218)
at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2079)
at
org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923)
at
org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778)
at
org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724)
at
org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
at
org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1308)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at
com.sun.proxy.$Proxy25.createTable(Unknown Source) at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1882)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore
[jira] [Created] (HIVE-23706) Fix nulls first sorting behavior
Krisztian Kasa created HIVE-23706: - Summary: Fix nulls first sorting behavior Key: HIVE-23706 URL: https://issues.apache.org/jira/browse/HIVE-23706 Project: Hive Issue Type: Bug Components: Parser Reporter: Krisztian Kasa Assignee: Krisztian Kasa Fix For: 4.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Open old PRs
Hi, > Sadly I think there is not much we can do for contributions >1 year - those > patches will be likely outdated already - and the contributor have probably > abandoned it. Another point of view would be that the project has failed to either review, merge or encourage the committer to continue to work on this project. Imagine you make your first coupe of small contribution to a project and then some point later you are told they are stale and won't be acted upon. Would you want to make further contributions to that project? Is doing this welcoming to new committers? Thanks, Justin
Re: Flaky test checker
Hey Zoltan, I've also seen this one several times recently: testExternalTablesReplLoadBootstrapIncr – org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios On Thu, Jun 11, 2020 at 10:54 AM David Mollitor wrote: > Zoltan, > > I've seen 'org.apache.hadoop.hive.kafka.TransactionalKafkaWriterTest' > failing quite a bit in some recent runs in GitHup precomit. > > Topic `TOPIC_TEST` to delete does not exist > Stacktrace > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: Topic > `TOPIC_TEST` to delete does not exist > > On Wed, Jun 10, 2020 at 9:53 AM Zoltan Haindrich wrote: > >> One more thing: there should be other builds running while the flaky >> check is being executed (otherwise it will be "alone" on a 12 core system) >> >> On 6/10/20 3:49 PM, Zoltan Haindrich wrote: >> > Hey All! >> > >> > I've fiddled around to build this into the main test system or not; but >> in the end I've concluded that it will be more usefull as a standalone tool >> (this makes the job a >> > bit uglier - but well...it would have made the main one uglier as well >> - so it doesn't matter which finger I'll bite) >> > >> > So...if you are suspecting that test is causing trouble for no good >> reason; you could launch a run of this job which will run it a 100 times in >> a row...if it fails...well: >> > * you could open a jira which references the check you executed which >> proves that the test is low quality >> >* please also add the "flaky-test" label to the jira >> > * add an Ignore to the test referencing the jira ticket >> > * push the commit which disables the test... >> > >> > The other use would be when enabling previously unreliable tests back: >> > * push your branch which supposed to stabilize the test to your own >> fork on github >> > * visit http://130.211.9.232/job/hive-flaky-check/ >> > * point the job to your user/repo/branch ; and configure to run the >> test in question to validate it >> > >> > >> > cheers, >> > Zoltan >> >
[jira] [Created] (HIVE-23705) Add tests for 'external.table.purge' and 'auto.purge'
Yu-Wen Lai created HIVE-23705: - Summary: Add tests for 'external.table.purge' and 'auto.purge' Key: HIVE-23705 URL: https://issues.apache.org/jira/browse/HIVE-23705 Project: Hive Issue Type: Test Components: Standalone Metastore Reporter: Yu-Wen Lai Assignee: Yu-Wen Lai The current unit tests did not include an external table with setting 'external.table.purge' or 'auto.purge'. It should be added into TestTableCreateDropAlterTruncate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23704) Thrift HTTP Server Does Not Handle Auth Handle Correctly
David Mollitor created HIVE-23704:
-
Summary: Thrift HTTP Server Does Not Handle Auth Handle Correctly
Key: HIVE-23704
URL: https://issues.apache.org/jira/browse/HIVE-23704
Project: Hive
Issue Type: Bug
Components: Security
Affects Versions: 2.3.7, 3.1.2
Reporter: David Mollitor
Assignee: David Mollitor
Fix For: 4.0.0
Attachments: Base64NegotiationError.png
{code:java|title=ThriftHttpServlet.java}
private String[] getAuthHeaderTokens(HttpServletRequest request,
String authType) throws HttpAuthenticationException {
String authHeaderBase64 = getAuthHeader(request, authType);
String authHeaderString = StringUtils.newStringUtf8(
Base64.decodeBase64(authHeaderBase64.getBytes()));
String[] creds = authHeaderString.split(":");
return creds;
}
{code}
So here, it takes the authHeaderBase64 (which is a base-64 string), and
converts it into bytes, and then it tries to decode those bytes. That is
incorrect It should covert base-64 string directly into bytes.
I tried to do this as part of [HIVE-22676] and the tests was failing because
the string that is being decoded is not actually Base-64 (see attached image).
Again, the existing code doesn't care because it's not parsing Base-64 text, it
is parsing the bytes generated by converting base-64 text to bytes.
I'm not sure what affect this has, what security issues this may present, but
it's definitely not correct.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23703) Major QB compaction with multiple FileSinkOperators results in data loss and one original file
Karen Coppage created HIVE-23703:
Summary: Major QB compaction with multiple FileSinkOperators
results in data loss and one original file
Key: HIVE-23703
URL: https://issues.apache.org/jira/browse/HIVE-23703
Project: Hive
Issue Type: Bug
Reporter: Karen Coppage
Assignee: Karen Coppage
h4. Problems
Example:
{code:java}
drop table if exists tbl2;
create transactional table tbl2 (a int, b int) clustered by (a) into 4 buckets
stored as ORC
TBLPROPERTIES('transactional'='true','transactional_properties'='default');
insert into tbl2 values(1,2),(1,3),(1,4),(2,2),(2,3),(2,4);
insert into tbl2 values(3,2),(3,3),(3,4),(4,2),(4,3),(4,4);
insert into tbl2 values(5,2),(5,3),(5,4),(6,2),(6,3),(6,4);{code}
E.g. in the example above, bucketId=0 when a=2 and a=6.
1. Data loss
In non-acid tables, an operator's temp files are named with their task id.
Because of this snippet, temp files in the FileSinkOperator for compaction
tables are identified by their bucket_id.
{code:java}
if (conf.isCompactionTable()) {
fsp.initializeBucketPaths(filesIdx, AcidUtils.BUCKET_PREFIX +
String.format(AcidUtils.BUCKET_DIGITS, bucketId),
isNativeTable(), isSkewedStoredAsSubDirectories);
} else {
fsp.initializeBucketPaths(filesIdx, taskId, isNativeTable(),
isSkewedStoredAsSubDirectories);
}
{code}
So 2 temp files containing data with a=2 and a=6 will be named bucket_0 and not
00_0 and 00_1 as they would normally.
In FileSinkOperator.commit, when data with a=2, filename: bucket_0 is moved
from _task_tmp.-ext-10002 to _tmp.-ext-10002, it overwrites the files already
there with a=6 data, because it too is named bucket_0. You can see in the logs:
{code:java}
WARN [LocalJobRunner Map Task Executor #0] exec.FileSinkOperator: Target path
file:.../hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnNoBuckets-1591107230237/warehouse/testmajorcompaction/base_002_v013/.hive-staging_hive_2020-06-02_07-15-21_771_8551447285061957908-1/_tmp.-ext-10002/bucket_0
with a size 610 exists. Trying to delete it.
{code}
2. Results in one original file
OrcFileMergeOperator merges the results of the FSOp into 1 file named 00_0.
h4. Fix
1. FSOp will store data as: taskid/bucketId. e.g. 0_0/bucket_0
2. OrcMergeFileOp, instead of merging a bunch of files into 1 file named
00_0, will merge all files named bucket_0 into one file named bucket_0, and
so on.
3. MoveTask will get rid of the taskId directories if present and only move the
bucket files in them, in case OrcMergeFileOp is not run.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: HIVE building on ARM
Hey, There is an effort by the Apache Infra to change the way Jenkins stuff is organized; a couple months ago Gavin wrote an email about it: http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E The resources for running these jobs are coming from the H0~H21 slaves which will be migrated to the new jenkins master eventually. >> So please >> suggest a way which direction we can move and can you share some details >> about the new ci-hadoop instance. Since Hadoop testing is also happening on ARM - I think the best would be to also migrate the armN slaves and the Hive arm nightly over to the new ci-hadoop instance. On 6/16/20 8:40 AM, Zhenyu Zheng wrote: Thanks for the info, I wonder if where does the resource of ci-hadoop and hive-test-kube come from? Do they include ARM resources? Interesting question; the resources for Hive testing are donated by Cloudera. About the ARM workers I think Chinna could provide more details. ...I've no idea don't know who sponsors the Hxx slaves Can you provide some more information about how the new hive-test-kube is running? It's basically a Jenkins instance which is using kubernetes pods to run things. The whole thing is running on a GKE cluster. While I was working on it I collected stuff needed for it in this repo: https://github.com/kgyrtkirk/hive-test-kube/ it should be possible to start a new deployment using that stuff cheers, Zoltan BR, Kevin Zheng On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < lalamchinnara...@gmail.com> wrote: Hi Zoltan, Thanks for the update. Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is targeting to run hive tests daily on "arm" slaves, it is using 2 arm slaves. To find any potential issues with "arm" and fix the issues. So please suggest a way which direction we can move and can you share some details about the new ci-hadoop instance. Thanks, Chinna On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich wrote: Hey all, In an ticket (INFRA-20416) Gavin asked me if we are completely off builds.apache.org - when I went over the jobs I've saw that https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once a day. Since builds.apache.org will be shut down in sometime in the future - we should move this job to the new ci-hadoop instance or to hive-test-kube. The key feature of the job is that it runs the test on the "armX" slaves; which are statically configured on b.a.o. Not sure which way to go - but we will have to move in some direction. cheers, Zoltan On 3/13/20 7:22 AM, Zhenyu Zheng wrote: Hi Chinna, Thanks alot for the reply, I uploaded a patch and also a github PR for https://issues.apache.org/jira/browse/HIVE-21939 . In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1 and added a new profile, this profile will identify the hardware architecture and if it is Aarch64, it will override the protobuf group.id and package to com.github.os72 which includes ARM support. For X86 platform, Hive will still download the protobuf packages from org.google repo. I think with this method, we can keep the influence to existing x86 users to the minimum. I hope this could be a acceptable short-term solution. I've manually tested on my machine and the github PR travis CI test has already passed, so the build process is OK, so let's wait for the full test result from builds.apache.org. BR, Zhenyu On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < lalamchinnara...@gmail.com> wrote: Hi Zhenyu, Until HBase dependency resolved, without effecting the existing code on X86 i suggest create a separate profile with "os72" repo. Down the line we should have common version for both X86 and ARM. Hope It Helps, Chinna On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < zhengzhenyul...@gmail.com> wrote: Hi Chinna, David and others might interested, Thanks for bring this up, we are currently working on improving enabling big-data software on the ARM platform, we have already done fixes and providing CIs to some of the well-know projects like: 1. Hadoop: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ 3. HBase: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ And we are now working on projects including Hive, Kudu, etc. Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and break dependency for HBase, there can be some possible short-term plan(or walk-arounds), doing thes can make Hive work on ARM without break any dependencies, and then we can interact with Hbase project to see how can we both upgrade to 3.x(since this make take some time). Those possible solutions can be: 1. Using pre-patched protobuf 2.5.0 with ARM support from org.openlabtesting repo, some projects(HBase did this: https://github.com/apache/hbase/pull/959, and we will add a prof
[jira] [Created] (HIVE-23702) Add metastore metrics to show age of the oldest initiated compaction
Peter Vary created HIVE-23702: - Summary: Add metastore metrics to show age of the oldest initiated compaction Key: HIVE-23702 URL: https://issues.apache.org/jira/browse/HIVE-23702 Project: Hive Issue Type: Improvement Components: Transactions Reporter: Peter Vary Assignee: Peter Vary It would be good to have a metrics which will show the age of the oldest initiated compaction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23701) SQL select failing with java.lang.ClassCastException
Shubham Sharma created HIVE-23701:
-
Summary: SQL select failing with java.lang.ClassCastException
Key: HIVE-23701
URL: https://issues.apache.org/jira/browse/HIVE-23701
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 3.0.0
Reporter: Shubham Sharma
Below SQL worked fine with no issues on Hive 3:
{code:java}
select distinct column_name from table_name
{code}
This SQL without distinct/count clause failing with below stack trace:
{code:java}
select column_name from table_name
{code}
After switching _task.conversion_ to _none_ query initiated Tez tasks schedule
and completed successfully, can't use this workaround for the deployment.
Here is the error trace:
{code:java}
WARN [HiveServer2-Handler-Pool: Thread-694088]: thrift.ThriftCLIService (:())
- Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
java.lang.ClassCastException
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_212]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_212]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at com.sun.proxy.$Proxy71.fetchResults(Unknown Source) ~[?:?]
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792)
~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.io.IOException: java.lang.ClassCastException
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2738)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
~[hive-
