[jira] [Created] (HIVE-27164) Create Temp Txn Table As Select is failing at tablePath validation
Naresh P R created HIVE-27164:
-
Summary: Create Temp Txn Table As Select is failing at tablePath
validation
Key: HIVE-27164
URL: https://issues.apache.org/jira/browse/HIVE-27164
Project: Hive
Issue Type: Bug
Components: HiveServer2, Metastore
Reporter: Naresh P R
Attachments: mm_cttas.q
After HIVE-25303, every CTAS goes for
HiveMetaStore$HMSHandler#translate_table_dryrun() call to fetch table location
for CTAS queries which fails with following exception for temp tables if
MetastoreDefaultTransformer is set.
{code:java}
2023-03-17 16:41:23,390 INFO
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
[pool-6-thread-196]: Starting translation for CreateTable for processor
HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD,
HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS,
HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT,
HIVEONLYMQTWRITE] on table test_temp
2023-03-17 16:41:23,392 ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-196]:
MetaException(message:Illegal location for managed table, it has to be within
database's managed location)
at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.validateTablePaths(MetastoreDefaultTransformer.java:886)
at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:666)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) {code}
I am able to repro this issue at apache upstream using attached testcase.
There are multiple ways to fix this issue
* Have temp txn table path under db's managed location path. This will help
with encryption zone tables as well.
* skips location check for temp tables at
MetastoreDefaultTransformer#validateTablePaths()
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27114) Provide a configurable filter for removing useless properties from PartitionDesc objects from getPartitions HMS Calls
Naresh P R created HIVE-27114:
-
Summary: Provide a configurable filter for removing useless
properties from PartitionDesc objects from getPartitions HMS Calls
Key: HIVE-27114
URL: https://issues.apache.org/jira/browse/HIVE-27114
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
HMS API calls are throwing following exception because of thrift upgrade
{code:java}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at
org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
at
org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
at
org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
{code}
Large size partition metadata is causing this issue
eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys =
(impala_intermediate_stats_chunk*){*}, these PARTITION_PARAM is not required
for Hive. These params should be skipped while preparing partition object from
HMS to HS2.
Similarly any user defined regex should be skipped in getPartitions HMS API
call. Similar to HIVE-25501
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3
+1 to remove Hive on Spark in branch-3 --- Regards, Naresh P R On Wed, Feb 22, 2023 at 5:37 AM Sankar Hariappan wrote: > +1, to remove Hive on Spark in branch-3. > > Thanks, > Sankar > > -Original Message- > From: Rajesh Balamohan > Sent: Wednesday, February 22, 2023 6:58 PM > To: dev@hive.apache.org > Subject: [EXTERNAL] Re: Proposal to deprecate Hive on Spark from branch-3 > > +1 on removing Hive on Spark in branch-3. > > It was not done earlier since it was removing a feature in the branch. But > if there is enough consensus, we should consider removing it. > > ~Rajesh.B > > On Wed, Feb 22, 2023 at 12:48 PM Aman Raj > wrote: > > > Hi team, > > > > We have been trying to fix Hive on Spark test failures for a long > > time. As of now, branch-3 has less than 12 test failures (whose fix > > have not been identified). 8 of them are related to Hive on Spark. I > > had mailed about the failures in my previous mail threads. Thanks to > > Vihang for working on them as well. But we have not been able to > identify the root cause till now. > > These fixes can be tracked in the following tickets : [HIVE-27087] Fix > > TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA ( > > apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https% > > 3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-27087=05%7C01%7C > > Sankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C > > 72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknow > > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC > > JXVCI6Mn0%3D%7C3000%7C%7C%7C=RbAqrwK6fQFDStufXYfpusNc81EzjtpiaHm > > qv5CFiAs%3D=0> and [HIVE-26940] Backport of HIVE-19882 : Fix > > QTestUtil session lifecycle - ASF JIRA > > (apache.org)<https://nam06.safelinks.protection.outlook.com/?url=https > > %3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-26940=05%7C01%7 > > CSankar.Hariappan%40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7 > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnkno > > wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > > CJXVCI6Mn0%3D%7C3000%7C%7C%7C=PaDtEZD569Sv0ER9sC4l6q1ZxyoBeER3zn > > Bsc51PWI8%3D=0> > > > > Until we have a green branch-3, we cannot go ahead to push new > > features for the Hive-3.2.0 release. This is kind of a blocker for this > release. > > Already bringing the test fixes to the current state took more than 2 > > months. > > > > I wanted to bring up a proposal to deprecate Hive on Spark from > > branch-3 altogether. This would ensure that branch-3 is aligned with > > the master as done in > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissu > > es.apache.org > %2Fjira%2Fbrowse%2FHIVE-26134=05%7C01%7CSankar.Hariappan% > 40microsoft.com%7C687a6a4dbd41454568e008db14d8cc23%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638126693641861742%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=j%2F%2BYTakSvfk1Mm%2Fq8kI84gKm6s%2Fg2iA1abIPzY7t%2Bsg%3D=0. > Just wanted to have a vote on this in parallel working on the test fixes. > If we have the approval from the community, we can deprecate it altogether. > > > > Please feel free to suggest any concerns or suggestions you have. > > Also, I welcome any possible fix suggestion for the test failures. > > > > Thanks, > > Aman. > > >
[jira] [Created] (HIVE-27069) Incorrect results with bucket map join
Naresh P R created HIVE-27069:
-
Summary: Incorrect results with bucket map join
Key: HIVE-27069
URL: https://issues.apache.org/jira/browse/HIVE-27069
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Attachments: test.q
Attaching test.q file for repro.
Following query in testcase is expected to produce 9 records, but it is
producing 5 records.
{code:java}
select * from DUP_TEST_TARGET T join (SELECT id , in_date , sample FROM (SELECT
id , in_date , sample ,ROW_NUMBER()
OVER(PARTITION BY id ORDER BY in_date DESC ) AS ROW_NUMB FROM DUP_TEST)
OUTQUERY WHERE ROW_NUMB =1) as S ON T.id = S.id;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26526) MSCK sync is not removing partitions with special characters
Naresh P R created HIVE-26526:
-
Summary: MSCK sync is not removing partitions with special
characters
Key: HIVE-26526
URL: https://issues.apache.org/jira/browse/HIVE-26526
Project: Hive
Issue Type: New Feature
Reporter: Naresh P R
PARTITIONS table were having encoding string & PARTITION_KEY_VALS were having
original string.
{code:java}
hive=> select * from "PARTITION_KEY_VALS" where "PART_ID" IN (46753, 46754,
46755, 46756);
PART_ID | PART_KEY_VAL | INTEGER_IDX
-+-+-
46753 | 2022-02-* | 0
46754 | 2011-03-01 | 0
46755 | 2022-01-* | 0
46756 | 2010-01-01 | 0
hive=> select * from "PARTITIONS" where "TBL_ID" = 23567 ;
PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID |
TBL_ID | WRITE_ID
-+-+--+---+---++--
46753 | 0 | 0 | part_date=2022-02-%2A | 70195 |
23567 | 0
46754 | 0 | 0 | part_date=2011-03-01 | 70196 |
23567 | 0
46755 | 0 | 0 | part_date=2022-01-%2A | 70197 |
23567 | 0
46756 | 0 | 0 | part_date=2010-01-01 | 70198 |
23567 | 0
(4 rows){code}
1) DirectSQL has a join condition on PARTITION_KEY_VALS.PART_KEY_VAL =
"2022-02-%2A" at here
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L883
2) Jdo is having filter condition on PARTITIONS.PART_NAME =
"part_date=2022-02-%252A" (ie., 2 times url encoded)
Once from HS2
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L353
2nd from HMS
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java#L365]
Above conditions returns 0 partitions, so those are not removed from HMS
metadata.
Attaching repro q file
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26495) MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus
Naresh P R created HIVE-26495:
-
Summary: MSCK repair perf issue HMSChecker ThreadPool is blocked
at fs.listStatus
Key: HIVE-26495
URL: https://issues.apache.org/jira/browse/HIVE-26495
Project: Hive
Issue Type: New Feature
Reporter: Naresh P R
Assignee: Naresh P R
With hive.metastore.fshandler.threads = 15, all 15 *MSCK-GetPaths-xx* are
slogging at following trace.
{code:java}
"MSCK-GetPaths-11" #12345 daemon prio=5 os_prio=0 tid= nid= waiting on
condition [0x7f9f099a6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0003f92d1668> (a
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
...
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3230)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1995)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.processPathDepthInfo(HiveMetaStoreChecker.java:550)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:543)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:525)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750){code}
We should take advantage of non-block listStatusIterator instead of listStatus
which is a blocking call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26462) SMTP mail alerts for critical DML's
Naresh P R created HIVE-26462: - Summary: SMTP mail alerts for critical DML's Key: HIVE-26462 URL: https://issues.apache.org/jira/browse/HIVE-26462 Project: Hive Issue Type: New Feature Reporter: Naresh P R Any alerting mechanism via smtp to single configured email for mission critical DMLs like Drop db/table/partition/materialized view/view on specific db/table pattern will be very useful on production systems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26276) Fix package to org.apache.hadoop.hive.serde2 for JsonSerDe & RegexSerDe in HMS DB
Naresh P R created HIVE-26276:
-
Summary: Fix package to org.apache.hadoop.hive.serde2 for
JsonSerDe & RegexSerDe in HMS DB
Key: HIVE-26276
URL: https://issues.apache.org/jira/browse/HIVE-26276
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Similar to HIVE-24770, JsonSerDe & RegexSerDe should be updated to newer package
{code:java}
// Avoid dependency of hive-hcatalog.jar
Old - org.apache.hive.hcatalog.data.JsonSerDe
New - org.apache.hadoop.hive.serde2.JsonSerDe
// Avoid dependency of hive-contrib.jar
Old - org.apache.hadoop.hive.contrib.serde2.RegexSerDe
New - org.apache.hadoop.hive.serde2.RegexSerDe
{code}
This should be handled in upgrade flow.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HIVE-26235) OR Condition on binary column is returning empty result
Naresh P R created HIVE-26235:
-
Summary: OR Condition on binary column is returning empty result
Key: HIVE-26235
URL: https://issues.apache.org/jira/browse/HIVE-26235
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Repro steps
{code:java}
create table test_binary(data_col timestamp, binary_col binary) partitioned by
(ts string);
insert into test_binary partition(ts='20220420') values ('2022-04-20
00:00:00.0', 'a'),('2022-04-20 00:00:00.0', 'b'), ('2022-04-20 00:00:00.0',
'c');
// Works
select * from test_binary where ts='20220420' and binary_col = unhex('61');
select * from test_binary where ts='20220420' and binary_col between
unhex('61') and unhex('62');
//Returns empty result
select * from test_binary where binary_col = unhex('61') or binary_col =
unhex('62');
select * from test_binary where ts='20220420' and (binary_col = unhex('61')
or binary_col = unhex('62'));
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HIVE-26096) Select on single column MultiDelimitSerDe table throws AIOBE
Naresh P R created HIVE-26096:
-
Summary: Select on single column MultiDelimitSerDe table throws
AIOBE
Key: HIVE-26096
URL: https://issues.apache.org/jira/browse/HIVE-26096
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Repro details
{code:java}
create table test_multidelim(col string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe'
with serdeproperties('field.delim'='!^') STORED AS TEXTFILE;
insert into test_multidelim values('aa'),('bb'),('cc'),('dd');
select * from test_multidelim;
{code}
Exception:
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit(LazyStruct.java:303)
at
org.apache.hadoop.hive.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:160)
at
org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:74)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:603){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26054) Distinct + Groupby with column alias is failing
Naresh P R created HIVE-26054:
-
Summary: Distinct + Groupby with column alias is failing
Key: HIVE-26054
URL: https://issues.apache.org/jira/browse/HIVE-26054
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
After [HIVE-16924|https://issues.apache.org/jira/browse/HIVE-16924], below
query is failing.
{code:java}
create table table1 (col1 bigint, col2 string);
create table table2 (t2_col1 string);
Select distinct col1 as alias_col1
from table1
where col2 = (SELECT max(t2_col1) as currentdate from table2 limit 1)
order by col1;
Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
Unsupported SubQuery Expression '1': Only SubQuery expressions that are top
level conjuncts are allowed (state=42000,code=4) {code}
Workaround is either remove distinct column alias "alias_col1" or use alias in
order by.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26047) Vectorized LIKE UDF should use Re2J regex to address JDK-8203458
Naresh P R created HIVE-26047:
-
Summary: Vectorized LIKE UDF should use Re2J regex to address
JDK-8203458
Key: HIVE-26047
URL: https://issues.apache.org/jira/browse/HIVE-26047
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Below pattern is taking a long time to validate regex in java8 with same trace
as shown in java bug
[[JDK-8203458||https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8203458]
[https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8203458]
[]|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8203458]
import java.util.regex.Pattern;
public class ABCD {
public static void main(String args[]) {
String pattern =
"a_b";
Pattern CHAIN_PATTERN = Pattern.compile("(%?[^%_]+%?)+");
CHAIN_PATTERN.matcher(pattern).matches();
}
}
Same is reproducible with following SQL
{code:java}
create table table1(name string);
insert into table1 (name) values
('a_b');
select * from table1 where name like
"a_b";{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26006) TopNKey and PTF with more than one column is failing with IOBE
Naresh P R created HIVE-26006: - Summary: TopNKey and PTF with more than one column is failing with IOBE Key: HIVE-26006 URL: https://issues.apache.org/jira/browse/HIVE-26006 Project: Hive Issue Type: Bug Reporter: Naresh P R java.lang.IndexOutOfBoundsException: toIndex = 2 at java.util.ArrayList.subListRangeCheck(ArrayList.java:1014) at java.util.ArrayList.subList(ArrayList.java:1006) at org.apache.hadoop.hive.ql.plan.TopNKeyDesc.combine(TopNKeyDesc.java:201) at org.apache.hadoop.hive.ql.optimizer.topnkey.TopNKeyPushdownProcessor.pushdownThroughGroupBy(TopNKeyPushdownProcessor.java:162) at org.apache.hadoop.hive.ql.optimizer.topnkey.TopNKeyPushdownProcessor.pushdown(TopNKeyPushdownProcessor.java:76) at org.apache.hadoop.hive.ql.optimizer.topnkey.TopNKeyPushdownProcessor.process(TopNKeyPushdownProcessor.java:57) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) at org.apache.hadoop.hive.ql.parse.TezCompiler.runTopNKeyOptimization(TezCompiler.java:1305) at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:173) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:159) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12646) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:283) at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:219) at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:103) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:215) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26000) Partition table with Skew columns, DirectSQL to pruning partitions fails with Postgres backend
Naresh P R created HIVE-26000:
-
Summary: Partition table with Skew columns, DirectSQL to pruning
partitions fails with Postgres backend
Key: HIVE-26000
URL: https://issues.apache.org/jira/browse/HIVE-26000
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
{code:java}
2022-03-02 20:37:56,421 INFO
org.apache.hadoop.hive.metastore.PartFilterExprUtil: [pool-6-thread-200]:
Unable to make the expression tree from expression string [((ds = '2008-04-08')
and (UDFToDouble(hr) = 11.0D))]Error parsing partition filter; lexer error:
null; exception NoViableAltException(24@[])
2022-03-02 20:37:56,593 WARN org.apache.hadoop.hive.metastore.ObjectStore:
[pool-6-thread-200]: Falling back to ORM path due to direct SQL failure (this
is not an error): Error executing SQL query "select
"SKEWED_COL_VALUE_LOC_MAP"."SD_ID", "SKEWED_STRING_LIST_VALUES".STRING_LIST_ID,
"SKEWED_COL_VALUE_LOC_MAP"."LOCATION",
"SKEWED_STRING_LIST_VALUES"."STRING_LIST_VALUE" from "SKEWED_COL_VALUE_LOC_MAP"
left outer join "SKEWED_STRING_LIST_VALUES" on
"SKEWED_COL_VALUE_LOC_MAP"."STRING_LIST_ID_KID" =
"SKEWED_STRING_LIST_VALUES"."STRING_LIST_ID" where
"SKEWED_COL_VALUE_LOC_MAP"."SD_ID" in (51010) and
"SKEWED_COL_VALUE_LOC_MAP"."STRING_LIST_ID_KID" is not null order by
"SKEWED_COL_VALUE_LOC_MAP"."SD_ID" asc,
"SKEWED_STRING_LIST_VALUES"."STRING_LIST_ID" asc,
"SKEWED_STRING_LIST_VALUES"."INTEGER_IDX" asc". at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391) at
org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216) at
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.loopJoinOrderedResult(MetastoreDirectSqlUtils.java:131)
at
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.loopJoinOrderedResult(MetastoreDirectSqlUtils.java:109)
at
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.setSkewedColLocationMaps(MetastoreDirectSqlUtils.java:414)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:967)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:788)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:117)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:530)
at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:521)
at
org.apache.hadoop.hive.metastore.ObjectStore$10.getSqlResult(ObjectStore.java:3722);
Caused by: ERROR: column SKEWED_STRING_LIST_VALUES.string_list_id does not
exist
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-25989) CTLT HBaseStorageHandler is dropping underlying HBase table when failed
Naresh P R created HIVE-25989:
-
Summary: CTLT HBaseStorageHandler is dropping underlying HBase
table when failed
Key: HIVE-25989
URL: https://issues.apache.org/jira/browse/HIVE-25989
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
With hive.strict.managed.tables & hive.create.as.acid,
Hive-Hbase rollback code is assuming it is a createTable failure instead of
CTLT & removing underlying hbase table while rolling back at here.
[https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java#L187-L195]
Repro
{code:java}
hbase
=
hbase shell
create 'hbase_hive_table', 'cf'
beeline
===
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.strict.managed.tables=true;
set hive.create.as.acid=true;
set hive.create.as.insert.only=true;
set hive.default.fileformat.managed=ORC;
> CREATE EXTERNAL TABLE `hbase_hive_table`(
`key` int COMMENT '',
`value` string COMMENT '')
ROW FORMAT SERDE
'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY
'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
'hbase.columns.mapping'=':key,cf:cf')
TBLPROPERTIES ('hbase.table.name'='hbase_hive_table');
> select * from hbase_hive_table;
+---+-+
| hbase_hive_table.key | hbase_hive_table.value |
+---+-+
+---+-+
> create table new_hbase_hive_table like hbase_hive_table;
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: The table must
be stored using an ACID compliant format (such as ORC):
default.new_hbase_hive_table
> select * from hbase_hive_table;
Error: java.io.IOException: org.apache.hadoop.hbase.TableNotFoundException:
hbase_hive_table
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [ANNOUNCE] Denys Kuzmenko joins Hive PMC
Congrats Denys, well deserved !!! --- Regards, Naresh P R On Mon, Feb 7, 2022 at 8:40 AM Ashutosh Chauhan wrote: > Hi, > > I'm pleased to announce that Denys has accepted an invitation to > join the Hive PMC. Denys has been a consistent and helpful > figure in the Hive community for which we are very grateful. We > look forward to the continued contributions and support. > > Please join me in congratulating Denys! > > Ashutosh (On behalf of Hive PMC) >
Re: [ANNOUNCE] New committer: Ayush Saxena
Congrats Ayush !!! --- Regards, Naresh P R On Mon, Feb 7, 2022 at 7:35 AM Ashutosh Chauhan wrote: > Hi all, > Apache Hive's Project Management Committee (PMC) has invited Ayush > to become a committer, and we are pleased to announce that he has accepted! > > Ayush welcome, thank you for your contributions, and we look forward to > your > further interactions with the community! > Ashutosh (on behalf of Hive PMC) >
[jira] [Created] (HIVE-25919) CCE when validating AND PPD in HBaseStorageHandler
Naresh P R created HIVE-25919:
-
Summary: CCE when validating AND PPD in HBaseStorageHandler
Key: HIVE-25919
URL: https://issues.apache.org/jira/browse/HIVE-25919
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
After HIVE-13815,
[this|https://github.com/apache/calcite/blob/branch-1.10/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L1797-L1810]
change is removing (EQUALS true) from predicate expression.
AND Condition with boolean=true is throwing CCE at here because of above change.
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java#L163]
{code:java}
ClassCastException org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be
cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
java.lang.ClassCastException: org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc
cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
at
org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer$1.process(IndexPredicateAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:178)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer.analyzePredicate(IndexPredicateAnalyzer.java:174)
at
org.apache.hadoop.hive.hbase.HBaseStorageHandler.decomposePredicate(HBaseStorageHandler.java:415)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: Welcome Marta to Hive PMC
Congrats Marta. --- Regards, Naresh P R On Mon, Aug 2, 2021 at 9:51 PM Ashutosh Chauhan wrote: > Hi all, > > It's an honor to announce that Apache Hive PMC has recently voted to invite > Marta Kuczora as a new Hive PMC member. Marta is a long time Hive > contributor and committer, and has made significant contributions in Hive. > Please join me in congratulating her and looking forward to a bigger role > that she will play in the Apache Hive project. > > Thanks, > Ashutosh >
[jira] [Created] (HIVE-25419) HMS Client filter fails with StorageBasedAuthorizationProvider
Naresh P R created HIVE-25419:
-
Summary: HMS Client filter fails with
StorageBasedAuthorizationProvider
Key: HIVE-25419
URL: https://issues.apache.org/jira/browse/HIVE-25419
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
hive.security.metastore.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.metastore.client.filter.enabled=true
"Show tables" fails at here
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java#L378-L379]
With following exception:
{code:java}
2021-07-29 17:09:12,278 ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-14-thread-5]:
MetaException(message:Error in
HiveMetaStoreAuthorizer.filterTables()java.lang.RuntimeException: class
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
not
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerFactory)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.filterTableNames(HiveMetaStoreAuthorizer.java:333)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.filterTableNames(HiveMetaStoreAuthorizer.java:163)
at
org.apache.hadoop.hive.metastore.utils.FilterUtils.filterTableNamesIfEnabled(FilterUtils.java:109)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_tables_by_type(HiveMetaStore.java:5905){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25338) AIOBE in conv UDF if input is empty
Naresh P R created HIVE-25338:
-
Summary: AIOBE in conv UDF if input is empty
Key: HIVE-25338
URL: https://issues.apache.org/jira/browse/HIVE-25338
Project: Hive
Issue Type: New Feature
Reporter: Naresh P R
Assignee: Naresh P R
Repro
{code:java}
create table test (a string);
insert into test values ("");
select conv(a,16,10) from test;{code}
Exception trace:
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.hive.ql.udf.UDFConv.evaluate(UDFConv.java:160){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25302) Add support for New MR API (org.apache.hadoop.mapreduce InputFormat and OutputFormat)
Naresh P R created HIVE-25302: - Summary: Add support for New MR API (org.apache.hadoop.mapreduce InputFormat and OutputFormat) Key: HIVE-25302 URL: https://issues.apache.org/jira/browse/HIVE-25302 Project: Hive Issue Type: New Feature Reporter: Naresh P R We have the mapred InputFormat and OutputFormat, but we need one for the newer API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25198) CTAS external table with camelcase & HMS translation ON is returning 0 records
Naresh P R created HIVE-25198:
-
Summary: CTAS external table with camelcase & HMS translation ON
is returning 0 records
Key: HIVE-25198
URL: https://issues.apache.org/jira/browse/HIVE-25198
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
create external table TarGet as select * from source;
Above query creates tableLocation with CamelCase if HMS Translation is ON,
whereas MoveTask will use lowerCase table path.
eg.,
{code:java}
==> Desc formatted target <==
Location: hdfs:///warehouse/tablespace/external/hive/test.db/TarGet
==> MoveTask <==
INFO : Moving data to directory
hdfs:///warehouse/tablespace/external/hive/test.db/target from
hdfs:///warehouse/tablespace/external/hive/test.db/.hive-staging_hive_2021-06-04_03-02-36_272_669287187808252905-12/-ext-10002
==> HMS Translation <==
2021-06-04 03:02:45,772 INFO
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
[pool-7-thread-8]: Transformer returning table:Table(tableName:TarGet,
dbName:test, owner:hive, createTime:1622775765, lastAccessTime:0, retention:0,
sd:StorageDescriptor(cols:[FieldSchema(name:id, type:varchar(10),
comment:null)], location:
hdfs:///warehouse/tablespace/external/hive/ajay.db/TarGet,{code}
Select query after CTAS will return 0 rows because of this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25118) CTAS accepts column's with dot(.) if CBO fails
Naresh P R created HIVE-25118:
-
Summary: CTAS accepts column's with dot(.) if CBO fails
Key: HIVE-25118
URL: https://issues.apache.org/jira/browse/HIVE-25118
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
create table t1(id int);
create table t2(id int);
create table t3 as select t1.id, t2.id from t1 join t2;
CBO fails if "hive.stats.column.autogather=true" with "SemanticException
Ambiguous column reference: id" & CTAS passes with following table schema
{code:java}
desc t3;
+---++--+
| col_name | data_type | comment |
+---++--+
| t1.id | int| |
| t2.id | int| |
+---++--+{code}
create table t3(`t1.id` int, `t2.id` int); will fail for dot(.) in column name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25084) Incorrect aggregate results on bucketed table
Naresh P R created HIVE-25084:
-
Summary: Incorrect aggregate results on bucketed table
Key: HIVE-25084
URL: https://issues.apache.org/jira/browse/HIVE-25084
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Steps to repro
{code:java}
CREATE TABLE test_table(
col1 int,
col2 char(32),
col3 varchar(3))
CLUSTERED BY (col2)
SORTED BY (
col2 ASC,
col3 ASC,
col1 ASC)
INTO 32 BUCKETS stored as orc;
set hive.query.results.cache.enabled=false;
insert into test_table values(2, "123456", "15");
insert into test_table values(1, "123456", "15");
SELECT col2, col3, max(col1) AS max_sequence FROM test_table GROUP BY col2,
col3;
==> LocalFetch correct result <==
123456 15 2
==> Wrong result with Tez/Llap <==
set hive.fetch.task.conversion=none;
123456 15 2
123456 15 1
==> Correct result with Tez/Llap disabling map aggregation <==
set hive.map.aggr=false;
123456 15 2
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24882) Compaction task reattempt fails with FileAlreadyExistsException for DeleteEventWriter
Naresh P R created HIVE-24882:
-
Summary: Compaction task reattempt fails with
FileAlreadyExistsException for DeleteEventWriter
Key: HIVE-24882
URL: https://issues.apache.org/jira/browse/HIVE-24882
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
If first attempt of compaction task is pre-empted by yarn or execution failed
because of environmental issues, re-attempted tasks will fail with
FileAlreadyExistsException
{noformat}
Error: org.apache.hadoop.fs.FileAlreadyExistsException:
/warehouse/tablespace/managed/hive/test.db/acid_table/dept=cse/_tmp_xxx/delete_delta_001_010/bucket_0
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1211) at
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1190) at
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1128) at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:531)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:528)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:542)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:469)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
at org.apache.orc.impl.PhysicalFsWriter.(PhysicalFsWriter.java:95)
at org.apache.orc.impl.WriterImpl.(WriterImpl.java:177)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.(WriterImpl.java:94)
at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:378)
at
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.getRawRecordWriter(OrcOutputFormat.java:299)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.getDeleteEventWriter(CompactorMR.java:1084)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:995)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:958){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24877) Support X'xxxx' syntax for hexadecimal values like spark & mysql
Naresh P R created HIVE-24877:
-
Summary: Support X'' syntax for hexadecimal values like spark
& mysql
Key: HIVE-24877
URL: https://issues.apache.org/jira/browse/HIVE-24877
Project: Hive
Issue Type: New Feature
Reporter: Naresh P R
Hive is currently not supporting following syntax
select x'abc';
{code:java}
org.apache.hadoop.hive.ql.parse.ParseException: line 2:8 cannot recognize input
near 'x' ''abc'' '' in selection target
org.apache.hadoop.hive.ql.parse.ParseException: line 2:8 cannot recognize input
near 'x' ''31FECC'' '' in selection target at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:125) at
org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:93) at
org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:85) at
org.apache.hadoop.hive.ql.Compiler.parse(Compiler.java:169) at
org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:102) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445){code}
Though we have same via hex/unhex built-in UDF's, it's better to have
{{X'value'}} and x'{{value'}} syntax support for Hive.
[https://spark.apache.org/docs/latest/sql-ref-literals.html#binary-literal]
[https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_hex]
https://mariadb.com/kb/en/hexadecimal-literals/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24716) jQuery file symlink is replaced by physical file which requires changes on both the places
Naresh P R created HIVE-24716: - Summary: jQuery file symlink is replaced by physical file which requires changes on both the places Key: HIVE-24716 URL: https://issues.apache.org/jira/browse/HIVE-24716 Project: Hive Issue Type: Bug Reporter: Naresh P R HIVE-22099 replaced symlink llap-server/src/main/resources/hive-webapps/llap/js/jquery.min.js -> service/src/resources/hive-webapps/static/js/jquery.min.js with a physical file, whenever jQuery version gets upgraded, same changes needs to be done on both places -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24634) Create table if not exists should validate whether table exists before doAuth()
Naresh P R created HIVE-24634: - Summary: Create table if not exists should validate whether table exists before doAuth() Key: HIVE-24634 URL: https://issues.apache.org/jira/browse/HIVE-24634 Project: Hive Issue Type: Bug Reporter: Naresh P R In Hive + Ranger cluster, Create table if not exist hive-ranger would validate privileges over complete files in table location even thought table already exist. Table exist check should be validated before doAuthorization in compile. at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:291) at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1337) at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:1101) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:710) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24628) Decimal values are displayed as Scientific notation in beeline
Naresh P R created HIVE-24628:
-
Summary: Decimal values are displayed as Scientific notation in
beeline
Key: HIVE-24628
URL: https://issues.apache.org/jira/browse/HIVE-24628
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
As we are using BigDecimal.toString() returns scientific notation instead of
original text, which confuse customer. It should be changed to toPlainString()
at here
[https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/Rows.java#L165]
Repro steps:
{code:java}
beeline> select cast(0 as decimal(20,10));
//output
0E-10
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24433) AutoCompaction is not getting triggered for CamelCase Partition Values
Naresh P R created HIVE-24433:
-
Summary: AutoCompaction is not getting triggered for CamelCase
Partition Values
Key: HIVE-24433
URL: https://issues.apache.org/jira/browse/HIVE-24433
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
partionKey=paritionValue is getting converted into lowerCase in below 2 places.
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L2728]
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L2851
Because of which TXN_COMPONENTS & HIVE_LOCKS tables are not having entries from
proper partition values.
When query completes, the entry moves from TXN_COMPONENTS to
COMPLETED_TXN_COMPONENTS. Hive AutoCompaction will not recognize the partition
& considers it as invalid partition
create table abc(name string) partitioned by(city string) stored as orc
tblproperties('transactional'='true');
insert into abc partition(city='Bangalore') values('aaa');
Example entry in COMPLETED_TXN_COMPONENTS
{noformat}
+---+--++---+-+-+---+
| CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION |
CTC_TIMESTAMP | CTC_WRITEID | CTC_UPDATE_DELETE |
+---+--++---+-+-+---+
| 2 | default | abc | city=bangalore | 2020-11-25 09:26:59 |
1 | N |
+---+--++---+-+-+---+
{noformat}
AutoCompaction fails to get triggered with below error
2020-11-25T09:35:10,364 INFO [Thread-9]: compactor.Initiator
(Initiator.java:run(98)) - Checking to see if we should compact
default.abc.city=bangalore
2020-11-25T09:35:10,380 INFO [Thread-9]: compactor.Initiator
(Initiator.java:run(155)) - Can't find partition
default.compaction_test.city=bhubaneshwar, assuming it has been dropped and
moving on
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24381) compressed text input returns 0 rows if skip header/footer is mentioned.
Naresh P R created HIVE-24381: - Summary: compressed text input returns 0 rows if skip header/footer is mentioned. Key: HIVE-24381 URL: https://issues.apache.org/jira/browse/HIVE-24381 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R Attached q file returns 0 rows with hive.fetch.task.conversion=none, instead correct result is 2 rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24305) avro decimal schema is not properly populating scale/precision if enclosed in quote
Naresh P R created HIVE-24305:
-
Summary: avro decimal schema is not properly populating
scale/precision if enclosed in quote
Key: HIVE-24305
URL: https://issues.apache.org/jira/browse/HIVE-24305
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
{code:java}
CREATE TABLE test_quoted_scale_precision STORED AS AVRO TBLPROPERTIES
('avro.schema.literal'='{"type":"record","name":"DecimalTest","namespace":"com.example.test","fields":[{"name":"Decimal24_6","type":["null",{"type":"bytes","logicalType":"decimal","precision":24,"scale":"6"}]}]}');
desc test_quoted_scale_precision;
// decimal24_6 decimal(24,0)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24294) TezSessionPool sessions can throw AssertionError
Naresh P R created HIVE-24294:
-
Summary: TezSessionPool sessions can throw AssertionError
Key: HIVE-24294
URL: https://issues.apache.org/jira/browse/HIVE-24294
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Whenever default TezSessionPool sessions are reopened for some reason, we are
setting dagResources to null before close & setting it back in openWhenever
default TezSessionPool sessions are reopened for some reason, we are setting
dagResources to null before close & setting it back in open
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L498-L503
If there is an exception in sessionState.close(), we are not restoring the
dagResource but moving the session back to TezSessionPool.eg., exception trace
when sessionState.close() failed
{code:java}
2020-10-15T09:20:28,749 INFO [HiveServer2-Background-Pool: Thread-25451]:
client.TezClient (:()) - Failed to shutdown Tez Session via proxy
org.apache.tez.dag.api.SessionNotRunning: Application not running,
applicationId=application_1602093123456_12345, yarnApplicationState=FINISHED,
finalApplicationStatus=SUCCEEDED,
trackingUrl=http://localhost:8088/proxy/application_1602093123456_12345/,
diagnostics=Session timed out, lastDAGCompletionTime=1602997683786 ms,
sessionTimeoutInterval=60 ms
Session stats:submittedDAGs=2, successfulDAGs=2, failedDAGs=0, killedDAGs=0
at org.apache.tez.client.TezClientUtils.getAMProxy(TezClientUtils.java:910)
at org.apache.tez.client.TezClient.getAMProxy(TezClient.java:1060)
at org.apache.tez.client.TezClient.stop(TezClient.java:743)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:789)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:756)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.close(TezSessionPoolSession.java:111)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.reopenInternal(TezSessionPoolManager.java:496)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.reopen(TezSessionPoolManager.java:487)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.reopen(TezSessionPoolSession.java:228)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.getNewTezSessionOnError(TezTask.java:531)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:546)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:221){code}
Because of this, all new queries using this corrupted sessions are failing with
below exception
{code:java}
Caused by: java.lang.AssertionError: Ensure called on an unitialized (or
closed) session 41774265-b7da-4d58-84a8-1bedfd597aecCaused by:
java.lang.AssertionError: Ensure called on an unitialized (or closed) session
41774265-b7da-4d58-84a8-1bedfd597aec at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.ensureLocalResources(TezSessionState.java:685){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
cwiki write access
Hi, Can you please provide hive cwiki write access? user id : nareshpr --- Regards, Naresh P R
[jira] [Created] (HIVE-24282) Show columns shouldn't sort table output columns unless explicitly mentioned.
Naresh P R created HIVE-24282:
-
Summary: Show columns shouldn't sort table output columns unless
explicitly mentioned.
Key: HIVE-24282
URL: https://issues.apache.org/jira/browse/HIVE-24282
Project: Hive
Issue Type: Improvement
Reporter: Naresh P R
Assignee: Naresh P R
CREATE TABLE foo_n7(c INT, b INT, a INT);
show columns in foo_n7;
{code:java}
// current output
a
b
c
// expected
c
b
a
{code}
HIVE-18373 changed the original behaviour to sorted output.
Suggesting to provide an optional keyword sorted to sort the show columns output
eg.,
{code:java}
show sorted columns in foo_n7;
a
b
c
show columns in foo_n7
c
b
a
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24255) StorageHandler with select-limit query is returning 0 rows
Naresh P R created HIVE-24255:
-
Summary: StorageHandler with select-limit query is returning 0 rows
Key: HIVE-24255
URL: https://issues.apache.org/jira/browse/HIVE-24255
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
{code:java}
CREATE EXTERNAL TABLE test_table(db_id bigint, db_location_uri string, name
string, owner_name string, owner_type string)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES ('hive.sql.database.type'='METASTORE', 'hive.sql.query'='SELECT
`DB_ID`, `DB_LOCATION_URI`, `NAME`, `OWNER_NAME`, `OWNER_TYPE` FROM `DBS`');
==> Wrong Result <==
set hive.limit.optimize.enable=true;
select * from test_table limit 1;
--
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--
Map 1 .. container SUCCEEDED 0 0 0 0 0 0
--
VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 0.91 s
--
++--+---+-+-+
| dbs.db_id | dbs.db_location_uri | dbs.name | dbs.owner_name | dbs.owner_type |
++--+---+-+-+
++--+---+-+-+
==> Correct Result <==
set hive.limit.optimize.enable=false;
select * from test_table limit 1;
--
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 1 0 0 0 0
--
VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 4.11 s
--
+++---+-+-+
| dbs.db_id | dbs.db_location_uri | dbs.name | dbs.owner_name | dbs.owner_type |
+++---+-+-+
| 1 | hdfs://abcd:8020/warehouse/tablespace/managed/hive | default | public |
ROLE |
{code}
+++---+-+-+
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24210) PartitionManagementTask fails if one of tables dropped after fetch TableMeta
Naresh P R created HIVE-24210:
-
Summary: PartitionManagementTask fails if one of tables dropped
after fetch TableMeta
Key: HIVE-24210
URL: https://issues.apache.org/jira/browse/HIVE-24210
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
{code:java}
2020-09-21T10:45:15,875 ERROR [pool-4-thread-150]:
metastore.PartitionManagementTask (PartitionManagementTask.java:run(163)) -
Exception while running partition discovery task for table: null
org.apache.hadoop.hive.metastore.api.NoSuchObjectException:
hive.default.test_table table not found
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:3391)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3315)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3291)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy30.get_table_req(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1804)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1791)
at
org.apache.hadoop.hive.metastore.PartitionManagementTask.run(PartitionManagementTask.java:130){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24188) CTLT from MM to External fails because table txn properties are not skipped
Naresh P R created HIVE-24188:
-
Summary: CTLT from MM to External fails because table txn
properties are not skipped
Key: HIVE-24188
URL: https://issues.apache.org/jira/browse/HIVE-24188
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Repro steps
{code:java}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
create table test_mm(age int, name string) partitioned by(dept string) stored
as orc tblproperties('transactional'='true',
'transactional_properties'='default');
create external table test_external like test_mm LOCATION
'${system:test.tmp.dir}/create_like_mm_to_external';
{code}
Fails with below exception
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:default.test_external cannot be declared transactional
because it's an external table) (state=08S01,code=1){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24036) Kryo Exception while serializing plan for getSplits UDF call
Naresh P R created HIVE-24036:
-
Summary: Kryo Exception while serializing plan for getSplits UDF
call
Key: HIVE-24036
URL: https://issues.apache.org/jira/browse/HIVE-24036
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
{code:java}
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
"org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
class: org.apache.hadoop.hive.llap.LlapOutputFormatCaused by:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
"org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
class: org.apache.hadoop.hive.llap.LlapOutputFormatSerialization
trace:outputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)tableInfo
(org.apache.hadoop.hive.ql.plan.FileSinkDesc)conf
(org.apache.hadoop.hive.ql.exec.FileSinkOperator)childOperators
(org.apache.hadoop.hive.ql.exec.UnionOperator)childOperators
(org.apache.hadoop.hive.ql.exec.SelectOperator)childOperators
(org.apache.hadoop.hive.ql.exec.MapJoinOperator)childOperators
(org.apache.hadoop.hive.ql.exec.SelectOperator)childOperators
(org.apache.hadoop.hive.ql.exec.PTFOperator)childOperators
(org.apache.hadoop.hive.ql.exec.SelectOperator) at
org.apache.hadoop.hive.ql.exec.SerializationUtilities.serializeObjectByKryo(SerializationUtilities.java:700)
at
org.apache.hadoop.hive.ql.exec.SerializationUtilities.serializePlan(SerializationUtilities.java:571)
at
org.apache.hadoop.hive.ql.exec.SerializationUtilities.serializePlan(SerializationUtilities.java:560)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23894) SubmitDag should not be retried incase of query cancel
Naresh P R created HIVE-23894: - Summary: SubmitDag should not be retried incase of query cancel Key: HIVE-23894 URL: https://issues.apache.org/jira/browse/HIVE-23894 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R Incase of query cancel, running tasks will be interrupted & TezTask shutdown flag is will be set. Below code is not required to be retried incase of Task shutdown [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L572-L586] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23811) deleteReader SARG rowId is not getting validated properly
Naresh P R created HIVE-23811: - Summary: deleteReader SARG rowId is not getting validated properly Key: HIVE-23811 URL: https://issues.apache.org/jira/browse/HIVE-23811 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R Though we are iterating over min/max stripeIndex, we always seem to pick ColumnStats from first stripe [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java#L596] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23779) BasicStatsTask Info is not getting printed in beeline console
Naresh P R created HIVE-23779:
-
Summary: BasicStatsTask Info is not getting printed in beeline
console
Key: HIVE-23779
URL: https://issues.apache.org/jira/browse/HIVE-23779
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
After HIVE-16061, partition basic stats are not getting printed in beeline
console.
{code:java}
INFO : Partition {dt=2020-06-29} stats: [numFiles=21, numRows=22,
totalSize=14607, rawDataSize=0]{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23607) Permission Issue: Create view on another view succeeds but alter view fails
Naresh P R created HIVE-23607:
-
Summary: Permission Issue: Create view on another view succeeds
but alter view fails
Key: HIVE-23607
URL: https://issues.apache.org/jira/browse/HIVE-23607
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
create table test_table (id int);
create view test_view as select * from test_table;
{code:java}
-- user "naresh" as read access on test_view
-- Create view succeeds
create view test_view_1 as select * from test_view;
-- Alter view fails
alter view test_view_1 as select * from test_view
Error: Error while compiling statement: FAILED: HiveAccessControlException
Permission denied: user [naresh] does not have [SELECT] privilege on
[test/test_table] (state=42000,code=4)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23409) If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry
Naresh P R created HIVE-23409: - Summary: If TezSession application reopen fails for Timeline service down, default TezSession from SessionPool is closed after a retry Key: HIVE-23409 URL: https://issues.apache.org/jira/browse/HIVE-23409 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R we are closing a default session from TezSessionPool at here. [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L589] If all the sessions in a pool are destroyed, queries wait indefinitely at TezSessionPool.getSession until HS2 restarts after other service recoveries. [HiveServer2-Background-Pool: Thread-12345]: tez.TezSessionPoolManager (:()) - We are closing a default session because of retry failure. It's better if we allow retry & fail than hung. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23179) Show create table is not showing SerDe Properties in unicode
Naresh P R created HIVE-23179: - Summary: Show create table is not showing SerDe Properties in unicode Key: HIVE-23179 URL: https://issues.apache.org/jira/browse/HIVE-23179 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R Table with special character delimiters are not shown in show create output eg., create external table test(age int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\u0001' stored as textfile; Show create output ++ | createtab_stmt | ++ | CREATE EXTERNAL TABLE `test`(| | `age` int, | | `name` string) | | ROW FORMAT SERDE | | 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' | | WITH SERDEPROPERTIES ( | | 'field.delim'='', | | 'serialization.format'='') | | STORED AS INPUTFORMAT | | 'org.apache.hadoop.mapred.TextInputFormat' | | OUTPUTFORMAT | | 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' | | LOCATION | | 'hdfs://abcd:8020/warehouse/tablespace/external/hive/testca' | | TBLPROPERTIES (| | 'bucketing_version'='2', | | 'discover.partitions'='true',| | 'transient_lastDdlTime'='1577162310')| ++ Few client console not able to show ^A (Ctrl + A) properly. It's better to show the output in unicode as shown in desc formatted. | Storage Desc Params: | NULL | NULL | | | field.delim | \u0001 | | | serialization.format | \u0001 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23173) User login success/failed attempts should be logged
Naresh P R created HIVE-23173: - Summary: User login success/failed attempts should be logged Key: HIVE-23173 URL: https://issues.apache.org/jira/browse/HIVE-23173 Project: Hive Issue Type: Improvement Reporter: Naresh P R User login success & failure attempts should be logged in server logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22478) Import command fails from lower version to higher version when hive.strict.managed.tables enabled
Naresh P R created HIVE-22478:
-
Summary: Import command fails from lower version to higher version
when hive.strict.managed.tables enabled
Key: HIVE-22478
URL: https://issues.apache.org/jira/browse/HIVE-22478
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Created non-acid managed orc table in lower version, after inserting some
records, exported the table.
In higher version where hive.strict.managed.enabled=true,
1) on first attempt, ACID Table is getting created, but LoadTable is failing
with below exception
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MoveTask : Write
id is not set in the config by open txn task for migration
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:400)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2712)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2383)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2055)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1753)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1747)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226){code}
2) On second attempt, as the table is already exist as ACID,
ImportSemanticAnalyzer is creating writeId for the ACID table & LoadTable
command is successful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22331) unix_timestamp without argument returns timestamp in millisecond instead of second.
Naresh P R created HIVE-22331: - Summary: unix_timestamp without argument returns timestamp in millisecond instead of second. Key: HIVE-22331 URL: https://issues.apache.org/jira/browse/HIVE-22331 Project: Hive Issue Type: Bug Reporter: Naresh P R Assignee: Naresh P R After HIVE-22170, select unix_timestamp(); is returning milliseconds, but expected output is in seconds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22178) Parquet FilterPredicate throws CastException after SchemaEvolution.
Naresh P R created HIVE-22178:
-
Summary: Parquet FilterPredicate throws CastException after
SchemaEvolution.
Key: HIVE-22178
URL: https://issues.apache.org/jira/browse/HIVE-22178
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Naresh P R
Assignee: Naresh P R
Below are the repro steps.
{code:java}
create table parq_test(age int, name string) stored as parquet;
insert into parq_test values(1, '');
alter table parq_test change age age string;
insert into parq_test values('b', '');
select * from parq_test where age='b';{code}
Exception thrown after changing column datatype is below
{code:java}
Caused by: java.lang.IllegalArgumentException: FilterPredicate column: age's
declared type (org.apache.parquet.io.api.Binary) does not match the schema
found in file metadata. Column age is of type: INT32
Valid types for this column are: [class java.lang.Integer]
at
org.apache.parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:126)
at
org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:181)
at
org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:151)
at
org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:85)
at
org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:58)
at org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:181)
at
org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:63)
at
org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:92)
at
org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:43)
at
org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:137)
at
org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:64)
at
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:111)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:147)
... 31 more{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Re: [ANNOUNCE] New committer: Rajkumar Singh
Congrats Raj!!! -- Regards, Naresh P R On Fri, Jul 26, 2019 at 9:23 AM Ashutosh Chauhan wrote: > Apache Hive's Project Management Committee (PMC) has invited Rajkumar Singh > to become a committer, and we are pleased to announce that he has accepted. > > Raj welcome, thank you for your contributions, and we look forward your > further interactions with the community! > > Ashutosh Chauhan (on behalf of the Apache Hive PMC) >
[jira] [Created] (HIVE-22009) CTLV with user specified location is not honoured
Naresh P R created HIVE-22009:
-
Summary: CTLV with user specified location is not honoured
Key: HIVE-22009
URL: https://issues.apache.org/jira/browse/HIVE-22009
Project: Hive
Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Naresh P R
Assignee: Naresh P R
Steps to repro :
{code:java}
CREATE TABLE emp_table (id int, name string, salary int);
insert into emp_table values(1,'a',2);
CREATE VIEW emp_view AS SELECT * FROM emp_table WHERE salary>1;
CREATE EXTERNAL TABLE emp_ext_table like emp_view LOCATION '/tmp/emp_ext_table';
show create table emp_ext_table;{code}
{code:java}
++
| createtab_stmt |
++
| CREATE EXTERNAL TABLE `emp_ext_table`( |
| `id` int, |
| `name` string, |
| `salary` int) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'hdfs://nn:8020/warehouse/tablespace/external/hive/emp_ext_table' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'transient_lastDdlTime'='1563467962') |
++{code}
Table Location is not '/tmp/emp_ext_table', instead location is set to default
warehouse path.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (HIVE-21852) Testcases are randomly failing with ArrayIndexOutOfBoundException
Naresh P R created HIVE-21852:
-
Summary: Testcases are randomly failing with
ArrayIndexOutOfBoundException
Key: HIVE-21852
URL: https://issues.apache.org/jira/browse/HIVE-21852
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
This is randomly failing with ArrayIndexOutOfBoundException in this lines
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L75-L83]
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException at
java.lang.System.arraycopy(Native Method) ~[?:1.8.0_102] at
java.util.ArrayList.addAll(ArrayList.java:580) ~[?:1.8.0_102] at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.(MetastoreDefaultTransformer.java:75)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21837) MapJoin is throwing exception when selected column is having completely null values
Naresh P R created HIVE-21837:
-
Summary: MapJoin is throwing exception when selected column is
having completely null values
Key: HIVE-21837
URL: https://issues.apache.org/jira/browse/HIVE-21837
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
With huge data, below query
{code:java}
SELECT
a.id,
null
FROM table a
JOIN table b ON (a.id = b.id)
JOIN (SELECT '1234' id, 'deptno' type UNION ALL SELECT '5678' id, 'deptno'
type) c ON (c.type = a.type){code}
is throwing below exception.
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Column vector
class org.apache.hadoop.hive.ql.exec.vector.VoidColumnVector is not supported!
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.makeLikeColumnVector(VectorizedBatchUtil.java:624)
at
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.makeLike(VectorizedBatchUtil.java:861)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.commonSetup(VectorMapJoinCommonOperator.java:583)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.commonSetup(VectorMapJoinGenerateResultOperator.java:128)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.commonSetup(VectorMapJoinInnerGenerateResultOperator.java:112)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerMultiKeyOperator.process(VectorMapJoinInnerMultiKeyOperator.java:119)
... 21 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20599) CAST(INTERVAL_DAY_TIME AS STRING) is throwing SemanticException
Naresh P R created HIVE-20599:
-
Summary: CAST(INTERVAL_DAY_TIME AS STRING) is throwing
SemanticException
Key: HIVE-20599
URL: https://issues.apache.org/jira/browse/HIVE-20599
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 3.1.0
Reporter: Naresh P R
Assignee: Naresh P R
Fix For: 3.1.0
SELECT CAST(from_utc_timestamp(timestamp '2018-05-02 15:30:30', 'PST') -
from_utc_timestamp(timestamp '1970-01-30 16:00:00', 'PST') AS STRING);
throws below Exception
{code:java}
Error: Error while compiling statement: FAILED: SemanticException Line 0:-1
Wrong arguments ''PST'': No matching method for class
org.apache.hadoop.hive.ql.udf.UDFToString with (interval_day_time). Possible
choices: _FUNC_(bigint) _FUNC_(binary) _FUNC_(boolean) _FUNC_(date)
_FUNC_(decimal(38,18)) _FUNC_(double) _FUNC_(float) _FUNC_(int)
_FUNC_(smallint) _FUNC_(string) _FUNC_(timestamp) _FUNC_(tinyint)
_FUNC_(void) (state=42000,code=4){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-19090) Running concatenate on ORC tables either increase or decrease the number of files depends on the order of file being picked
Naresh P R created HIVE-19090: - Summary: Running concatenate on ORC tables either increase or decrease the number of files depends on the order of file being picked Key: HIVE-19090 URL: https://issues.apache.org/jira/browse/HIVE-19090 Project: Hive Issue Type: Bug Components: ORC Reporter: Naresh P R I ran concatenate 2 times without changing any config For the 1st run, 14 files merged to 8 files For the 2nd run, 8 files expanded to 10 files >From logs i could see Input files are same, whereas output splits from >CombineHiveInputFormat are varying depending upon the file being picked first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18112) show create for view having special char in where clause is not showing properly
Naresh P R created HIVE-18112: - Summary: show create for view having special char in where clause is not showing properly Key: HIVE-18112 URL: https://issues.apache.org/jira/browse/HIVE-18112 Project: Hive Issue Type: Bug Reporter: Naresh P R Priority: Minor Fix For: 2.3.2 e.g., CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` where `evil_byte1`.`a` = 'abcÖdefÖgh'; Output: == 0: jdbc:hive2://172.26.122.227:1> show create table v2; ++--+ | createtab_stmt | ++--+ | CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` where `evil_byte1`.`a` = 'abc�def�gh' | ++--+ Only show create output is having invalid characters, actual source table content is displayed properly in the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17845) insert fails if target table columns are not lowercase
Naresh P R created HIVE-17845: - Summary: insert fails if target table columns are not lowercase Key: HIVE-17845 URL: https://issues.apache.org/jira/browse/HIVE-17845 Project: Hive Issue Type: Bug Affects Versions: 1.2.1 Reporter: Naresh P R Assignee: Naresh P R Priority: Minor Fix For: 2.3.0 eg., INSERT INTO TABLE EMP(ID,NAME) select * FROM SRC; FAILED: SemanticException 1:27 '[ID,NAME]' in insert schema specification are not found among regular columns of default.EMP nor dynamic partition columns.. Error encountered near token 'NAME' Whereas below insert is successful: INSERT INTO TABLE EMP(id,name) select * FROM SRC; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
