[jira] [Created] (HIVE-21564) Load data into a bucketed table is ignoring partitions specs and loading data into default partition.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-21564:
---

 Summary: Load data into a bucketed table is ignoring partitions 
specs and loading data into default partition.
 Key: HIVE-21564
 URL: https://issues.apache.org/jira/browse/HIVE-21564
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 4.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


When running below command to load data into bucketed tables it is not loading 
into specified partition instead loaded into default partition.
LOAD DATA INPATH '/tmp/files/00_0' OVERWRITE INTO TABLE call 
PARTITION(year_partition=2012, month=12);

SELECT * FROM call WHERE year_partition=2012 AND month=12; --> returns 0 rows.

{code}
CREATE TABLE call( 
date_time_date date, 
ssn string, 
name string, 
location string) 
PARTITIONED BY ( 
year_partition int, 
month int) 
CLUSTERED BY ( 
date_time_date) 
SORTED BY ( 
date_time_date ASC) 
INTO 1 BUCKETS 
STORED AS ORC;
{code}

If set hive.exec.dynamic.partition to false, it fails with below error.
{code}
Error: Error while compiling statement: FAILED: SemanticException 1:18 Dynamic 
partition is disabled. Either enable it by setting 
hive.exec.dynamic.partition=true or specify partition column values. Error 
encountered near token 'month' (state=42000,code=4)
{code}

When we "set hive.strict.checks.bucketing=false;", the load works fine.
This is a behaviour imposed by HIVE-15148 to avoid incorrectly named data files 
being loaded to the bucketed tables. In customer use case, if the files are 
named properly with bucket_id (0_0, 0_1 etc), then it is safe to set 
this flag to false.
However, current behaviour of loading into default partitions when 
hive.strict.checks.bucketing=true and partitions specified, was a bug injected 
by HIVE-19311 where the given query is re-written into a insert query (to 
handle incorrect file names and Orc versions) but missed to incorporate the 
partitions specs to it. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21563) Improve Table#getEmptyTable performance by disable registerAllFunctionsOnce

[Yuming Wang (JIRA)]

Yuming Wang created HIVE-21563:
--

 Summary: Improve Table#getEmptyTable performance by disable 
registerAllFunctionsOnce
 Key: HIVE-21563
 URL: https://issues.apache.org/jira/browse/HIVE-21563
 Project: Hive
  Issue Type: Improvement
Reporter: Yuming Wang
Assignee: Yuming Wang


We do not need registerAllFunctionsOnce when {{Table#getEmptyTable}}. The stack 
trace:
{noformat}
  at 
org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:177)
  at 
org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:170)
  at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:209)
  at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:247)
  at 
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
  at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:388)
  at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
  at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
  at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
  at 
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:913)
  at 
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:877)
  at 
org.apache.hadoop.hive.ql.session.SessionState.getAuthenticator(SessionState.java:1479)
  at 
org.apache.hadoop.hive.ql.session.SessionState.getUserFromAuthenticator(SessionState.java:1150)
  at org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(Table.java:180)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21562) Ranger should add service admin privilege support for hive service objects - LLAP command sets

[Ramesh Mani (JIRA)]

Ramesh Mani created HIVE-21562:
--

 Summary: Ranger should add service admin privilege support for 
hive service objects - LLAP command sets
 Key: HIVE-21562
 URL: https://issues.apache.org/jira/browse/HIVE-21562
 Project: Hive
  Issue Type: Bug
Reporter: Ramesh Mani


Ranger should add service admin privilege support for hive service objects - 
LLAP command sets
Functionality for workload management commands being added in Hive with 
HIVE-17481  and HIVE-19033



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21561) Revert removal of TableType.INDEX_TABLE enum

[Jason Dere (JIRA)]

Jason Dere created HIVE-21561:
-

 Summary: Revert removal of TableType.INDEX_TABLE enum
 Key: HIVE-21561
 URL: https://issues.apache.org/jira/browse/HIVE-21561
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
 Attachments: HIVE-21561.1.patch

Index tables have been removed from Hive as of HIVE-18715.
However, in case users still have index tables defined in the metastore, we 
should keep the TableType.INDEX_TABLE enum around so that users can drop these 
tables. Without the enum defined Hive cannot do anything with them as it fails 
with IllegalArgumentException errors when trying to call TableType.valueOf() on 
INDEX_TABLE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

HIVE-18624 SQL parser performance bug

[Julian Hyde]

HIVE-18624 [1] is a serious performance bug in the SQL parser. It causes parse 
times that are literally exponential in the number of parentheses in the 
expression, thus parsing of a query that has complex expressions may take 
minutes or not terminate. According to JIRA, the bug was fixed on 2.4.0, 3.1.0, 
4.0.0 code lines in August but has not yet been released.

I work for Looker, a BI tool that generates SQL with deeply nested expressions, 
and therefore they hit this bug. Hive 2.2, 2.3 and 3.0 are unusable for our 
customers due to this bug.

I do not know the schedule for 2.4.0, 3.1.0 or 4.0.0 releases, but if they are 
a way off, would it be possible to fix this bug in a patch release?

Julian

[1] https://issues.apache.org/jira/browse/HIVE-18624 

[jira] [Created] (HIVE-21560) Update Derby DDL to use CLOB instead of LONG VARCHAR

[Shawn Weeks (JIRA)]

Shawn Weeks created HIVE-21560:
--

 Summary: Update Derby DDL to use CLOB instead of LONG VARCHAR
 Key: HIVE-21560
 URL: https://issues.apache.org/jira/browse/HIVE-21560
 Project: Hive
  Issue Type: Bug
Reporter: Shawn Weeks


in the Hive 1.x and 2.x metastore version for Derby there are two column in 
"TBLS" that are set to LONG VARCHAR. This causes larger create view statements 
to fail when using embedded metastore for testing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21559) [performance] select distinct partitioncolumn from external table takes 10s rather than pull from HMS

[t oo (JIRA)]

t oo created HIVE-21559:
---

 Summary: [performance] select distinct partitioncolumn from 
external table takes 10s rather than pull from HMS
 Key: HIVE-21559
 URL: https://issues.apache.org/jira/browse/HIVE-21559
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: t oo


select distinct partitioncolumn from external table with only 3 partitions (on 
s3a) takes 10 seconds, why can't this pull from HiveMetastore partitions in < 2 
seconds?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21558) Query based compaction fails if the temporary FS is different than the table FS

[Peter Vary (JIRA)]

Peter Vary created HIVE-21558:
-

 Summary: Query based compaction fails if the temporary FS is 
different than the table FS
 Key: HIVE-21558
 URL: https://issues.apache.org/jira/browse/HIVE-21558
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Peter Vary
Assignee: Peter Vary


The Exception I got is like this:
{code:java}
2019-04-01T13:45:44,035 ERROR [PeterVary-MBP15.local-33] compactor.Worker: 
Caught exception while trying to compact 
id:24,dbname:default,tableName:acid,partName:null,state:,type:MAJOR,properties:null,runAs:petervary,tooManyAborts:false,highestWriteId:9.
 Marking failed to avoid repeated failures, java.lang.IllegalArgumentException: 
Wrong FS: 
pfile:/Users/petervary/data/apache/hive/warehouse/acid/base_009_v284/bucket_0,
 expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
at 
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
at 
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
at 
org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem.rename(ProxyLocalFileSystem.java:34)
at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.commitCrudMajorCompaction(CompactorMR.java:583)
at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runCrudCompaction(CompactorMR.java:401)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:248)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:195){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21557) Query based compaction fails with NullPointerException: Non-local session path expected to be non-null

[Peter Vary (JIRA)]

Peter Vary created HIVE-21557:
-

 Summary: Query based compaction fails with NullPointerException: 
Non-local session path expected to be non-null
 Key: HIVE-21557
 URL: https://issues.apache.org/jira/browse/HIVE-21557
 Project: Hive
  Issue Type: Bug
Reporter: Peter Vary


{code:java}
2019-03-29T13:04:19.282Z hiveserver2-65d5bb4bd8-xx24r hiveserver2 1 
db896a5e-5215-11e9-87ec-020c4712c37c [mdc@18060 class="compactor.CompactorMR" 
level="ERROR" thread="hiveserver2-65d5bb4bd8-xx24r-28"] 
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run drop table if 
exists default_tmp_compactor_asd_1553864659196
at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:57)
at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:34)
at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runCrudCompaction(CompactorMR.java:408)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:250)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:194)
Caused by: java.lang.NullPointerException: Non-local session path expected to 
be non-null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:228)
at 
org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:838)
at org.apache.hadoop.hive.ql.Context.(Context.java:319)
at org.apache.hadoop.hive.ql.Context.(Context.java:305)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:603)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1881)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1764)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1753)
at org.apache.hadoop.hive.ql.DriverUtils.runOnDriver(DriverUtils.java:54){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21556) Useless configuration for old jetty in log4j.properties

[Chen Zhi (JIRA)]

Chen Zhi created HIVE-21556:
---

 Summary: Useless configuration for old jetty in log4j.properties
 Key: HIVE-21556
 URL: https://issues.apache.org/jira/browse/HIVE-21556
 Project: Hive
  Issue Type: Improvement
  Components: Configuration
Reporter: Chen Zhi


We has upgrade to jetty 9 in 
[HIVE-16049](https://issues.apache.org/jira/browse/HIVE-16049), the 
configuration `org.mortbay` in log4j.properties for old version of jetty is 
useless. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)