[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP reconnect to already running Yarn app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Description:
Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
llap yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't (re)connecting and making use of the running llap app.
It's also possible that the containers were simply slow to allocate as the
cluster was operating at 100% capacity and therefore weren't fully initialized
when the app failed, but the error feedback doesn't give enough details about
the state of the llap0 app.
was:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't reconnecting and making use of the running llap app.
> HiveServer2 Interactive LLAP reconnect to already running Yarn app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't (re)connecting and making use of the running llap app.
> It's also possible that the containers were simply slow to allocate as the
> cluster was operating at 100% capacity and therefore weren't fully
> initialized when the app failed, but the error feedback doesn't give enough
> details about the state of the llap0 app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP (re)connect to already running Yarn llap0 app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Summary: HiveServer2 Interactive LLAP (re)connect to already running Yarn
llap0 app (was: HiveServer2 Interactive LLAP reconnect to already running Yarn
app)
> HiveServer2 Interactive LLAP (re)connect to already running Yarn llap0 app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't (re)connecting and making use of the running llap app.
> It's also possible that the containers were simply slow to allocate as the
> cluster was operating at 100% capacity and therefore weren't fully
> initialized when the app failed, but the error feedback doesn't give enough
> details about the state of the llap0 app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP reconnect to already running Yarn app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Description:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't reconnecting and making use of the running llap app.
was:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
> HiveServer2 Interactive LLAP reconnect to already running Yarn app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to reconnect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't reconnecting and making use of the running llap app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-7782) tez default engine not overridden by hive.execution.engine=mr in hive cli session
[
https://issues.apache.org/jira/browse/HIVE-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-7782:
--
Description:
I've deployed hive.execution.engine=tez as the default on my secondary HDP
cluster I find that hive cli interactive sessions where I do
{code:java}
set hive.execution.engine=mr
{code}
still execute with Tez as shown in the Resource Manager applications view. Now
this may make sense since it's connected a Tez session by that point but it's
also misleading because the job progress output in the cli changes to look like
MapReduce rather than Tez and the query time is increased from 8 to to 15-16
secs but still less than the 25-30+ secs I usually see with MR. The Resource
Manager shows both of these jobs as TEZ application type regardless of setting
hive.execution.engine=mr. Is this a bug in the way Hive is submitting the job
(Tez vs MR) or a bug in the way the RM is reporting it?
{code:java}
hive
Logging initialized using configuration in
file:/etc/hive/conf.dist/hive-log4j.properties
hive> select count(*) from sample_07;
Query ID = hari_20140819164848_c03824c7-0e76-4507-b619-6a22cb0fbc4c
Total jobs = 1
Launching Job 1 out of 1
Status: Running (application id: application_1408444369445_0031)
Map 1: -/- Reducer 2: 0/1
Map 1: 0/1 Reducer 2: 0/1
Map 1: 0/1 Reducer 2: 0/1
Map 1: 1/1 Reducer 2: 0/1
Map 1: 1/1 Reducer 2: 1/1
Status: Finished successfully
OK
823
Time taken: 8.492 seconds, Fetched: 1 row(s)
hive> set hive.execution.engine=mr;
hive> select count(*) from sample_07;
Query ID = hari_20140819164848_b620d990-b405-479c-be5b-d9616527cefe
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
Starting Job = job_1408444369445_0032, Tracking URL =
http://:8088/proxy/application_1408444369445_0032/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1408444369445_0032
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2014-08-19 16:48:35,242 Stage-1 map = 0%, reduce = 0%
2014-08-19 16:48:40,539 Stage-1 map = 100%, reduce = 0%
2014-08-19 16:48:44,676 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1408444369445_0032
MapReduce Jobs Launched:
Job 0: HDFS Read: 0 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
823
Time taken: 16.579 seconds, Fetched: 1 row(s)
{code}
If I exit hive shell and restart it instead using
{code:java}
--hiveconf hive.execution.engine=mr{code}
to set before session is established then it does a proper MapReduce job
according to RM and it also takes the longer expected 25 secs instead of the 8
in Tez or 15 in trying to do MR instead Tez session.
was:
I've deployed hive.execution.engine=tez as the default on my secondary HDP
cluster I find that hive cli interactive sessions where I do
{code}
set hive.execution.engine=mr
{code}
still execute with Tez as shown in the Resource Manager applications view. Now
this may make sense since it's connected a Tez session by that point but it's
also misleading because the job progress output in the cli changes to look like
MapReduce rather than Tez and the query time is increased from 8 to to 15-16
secs but still less than the 25-30+ secs I usually see with MR. The Resource
Manager shows both of these jobs as TEZ application type regardless of setting
hive.execution.engine=mr. Is this a bug in the way Hive is submitting the job
(Tez vs MR) or a bug in the way the RM is reporting it?
{code}
hive
Logging initialized using configuration in
file:/etc/hive/conf.dist/hive-log4j.properties
hive> select count(*) from sample_07;
Query ID = hari_20140819164848_c03824c7-0e76-4507-b619-6a22cb0fbc4c
Total jobs = 1
Launching Job 1 out of 1
Status: Running (application id: application_1408444369445_0031)
Map 1: -/- Reducer 2: 0/1
Map 1: 0/1 Reducer 2: 0/1
Map 1: 0/1 Reducer 2: 0/1
Map 1: 1/1 Reducer 2: 0/1
Map 1: 1/1 Reducer 2: 1/1
Status: Finished successfully
OK
823
Time taken: 8.492 seconds, Fetched: 1 row(s)
hive> set hive.execution.engine=mr;
hive> select count(*) from sample_07;
Query ID = hari_20140819164848_b620d990-b405-479c-be5b-d9616527cefe
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
Starting Job = job_1408444369445_0032, Tracking URL =
http://lonsl1101827-data.uk.net.intra:8088/proxy/application_1408444369445_0032/
Kill
[jira] [Commented] (HIVE-12349) NPE in ORC SARG for IS NULL queries on Timestamp and Date columns
[
https://issues.apache.org/jira/browse/HIVE-12349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037708#comment-15037708
]
Hari Sekhon commented on HIVE-12349:
Any ideas on the timeline for Hive 1.3 release including this fix? This problem
is currently affecting my users so we're eager to get a new release of Hive.
Hortonworks HDP 2.4 is due to be released some time this or next month based on
the bi-annual release schedule, would be good to get Hive 1.3 included in that
release.
> NPE in ORC SARG for IS NULL queries on Timestamp and Date columns
> -
>
> Key: HIVE-12349
> URL: https://issues.apache.org/jira/browse/HIVE-12349
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1, 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0, 2.0.0
>
>
> IS NULL queries can trigger an NPE for timestamp and date columns. All column
> values per row group or stripe should be NULL to trigger this case. Following
> is the exception stack trace
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$TimestampStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:795)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getMin(RecordReaderImpl.java:2343)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:2366)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:2564)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2627)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:3060)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:3102)
>
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:288)
>
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:534)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
>
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
>
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
>
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1141)
>
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1039)
>
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:246)
>
> ... 26 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12241) SQLStdAuth grant on .* not recognized
[
https://issues.apache.org/jira/browse/HIVE-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987372#comment-14987372
]
Hari Sekhon commented on HIVE-12241:
Agreed Ranger is nice but not everybody has Ranger, I have it on the other
cluster but haven't deployed it to this one yet, and other distributions don't
even have Ranger yet.
>From Hive's perspective this should be solvable using native SQL grants.
The whole point of SQLStdAuth was to be similar to grants in RDBMS and since
they can do grant on .* then Hive should support this syntax also.
> SQLStdAuth grant on .* not recognized
> -
>
> Key: HIVE-12241
> URL: https://issues.apache.org/jira/browse/HIVE-12241
> Project: Hive
> Issue Type: Bug
> Components: Parser, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2
>Reporter: Hari Sekhon
>
> Using SQLStdAuthorizer Hive doesn't recognize doing a grant on all tables
> like I've done before in RDBMS. If having a lot of tables this becomes very
> inconvenient to grant on a table-by-table basis and granting on database
> succeeds but still doesn't allow user to query tables in that database:
> {code}
> > grant all on myDB.* to user hari;
> Error: Error while compiling statement: FAILED: ParseException line 1:15
> mismatched input '.' expecting TO near 'myDB' in grant privileges
> (state=42000,code=4)
> > grant all on myDB.`*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on `myDB.*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on all to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.all (state=42S02,code=10001)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive') (was: HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive')) > HiveServer2 put username in job name when doAs=false (for viewing in RM > otherwise all jobs show as 'hive') > -- > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job when doAs=false for viewing in RM
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job when doAs=false for viewing in RM (was: Hive when not doAs put username in job name for viewing in RM) > HiveServer2 put username in job when doAs=false for viewing in RM > - > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Summary: HiveServer2 put username in job when doAs=false (for viewing in RM otherwise all jobs show as 'hive') (was: HiveServer2 put username in job when doAs=false for viewing in RM) > HiveServer2 put username in job when doAs=false (for viewing in RM otherwise > all jobs show as 'hive') > - > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12240) HiveServer2 put username in job name when doAs=false (for viewing in RM otherwise all jobs show as 'hive')
[ https://issues.apache.org/jira/browse/HIVE-12240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12240: --- Description: When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the username in the HIVE- job submission so that we can see which user submitted which job in the RM queues as all the jobs just show as 'hive' user (this is running on Tez, so the query isn't shown in the name column making differentiating queries/job difficult). (was: When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the username in the HIVE- job submission so that we can see which user submitted which job in the RM queues as all the jobs just show as 'hive' user.) > HiveServer2 put username in job name when doAs=false (for viewing in RM > otherwise all jobs show as 'hive') > -- > > Key: HIVE-12240 > URL: https://issues.apache.org/jira/browse/HIVE-12240 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > When using HiveServer2 with SQLStdAuthorizer and doAs=false please put the > username in the HIVE- job submission so that we can see which user > submitted which job in the RM queues as all the jobs just show as 'hive' user > (this is running on Tez, so the query isn't shown in the name column making > differentiating queries/job difficult). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12242) SQLStdAuth add ability to map Roles to existing Groups
[ https://issues.apache.org/jira/browse/HIVE-12242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12242: --- Summary: SQLStdAuth add ability to map Roles to existing Groups (was: SQLStdAuth map Roles to existing Groups) > SQLStdAuth add ability to map Roles to existing Groups > -- > > Key: HIVE-12242 > URL: https://issues.apache.org/jira/browse/HIVE-12242 > Project: Hive > Issue Type: New Feature > Components: SQLStandardAuthorization >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > Feature request to be able to map Hive roles to groups so that LDAP groups > can be reused rather than having to recreate all the information in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6674) "show grant on all" throws NPE
[
https://issues.apache.org/jira/browse/HIVE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-6674:
--
Affects Version/s: (was: 0.14.0)
> "show grant on all" throws NPE
> --
>
> Key: HIVE-6674
> URL: https://issues.apache.org/jira/browse/HIVE-6674
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Navis
> Fix For: 0.13.0
>
> Attachments: HIVE-6674.1.patch.txt, HIVE-6674.2.patch.txt
>
>
> "show grant on all" is supposed to show all the grants in the system on all
> the objects. But it fails with NPE with both SQL standard auth, and legacy
> auth.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalAllDBGrant(ObjectStore.java:4206)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalDBGrantsAll(ObjectStore.java:4169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy6.listPrincipalDBGrantsAll(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.list_db_privileges(HiveMetaStore.java:4295)
> ... 36 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12242) SQLStdAuth map Roles to existing Groups
[ https://issues.apache.org/jira/browse/HIVE-12242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-12242: --- Summary: SQLStdAuth map Roles to existing Groups (was: SQLStdAuth map roles to groups (ldap)) > SQLStdAuth map Roles to existing Groups > --- > > Key: HIVE-12242 > URL: https://issues.apache.org/jira/browse/HIVE-12242 > Project: Hive > Issue Type: New Feature > Components: SQLStandardAuthorization >Affects Versions: 0.14.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > > Feature request to be able to map Hive roles to groups so that LDAP groups > can be reused rather than having to recreate all the information in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Component/s: SQLStandardAuthorization
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs (despite "should use beeline", an NPE is still a bug):
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12241) SQLStdAuth grant on .* not recognized
[
https://issues.apache.org/jira/browse/HIVE-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12241:
---
Component/s: SQLStandardAuthorization
> SQLStdAuth grant on .* not recognized
> -
>
> Key: HIVE-12241
> URL: https://issues.apache.org/jira/browse/HIVE-12241
> Project: Hive
> Issue Type: Bug
> Components: Parser, SQLStandardAuthorization
>Affects Versions: 0.14.0
> Environment: HDP 2.2
>Reporter: Hari Sekhon
>
> Using SQLStdAuthorizer Hive doesn't recognize doing a grant on all tables
> like I've done before in RDBMS. If having a lot of tables this becomes very
> inconvenient to grant on a table-by-table basis and granting on database
> succeeds but still doesn't allow user to query tables in that database:
> {code}
> > grant all on myDB.* to user hari;
> Error: Error while compiling statement: FAILED: ParseException line 1:15
> mismatched input '.' expecting TO near 'myDB' in grant privileges
> (state=42000,code=4)
> > grant all on myDB.`*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on `myDB.*` to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.* (state=42S02,code=10001)
> > grant all on all to user hari;
> Error: Error while compiling statement: FAILED: SemanticException [Error
> 10001]: Table not found myDB.all (state=42S02,code=10001)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6674) "show grant on all" throws NPE
[
https://issues.apache.org/jira/browse/HIVE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-6674:
--
Affects Version/s: 0.14.0
> "show grant on all" throws NPE
> --
>
> Key: HIVE-6674
> URL: https://issues.apache.org/jira/browse/HIVE-6674
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Thejas M Nair
>Assignee: Navis
> Fix For: 0.13.0
>
> Attachments: HIVE-6674.1.patch.txt, HIVE-6674.2.patch.txt
>
>
> "show grant on all" is supposed to show all the grants in the system on all
> the objects. But it fails with NPE with both SQL standard auth, and legacy
> auth.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalAllDBGrant(ObjectStore.java:4206)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.listPrincipalDBGrantsAll(ObjectStore.java:4169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy6.listPrincipalDBGrantsAll(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.list_db_privileges(HiveMetaStore.java:4295)
> ... 36 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Component/s: HiveServer2
Hive
Authorization
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Critical
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs:
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Component/s: (was: HiveServer2)
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Critical
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs:
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Summary: NullPointerException StdSQLAuthorizer showing grants via Hive CLI
(was: NullPointerException when trying to get grants with StdSQLAuthorizer via
Hive CLI)
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Critical
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs:
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Affects Version/s: 0.14.0
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive
>Affects Versions: 0.14.0
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Critical
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs:
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Description:
When trying to see the grants in Hive CLI the following NullPointerException
bug occurs (despite "should use beeline", an NPE is still a bug):
{code}
2015-10-22 16:36:31,535 INFO [main]: ql.Driver (Driver.java:launchTask(1604))
- Starting task [Stage-0:DDL] in serial mode
2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511)) -
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
Error showing privileges: null
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
... 23 more
{code}
was:
When trying to see the grants in Hive CLI the following NullPointerException
bug occurs:
{code}
2015-10-22 16:36:31,535 INFO [main]: ql.Driver (Driver.java:launchTask(1604))
- Starting task [Stage-0:DDL] in serial mode
2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511)) -
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
Error showing privileges: null
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
at
[jira] [Updated] (HIVE-12233) NullPointerException StdSQLAuthorizer showing grants via Hive CLI
[
https://issues.apache.org/jira/browse/HIVE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-12233:
---
Priority: Major (was: Critical)
> NullPointerException StdSQLAuthorizer showing grants via Hive CLI
> -
>
> Key: HIVE-12233
> URL: https://issues.apache.org/jira/browse/HIVE-12233
> Project: Hive
> Issue Type: Bug
> Components: Authorization, Hive
>Affects Versions: 0.14.0
> Environment: HDP 2.2 + Kerberos
>Reporter: Hari Sekhon
>
> When trying to see the grants in Hive CLI the following NullPointerException
> bug occurs (despite "should use beeline", an NPE is still a bug):
> {code}
> 2015-10-22 16:36:31,535 INFO [main]: ql.Driver
> (Driver.java:launchTask(1604)) - Starting task [Stage-0:DDL] in serial mode
> 2015-10-22 16:36:31,536 ERROR [main]: exec.DDLTask (DDLTask.java:failed(511))
> -
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException:
> Error showing privileges: null
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPluginException(SQLAuthorizationUtils.java:419)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:445)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.showPrivileges(SQLStdHiveAccessControllerWrapper.java:141)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.showPrivileges(HiveAuthorizerImpl.java:96)
> at org.apache.hadoop.hive.ql.exec.DDLTask.showGrants(DDLTask.java:649)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:465)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1606)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1367)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1006)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:996)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.isUserAdmin(SQLStdHiveAccessController.java:561)
> at
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.showPrivileges(SQLStdHiveAccessController.java:387)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9544) Error dropping fully qualified partitioned table - Internal error processing get_partition_names
[
https://issues.apache.org/jira/browse/HIVE-9544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14714424#comment-14714424
]
Hari Sekhon commented on HIVE-9544:
---
[~rnpridgeon] No it was on Hortonworks so no Sentry.
Error dropping fully qualified partitioned table - Internal error processing
get_partition_names
Key: HIVE-9544
URL: https://issues.apache.org/jira/browse/HIVE-9544
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Environment: HDP 2.2
Reporter: Hari Sekhon
Priority: Minor
When attempting to drop a partitioned table using a fully qualified name I
get this error:
{code}
hive -e 'drop table myDB.my_table_name;'
Logging initialized using configuration in
file:/etc/hive/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.2.0.0-2041/hive/lib/hive-jdbc-0.14.0.2.2.0.0-2041-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.thrift.TApplicationException: Internal error processing
get_partition_names
{code}
It succeeds if I instead do:
{code}hive -e 'use myDB; drop table my_table_name;'{code}
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11558) Hive generates Parquet files with broken footers, causes NullPointerException in Spark / Drill / Parquet tools
[
https://issues.apache.org/jira/browse/HIVE-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-11558:
---
Description:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
at
scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
at
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
at
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
at
scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
What's interesting is that the table works fine in Hive when selecting out of
it, even when doing select * on the whole table and letting it run to the end
(it's a sample data set), it's only other tools it causes problems for.
All fields are string exception for the first one which is timestamp, but this
is not that known issue since if I create another table with 3 fields including
the timestamp and two string fields it works fine in other tools.
The only thing I can see which appears to cause this is the other fields have
lots of NULLs in them as those json fields may or may not be present.
I've converted this exact same json data set to parquet using Apache Drill and
also using Apache SparkSQL and both of those tools create parquet files from
this data set as a straight conversion that are fine when accessed via Parquet
tools or Drill or Spark or Hive (using an external Hive table definition
layered over the generated parquet files).
This implies that it's Hive's generation of Parquet that is broken since both
Drill and Spark can convert the dataset from JSON to Parquet without any issues
on reading the files back in any of other tools.
was:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
[jira] [Updated] (HIVE-11558) Hive generates Parquet files with broken footers, causes NullPointerException in Spark / Drill / Parquet tools
[
https://issues.apache.org/jira/browse/HIVE-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-11558:
---
Description:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
at
scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
at
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
at
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
at
scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
What's interesting is that the table works fine in Hive when selecting out of
it, even when doing select * on the whole table and letting it run to the end
(it's a sample data set), it's only other tools it causes problems for.
All fields are string except for the first one which is timestamp, but this is
not that known issue since if I create another parquet table with 3 fields
including the timestamp and two string fields using CTAS those hive generated
parquet files works fine in the other tools.
The only thing I can see which appears to cause this is the other fields have
lots of NULLs in them as those json fields may or may not be present.
I've converted this exact same json data set to parquet using Apache Drill and
also using Apache Spark SQL and both of those tools create parquet files from
this data set as a straight conversion that are fine when accessed via Parquet
tools or Drill or Spark or Hive (using an external Hive table definition
layered over the generated parquet files).
This implies that it's Hive's generation of Parquet that is broken since both
Drill and Spark can convert the dataset from JSON to Parquet without any issues
on reading the files back in any of other tools.
was:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
[jira] [Updated] (HIVE-11558) Hive generates Parquet files with broken footers, causes NullPointerException in Spark / Drill / Parquet tools
[
https://issues.apache.org/jira/browse/HIVE-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-11558:
---
Description:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at
scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
at
scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
at
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
at
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
at
scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
What's interesting is that the table works fine in Hive when selecting out of
it, even when doing select * on the whole table and letting it run to the end
(it's a sample data set), it's only other tools it causes problems for.
All fields are string except for the first one which is timestamp, but this is
not that known issue since if I create another parquet table with 3 fields
including the timestamp and two string fields using CTAS those hive generated
parquet files works fine in the other tools.
The only thing I can see which appears to cause this is the other fields have
lots of NULLs in them as those json fields may or may not be present.
I've converted this exact same json data set to parquet using Apache Drill and
also using Apache Spark SQL and both of those tools create parquet files from
this data set as a straight conversion that are fine when accessed via Parquet
tools or Drill or Spark or Hive (using an external Hive table definition
layered over the generated parquet files).
This implies that it's Hive's generation of Parquet that is broken since both
Drill and Spark can convert the dataset from JSON to Parquet without any issues
on reading the files back in any of the other mentioned tools.
was:
When creating a Parquet table in Hive from a table in another format (in this
case JSON) using CTAS, the generated parquet files are created with broken
footers and cause NullPointerExceptions in both Parquet tools and Spark when
reading the files directly.
Here is the error from parquet tools:
{code}Could not read footer: java.lang.NullPointerException{code}
Here is the error from Spark reading the parquet file back:
{code}java.lang.NullPointerException
at
parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:249)
at
parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543)
at
parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:520)
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:426)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:298)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:297)
at
scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at
[jira] [Commented] (HIVE-10304) Add deprecation message to HiveCLI
[ https://issues.apache.org/jira/browse/HIVE-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559313#comment-14559313 ] Hari Sekhon commented on HIVE-10304: If just recommending to users to use Beeline instead of Hive CLI that is fine but if Hive 1 CLI was every removed that would cause major headaches to users such as myself who have lots of scripts and programs that make calls to Hive CLI and rewriting things that already work fine for years is not cool. In fact it's the opposite of cool. Add deprecation message to HiveCLI -- Key: HIVE-10304 URL: https://issues.apache.org/jira/browse/HIVE-10304 Project: Hive Issue Type: Sub-task Components: CLI Affects Versions: 1.1.0 Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC1.2 Attachments: HIVE-10304.2.patch, HIVE-10304.3.patch, HIVE-10304.patch As Beeline is now the recommended command line tool to Hive, we should add a message to HiveCLI to indicate that it is deprecated and redirect them to Beeline. This is not suggesting to remove HiveCLI for now, but just a helpful direction for user to know the direction to focus attention in Beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10399) from_unixtime_millis() Hive UDF
[
https://issues.apache.org/jira/browse/HIVE-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon resolved HIVE-10399.
Resolution: Not A Problem
Yes that does exactly what I need, and is an easier workaround than the one I
had implemented. Thanks!
I'm going to close this ticket as the only other reason to extend
from_unixtime() support for millis is for custom formatting which seems to be
added in 1.2 anyway with date_format.
from_unixtime_millis() Hive UDF
---
Key: HIVE-10399
URL: https://issues.apache.org/jira/browse/HIVE-10399
Project: Hive
Issue Type: New Feature
Components: UDF
Environment: HDP 2.2
Reporter: Hari Sekhon
Priority: Minor
Feature request for a
{code}from_unixtime_millis(){code}
Hive UDF - from_unixtime() accepts only secs since epoch, and right now the
solution is to create a custom UDF, but this seems like quite a standard
thing to support millisecond precision dates in Hive natively.
Hari Sekhon
http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-633) ADD FILE command does not accept quoted filenames
[ https://issues.apache.org/jira/browse/HIVE-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499883#comment-14499883 ] Hari Sekhon commented on HIVE-633: -- This has been bugging me for such a long time with add jar 'blah.jar', add file blah.py, source file.sql etc. Seems like it would be a minor improvement to have the parser to do a string.replaceAll() or similar method to replace single and double quotes in these file tokens? ADD FILE command does not accept quoted filenames - Key: HIVE-633 URL: https://issues.apache.org/jira/browse/HIVE-633 Project: Hive Issue Type: Bug Affects Versions: 0.3.0 Environment: Ubuntu Linux (intrepid) Reporter: Saurabh Nanda Priority: Minor The following command says file does not exist. Removing the quotes around the filename makes it work. hive add files '/tmp/testing.jar'; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9768) Hive LLAP Metadata pre-load for low latency, + cluster-wide metadata refresh/invalidate command
[ https://issues.apache.org/jira/browse/HIVE-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HIVE-9768: -- Description: Feature request for Hive LLAP to preload table metadata across all running nodes to reduce query latency (this is what Impala does). The design decision behind this in Impala was to avoid the latency overhead of fetching the metadata at query time, since that's an extra database query (or possibly HBase query in future HIVE-9452) that must first be completely fullfilled before the Hive LLAP query even starts to run, which would slow down the response to the user if not pre-loaded. Also, any temporary outage of the metadata layer would affect the speed LLAP layer so pre-loading and caching the metadata adds resilience against this. This pre-loaded metadata also requires a cluster-wide refresh metadata operation, something Impala added later, and now calls INVALIDATE METADATA in it's SQL dialect. I propose using a more intuitive REFRESH METADATA Hive command instead. (Fyi I was in the original trio of Impala SMEs at Cloudera in early 2013) Regards, Hari Sekhon ex-Cloudera http://www.linkedin.com/in/harisekhon was: Feature request for Hive LLAP to preload table metadata across all running nodes to reduce query latency (this is what Impala does). The design decision behind this in Impala was to avoid the latency overhead of fetching the metadata at query time, since that's an extra database query (or possibly HBase query in future HIVE-9452) that must first be completely fullfilled before the Hive LLAP query even starts to run, which would slow down the response to the user if not pre-loaded. Also, any temporary outage of the metadata layer would affect the speed LLAP layer so pre-loading and caching the metadata adds resilience against this. This pre-loaded metadata also requires a cluster-wide refresh metadata operation, something Impala added later, and now calls INVALIDATE METADATA in it's SQL dialect. I propose using a more intuitive REFRESH METADATA Hive command instead. (Fyi I was in the first trio of Impala SMEs at Cloudera in early 2013) Regards, Hari Sekhon ex-Cloudera http://www.linkedin.com/in/harisekhon Hive LLAP Metadata pre-load for low latency, + cluster-wide metadata refresh/invalidate command --- Key: HIVE-9768 URL: https://issues.apache.org/jira/browse/HIVE-9768 Project: Hive Issue Type: New Feature Components: HCatalog, Metastore, Query Planning, Query Processor Affects Versions: llap Environment: HDP 2.2 Reporter: Hari Sekhon Feature request for Hive LLAP to preload table metadata across all running nodes to reduce query latency (this is what Impala does). The design decision behind this in Impala was to avoid the latency overhead of fetching the metadata at query time, since that's an extra database query (or possibly HBase query in future HIVE-9452) that must first be completely fullfilled before the Hive LLAP query even starts to run, which would slow down the response to the user if not pre-loaded. Also, any temporary outage of the metadata layer would affect the speed LLAP layer so pre-loading and caching the metadata adds resilience against this. This pre-loaded metadata also requires a cluster-wide refresh metadata operation, something Impala added later, and now calls INVALIDATE METADATA in it's SQL dialect. I propose using a more intuitive REFRESH METADATA Hive command instead. (Fyi I was in the original trio of Impala SMEs at Cloudera in early 2013) Regards, Hari Sekhon ex-Cloudera http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
