[jira] [Created] (HIVE-20444) Parameter is not properly quoted in DbNotificationListener.addWriteNotificationLog

2018-08-22 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-20444:
-

 Summary: Parameter is not properly quoted in 
DbNotificationListener.addWriteNotificationLog
 Key: HIVE-20444
 URL: https://issues.apache.org/jira/browse/HIVE-20444
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Daniel Dai


See exception:
{code}
2018-08-22T04:44:22,758 INFO  [pool-8-thread-190]: 
listener.DbNotificationListener 
(DbNotificationListener.java:addWriteNotificationLog(765)) - Going to execute 
insert 
2018-08-22T04:44:22,773 ERROR [pool-8-thread-190]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Unable to 
add write notification log org.postgresql.util.PSQLException: ERROR: syntax 
error at or near "UTC"
  Position: 1032
at 
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
at 
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
at 
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at 
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
at 
com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.addWriteNotificationLog(DbNotificationListener.java:766)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:657)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore.java:7563)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.add_write_notification_log(HiveMetaStore.java:7589)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy34.add_write_notification_log(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_write_notification_log.getResult(ThriftHiveMetastore.java:19071)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$add_write_notification_log.getResult(ThriftHiveMetastore.java:19056)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:659)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore.java:7563)
at

Re: External vs. Managed Tables - Current state

2018-08-22 Thread Lars Francke 
Thanks for the input here and on the issue itself.

@Sergey: That is a good summary. I would try to distill that into a Wiki
page.

 I still think that it'd be good to list the actual technical differences.

1) Because this is not actually written down in a concise way so lots of
our users don't know this distinction. I for example have heard all kinds
of "rumors"

2) I've been using Hive for 8+ years or so and I use it almost weekly and
follow the dev list and Jira issues and I still missed about 80% of the
features that have been implemented in the last few years affecting this.
And these things can change a decision. Back when the only difference was
whether a DROP would delete data it was much simpler. Now you might opt for
an internal table even if it does not match your ETL flows etc. because it
gives you better performance or some other feature.

3) It's a pity to have all these features not being used because no one
outside of a small circle knows about them :)

So I'd still appreciate any and all input you might have for me.

Cheers,
Lars


On Wed, Aug 22, 2018 at 9:45 PM, Sergey Shelukhin 
wrote:

> I think it’s very simple conceptually (well the way I’ve been thinking
> about it), and any limitations can be derived from this using logic. If
> someone has specific questions we can clarify; if some limitations are
> missing (e.g. when Hive was using stats for hbase tables) it is a bug.
>
> For managed tables, Hive owns and manages the data:
>
> * Data, properties, etc. can only be changed via Hive commands or APIs;
> Hive assumes it’s true and any other modification on the underlying
> storage is undefined behavior (not just results - basically all bets are
> off ;)).
> * Data is attached to the Hive entity (partition, table). Whatever happens
> to the entity happens to the data (e.g. the behavior mentioned in the
> wiki, where  dropping the table/partition deletes the data).
>
>
> For external tables (also SH tables before Hive made those explicitly
> external), Hive doesn’t manage or own the data:
>
> * Hive assumes the data can be modified externally at any time. Any
> feature that requires assumptions about data being unchanged (queries from
> stats, ACID, some MV, query cache) won’t work. If it works, it’s a bug.
> * Data is detached from Hive entities, e.g. dropping the table by default
> doesn’t delete the data - we assume someone else owns it and we are just
> pointing at it.
> * Additionally, since Hive cannot track modifications, external
> products/users/etc. are fully responsible for any concurrency control. In
> particular, modifying data under the running query may produce
> inconsistent results (no atomicity between tasks), undefined results due
> to retries, etc.
>
>
>
>
>
> On 18/8/21, 10:31, "Mithun RK"  wrote:
>
> >+1. I'm out of the loop as well. I'd be keen on grokking what's not
> >available with EXTERNAL tables, given that these are popular on our
> >clusters as well. It would be good to have that documented in one place.
> >
> >On Tue, Aug 21, 2018 at 8:19 AM Lars Francke 
> >wrote:
> >
> >> As this came up with yet another customer and I've heard more things
> >>that I
> >> believe to be false:
> >>
> >> Please, anyone have any kind of feedback for me here? Is my list
> >>correct,
> >> can you add stuff to it, have more details to certain points etc.?
> >>
> >> Thank you!
> >>
> >> On Wed, Aug 1, 2018 at 12:17 PM, Lars Francke 
> >> wrote:
> >>
> >> > I have opened https://issues.apache.org/jira/browse/HIVE-20287 and
> >>would
> >> > love to get feedback
> >> >
> >> > On Wed, Aug 1, 2018 at 11:14 AM, Lars Francke  >
> >> > wrote:
> >> >
> >> >> Thanks Lefty! I haven't created a JIRA but it's on my list.
> >> >>
> >> >> Still interested in any input anyone might have?
> >> >>
> >> >> Some of you guys _must_ have some knowledge on this stuff.
> >> >>
> >> >> You're correct that the DROP stuff is documented. What I meant is
> >>that I
> >> >> couldn't find the relevant parts in the code.
> >> >>
> >> >> On Sat, Jul 21, 2018 at 6:15 AM, Lefty Leverenz <
> >> leftylever...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Agreed, the Hive wiki is woefully incomplete.  But it does mention
> >>DROP
> >> >>> for
> >> >>> external tables in two sections of the DDL doc:
> >> >>>
> >> >>>- External Tables
> >> >>> >> >>> anual+DDL#LanguageManualDDL-ExternalTables>:
> >> >>> "When dropping an EXTERNAL table, data in the table is NOT
> >>deleted
> >> >>> from
> >> >>>the file system."
> >> >>>- Drop Table
> >> >>> >> >>> anual+DDL#LanguageManualDDL-DropTable>:
> >> >>> "When dropping an EXTERNAL table, data in the table will NOT be
> >> >>> deleted
> >> >>>from the file system."
> >> >>>
> >> >>> A wiki page comparing managed and external tables would be very
> >> helpful,
> >> >>> and the list is a good start.  I suggest you open a JIRA

[jira] [Created] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded

2018-08-22 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-20443:
---

 Summary: txn stats cleanup in compaction txn handler is unneeded
 Key: HIVE-20443
 URL: https://issues.apache.org/jira/browse/HIVE-20443
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20442) Hive stale lock when the hiveserver2 background thread died abruptly

2018-08-22 Thread Rajkumar Singh (JIRA) 
Rajkumar Singh created HIVE-20442:
-

 Summary: Hive stale lock when the hiveserver2 background thread 
died abruptly
 Key: HIVE-20442
 URL: https://issues.apache.org/jira/browse/HIVE-20442
 Project: Hive
  Issue Type: Bug
  Components: Hive, Transactions
Affects Versions: 2.1.1
 Environment: Hive-2.1
Reporter: Rajkumar Singh


this look like a race condition where background thread is not able to release 
the lock it aquired.

1. hiveserver2 background thread request for lock
{code}
2018-08-20T14:13:38,813 INFO  [HiveServer2-Background-Pool: Thread-X]: 
lockmgr.DbLockManager (DbLockManager.java:lock(100)) - Requesting: 
queryId=hive_xxx LockRequest(component:[LockComponent(type:SHARED_READ, 
level:TABLE, dbname:testdb, tablename:test_table, operationType:SELECT)], 
txnid:0, user:hive, hostname:HOSTNAME, agentInfo:hive_xxx)
{code}
2. acquired the lock and start heartbeating
{code}
2018-08-20T14:36:30,233 INFO  [HiveServer2-Background-Pool: Thread-X]: 
lockmgr.DbTxnManager (DbTxnManager.java:startHeartbeat(517)) - Started 
heartbeat with delay/interval = 15/15 MILLISECONDS for 
query: agentInfo:hive_xxx
{code}

3. during time between event #1 and #2, client disconnected and deleteContext 
cleanup the session dir
{code}
2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-XXX]: 
thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(136)) - 
Session disconnected without closing properly.
2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-]: 
thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(140)) - 
Closing the session: SessionHandle [3be07faf-5544-4178-8b50-8173002b171a]
2018-08-21T15:39:57,820 INFO  [HiveServer2-Handler-Pool: Thread-]: 
service.CompositeService (SessionManager.java:closeSession(363)) - Session 
closed, SessionHandle [xxx], current sessions:2
{code}

4. background thread died with NPE while trying to get the queryid 
{code}
java.lang.NullPointerException: null
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1568) 
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414) 
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211) 
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204) 
~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
 [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at 
org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
 [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
 [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
at java.security.AccessController.doPrivileged(Native Method) 
[?:1.8.0_77]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_77]
{code}
did not get a chance to release the lock and heartbeater thread continue 
heartbeat indefinately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: External vs. Managed Tables - Current state

2018-08-22 Thread Sergey Shelukhin 
I think it’s very simple conceptually (well the way I’ve been thinking
about it), and any limitations can be derived from this using logic. If
someone has specific questions we can clarify; if some limitations are
missing (e.g. when Hive was using stats for hbase tables) it is a bug.

For managed tables, Hive owns and manages the data:

* Data, properties, etc. can only be changed via Hive commands or APIs;
Hive assumes it’s true and any other modification on the underlying
storage is undefined behavior (not just results - basically all bets are
off ;)).
* Data is attached to the Hive entity (partition, table). Whatever happens
to the entity happens to the data (e.g. the behavior mentioned in the
wiki, where  dropping the table/partition deletes the data).


For external tables (also SH tables before Hive made those explicitly
external), Hive doesn’t manage or own the data:

* Hive assumes the data can be modified externally at any time. Any
feature that requires assumptions about data being unchanged (queries from
stats, ACID, some MV, query cache) won’t work. If it works, it’s a bug.
* Data is detached from Hive entities, e.g. dropping the table by default
doesn’t delete the data - we assume someone else owns it and we are just
pointing at it.
* Additionally, since Hive cannot track modifications, external
products/users/etc. are fully responsible for any concurrency control. In
particular, modifying data under the running query may produce
inconsistent results (no atomicity between tasks), undefined results due
to retries, etc.





On 18/8/21, 10:31, "Mithun RK"  wrote:

>+1. I'm out of the loop as well. I'd be keen on grokking what's not
>available with EXTERNAL tables, given that these are popular on our
>clusters as well. It would be good to have that documented in one place.
>
>On Tue, Aug 21, 2018 at 8:19 AM Lars Francke 
>wrote:
>
>> As this came up with yet another customer and I've heard more things
>>that I
>> believe to be false:
>>
>> Please, anyone have any kind of feedback for me here? Is my list
>>correct,
>> can you add stuff to it, have more details to certain points etc.?
>>
>> Thank you!
>>
>> On Wed, Aug 1, 2018 at 12:17 PM, Lars Francke 
>> wrote:
>>
>> > I have opened https://issues.apache.org/jira/browse/HIVE-20287 and
>>would
>> > love to get feedback
>> >
>> > On Wed, Aug 1, 2018 at 11:14 AM, Lars Francke 
>> > wrote:
>> >
>> >> Thanks Lefty! I haven't created a JIRA but it's on my list.
>> >>
>> >> Still interested in any input anyone might have?
>> >>
>> >> Some of you guys _must_ have some knowledge on this stuff.
>> >>
>> >> You're correct that the DROP stuff is documented. What I meant is
>>that I
>> >> couldn't find the relevant parts in the code.
>> >>
>> >> On Sat, Jul 21, 2018 at 6:15 AM, Lefty Leverenz <
>> leftylever...@gmail.com>
>> >> wrote:
>> >>
>> >>> Agreed, the Hive wiki is woefully incomplete.  But it does mention
>>DROP
>> >>> for
>> >>> external tables in two sections of the DDL doc:
>> >>>
>> >>>- External Tables
>> >>>> >>> anual+DDL#LanguageManualDDL-ExternalTables>:
>> >>> "When dropping an EXTERNAL table, data in the table is NOT
>>deleted
>> >>> from
>> >>>the file system."
>> >>>- Drop Table
>> >>>> >>> anual+DDL#LanguageManualDDL-DropTable>:
>> >>> "When dropping an EXTERNAL table, data in the table will NOT be
>> >>> deleted
>> >>>from the file system."
>> >>>
>> >>> A wiki page comparing managed and external tables would be very
>> helpful,
>> >>> and the list is a good start.  I suggest you open a JIRA issue.
>> >>>
>> >>> Who can help Lars with this task?
>> >>>
>> >>> -- Lefty
>> >>>
>> >>>
>> >>> On Thu, Jul 19, 2018 at 3:04 PM Lars Francke
>>
>> >>> wrote:
>> >>>
>> >>> > Hi,
>> >>> >
>> >>> > I've been wondering if anyone can tell me what the differences and
>> >>> > limitations of managed vs. external tables are these days.
>> >>> >
>> >>> > I've seen the docs from Hortonworks[1] that list these features as
>> not
>> >>> > supported by External tables:
>> >>> > * Query cache
>> >>> > * Materialized views, except in a limited way
>> >>> > * Default statistics gathering
>> >>> > * Compute queries using statistics
>> >>> > * Automatic runtime filtering
>> >>> > * File merging after insert
>> >>> >
>> >>> > But there are no details. For some of them I'm not even sure what
>> they
>> >>> are
>> >>> > supposed to mean because I couldn't find any documentation in our
>> Wiki
>> >>> :(
>> >>> >
>> >>> > So I'd love to create a Wiki page that describes the differences
>> >>> between
>> >>> > managed & external tables and list relevant JIRAs but I need your
>> help
>> >>> for
>> >>> > that.
>> >>> >
>> >>> > I scanned the code for all references to TableType and this is the
>> >>> list I
>> >>> > found but I'm almost certainly missing something:
>> >>> >
>> >>> > * ARCHIVE/UNARCHIVE - DDLTask - Only

Review Request 68476: HIVE-20433

2018-08-22 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68476/
---

Review request for hive, Ashutosh Chauhan and Gopal V.


Bugs: HIVE-20433
https://issues.apache.org/jira/browse/HIVE-20433


Repository: hive-git


Description
---

Implicit String to Timestamp conversion is slow


Diffs (updated)
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
 8a057d1dab 


Diff: https://reviews.apache.org/r/68476/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true

2018-08-22 Thread Hui Huang (JIRA) 
Hui Huang created HIVE-20441:


 Summary: NPE in ExprNodeGenericFuncDesc  when 
hive.allow.udf.load.on.demand is set to true
 Key: HIVE-20441
 URL: https://issues.apache.org/jira/browse/HIVE-20441
 Project: Hive
  Issue Type: Bug
  Components: CLI, HiveServer2
Affects Versions: 1.2.1, 2.3.3
Reporter: Hui Huang
Assignee: Hui Huang


When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been 
started, the new created function from other clients or hiveserver2 will be 
loaded from the metastore at the first time. 

When the udf is used in where clause, we got a NPE like:

{code:java}
Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: NullPointerException null
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:320) 
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
SHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
T]
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
PSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
PSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
 ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
 ~[hive-exec-2.
3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76) 
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:229)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:176)
 ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11613)

Review Request 68474: HIVE-20440: Create better cache eviction policy for SmallTableCache

2018-08-22 Thread Antal Sinkovits via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68474/
---

Review request for hive, Naveen Gangam, Sahil Takiar, Adam Szita, and Xuefu 
Zhang.


Repository: hive-git


Description
---

I've modified the SmallTableCache to use guava cache, with soft references. 
By using a value loader, I've also eliminated the synchronization on the 
intern-ed string of the path.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java 
cf27e92bafdc63096ec0fa8c3106657bab52f370 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java 
3293100af96dc60408c53065fa89143ead98f818 


Diff: https://reviews.apache.org/r/68474/diff/1/


Testing
---


Thanks,

Antal Sinkovits

[jira] [Created] (HIVE-20440) Create better cache eviction policy for SmallTableCache

2018-08-22 Thread Antal Sinkovits (JIRA) 
Antal Sinkovits created HIVE-20440:
--

 Summary: Create better cache eviction policy for SmallTableCache
 Key: HIVE-20440
 URL: https://issues.apache.org/jira/browse/HIVE-20440
 Project: Hive
  Issue Type: Improvement
  Components: Spark
Reporter: Antal Sinkovits
Assignee: Antal Sinkovits


Enhance the SmallTableCache, to use guava cache with soft references, so that 
we evict when there is memory pressure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20439) Use the inflated memory limit during join selection for llap

2018-08-22 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-20439:
---

 Summary: Use the inflated memory limit during join selection for 
llap
 Key: HIVE-20439
 URL: https://issues.apache.org/jira/browse/HIVE-20439
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20438) Add possibility to obtain Hive version using HiveMetaStoreClient

2018-08-22 Thread Bohdan Kazydub (JIRA) 
Bohdan Kazydub created HIVE-20438:
-

 Summary: Add possibility to obtain Hive version using 
HiveMetaStoreClient
 Key: HIVE-20438
 URL: https://issues.apache.org/jira/browse/HIVE-20438
 Project: Hive
  Issue Type: Improvement
Reporter: Bohdan Kazydub
Assignee: Bohdan Kazydub


Sometimes it may prove useful to be able to obtain version of remote Hive 
server using HiveMetaStoreClient (through Thrift client). E.g. for a case, when 
you need to check client-server compatibility before issuing actual metadata 
request or querying data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)