[jira] [Created] (HIVE-24563) Check if we can interchange client and server sides for umbilical for external client flow
Shubham Chaurasia created HIVE-24563:
Summary: Check if we can interchange client and server sides for
umbilical for external client flow
Key: HIVE-24563
URL: https://issues.apache.org/jira/browse/HIVE-24563
Project: Hive
Issue Type: Sub-task
Components: Hive, llap
Reporter: Shubham Chaurasia
Assignee: Shubham Chaurasia
Currently we open three tcp connections when llap external client communicates
to llap.
{noformat}
llap-ext-client ... llap
connection1: client ...>>... server
(RPC for submitting fragments - say t1, t2, t3. llap-ext-client initiates
connection)
connection2: client ...>>... server
(for reading the output of t1, t2, t3. llap-ext-client initiates connection)
connection3: umbilical server ...<<... client
(RPC for status updates/heartbeat of t1, t2, t3. llap Daemon initiates
connection)
{noformat}
connection3 starts a umbilical(RPC) server at the client side to which llap
daemon keeps sending the task statuses / heartbeats and node heartbeats.
*The Problem*
In cloud based deployment, we need to open tcp traffic.
1. For connection1 and connection2, we need to open incoming tcp traffic on the
machines running llap from client.
2. For connection3, we need to open incoming tcp traffic on the machines where
llap-ext-client is running, from llap daemon.
Here clients also need to worry about opening traffic(from llap) at their end.
This jira is to evaluate the possibility of interchanging Umbilical server and
client sides i.e. umbilical server will run in llap only and llap-ext-client
will act as client and initiate the connection.
We can have umbilical address in llap splits (when get_splits is called by
external client) which the client can later connect to.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24562) Deflake TestHivePrivilegeObjectOwnerNameAndType
Vihang Karajgaonkar created HIVE-24562:
--
Summary: Deflake TestHivePrivilegeObjectOwnerNameAndType
Key: HIVE-24562
URL: https://issues.apache.org/jira/browse/HIVE-24562
Project: Hive
Issue Type: Test
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar
One of my unrelated PRs fails this test
{{TestHivePrivilegeObjectOwnerNameAndType}}. The exception which I see in the
logs is below:
{noformat}
Caused by: ERROR 42X05: Table/View 'TXN_LOCK_TBL' does not exist.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
org.apache.derby.impl.sql.compile.LockTableNode.bindStatement(Unknown Source)
at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
at
org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown
Source)
... 73 more
)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:651)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy46.open_txns(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3634)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:218)
at com.sun.proxy.$Proxy47.openTxn(Unknown Source)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:243)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:227)
at org.apache.hadoop.hive.ql.Compiler.openTransaction(Compiler.java:268)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:215)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:178)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:150)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:137)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHivePrivilegeObjectOwnerNameAndType.runCmd(TestHivePrivilegeObjectOwnerNameAndType.java:86)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHivePrivilegeObjectOwnerNameAndType.beforeTest(TestHivePrivilegeObjectOwnerNameAndType.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
at
[jira] [Created] (HIVE-24561) Deflake TestCachedStoreUpdateUsingEvents
Vihang Karajgaonkar created HIVE-24561:
--
Summary: Deflake TestCachedStoreUpdateUsingEvents
Key: HIVE-24561
URL: https://issues.apache.org/jira/browse/HIVE-24561
Project: Hive
Issue Type: Test
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar
TestCachedStoreUpdateUsingEvents seems to use "file:/tmp" as the table and
database directory. The cleanUp method will clean all the sub-directories
directories in /tmp which can be error prone.
Also noticed that I see a lot NPEs from {{SharedCache#getMemorySizeEstimator}}
because the {{sizeEstimators}} field is null. We should add a null check for
that field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-24560) Move Column Name and Type Parsing to AbstractSerde Class
David Mollitor created HIVE-24560: - Summary: Move Column Name and Type Parsing to AbstractSerde Class Key: HIVE-24560 URL: https://issues.apache.org/jira/browse/HIVE-24560 Project: Hive Issue Type: Improvement Reporter: David Mollitor Assignee: David Mollitor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24559) Fix a spelling issue in TxnHandler
RickyMa created HIVE-24559: -- Summary: Fix a spelling issue in TxnHandler Key: HIVE-24559 URL: https://issues.apache.org/jira/browse/HIVE-24559 Project: Hive Issue Type: Improvement Reporter: RickyMa -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24558) Handle update in table level regular expression.
Aasha Medhi created HIVE-24558: -- Summary: Handle update in table level regular expression. Key: HIVE-24558 URL: https://issues.apache.org/jira/browse/HIVE-24558 Project: Hive Issue Type: Task Reporter: Aasha Medhi Assignee: Aasha Medhi -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24557) Misalignment in matching db name from Hive and Webhcat
Davide Vergari created HIVE-24557:
--
Summary: Misalignment in matching db name from Hive and Webhcat
Key: HIVE-24557
URL: https://issues.apache.org/jira/browse/HIVE-24557
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 2.3.6
Reporter: Davide Vergari
I found a small misalignment on how Hive and WebHCat parse database name
starting with numbers.
If you execute:
{quote}hive> create database 2020db;
OK
Time taken: 0.44 seconds
hive>
{quote}
it works, but if you do the same on webhcatalog it doesn't:
{quote}curl -i -s -X PUT -HContent-type:application/json -d '\{"comment":
"Test", "location":
"hdfs://singlenode.localdomain:8020/apps/hive/warehouse/2020db"}'
'http://singlenode.localdomain:50111/templeton/v1/ddl/database/2020db?user.name=hdfs'
HTTP/1.1 400 Bad Request
Set-Cookie:
hadoop.auth="u=hdfs=hdfs=simple=1608581595777=JzsCUqvyyAYHykAHdteybh1tI4jFLLVVJC5D9FaUu0A=";
Path=/; HttpOnly
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{"error":"Invalid DDL identifier :db"}
{quote}
I suggest to harmonize the behavior, maybe applying a patch like the following
one:
{quote}---
a/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java
+++
b/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java
@@ -1068,7 +1068,7 @@ public void verifyParam(List param, String name)
}
}
- public static final Pattern DDL_ID = Pattern.compile("[a-zA-Z]\\w*");
+ public static final Pattern DDL_ID = Pattern.compile("[a-zA-Z0-9]\\w*");
public static final Pattern PROPERTY_ID =
Pattern.compile("[a-zA-Z0-9][\\w\\.\\-]*(?
