[jira] [Work logged] (HIVE-24417) Add config options for Atlas client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?focusedWorklogId=515945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515945 ] ASF GitHub Bot logged work on HIVE-24417: - Author: ASF GitHub Bot Created on: 24/Nov/20 07:58 Start Date: 24/Nov/20 07:58 Worklog Time Spent: 10m Work Description: pkumarsinha opened a new pull request #1700: URL: https://github.com/apache/hive/pull/1700 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515945) Remaining Estimate: 0h Time Spent: 10m > Add config options for Atlas client timeouts > > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24417) Add config options for Atlas client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24417: -- Labels: pull-request-available (was: ) > Add config options for Atlas client timeouts > > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24417) Add config options for Atlas client timeouts
[ https://issues.apache.org/jira/browse/HIVE-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha reassigned HIVE-24417: --- > Add config options for Atlas client timeouts > > > Key: HIVE-24417 > URL: https://issues.apache.org/jira/browse/HIVE-24417 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24389) Trailing zeros of constant decimal numbers are removed
[
https://issues.apache.org/jira/browse/HIVE-24389?focusedWorklogId=515935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515935
]
ASF GitHub Bot logged work on HIVE-24389:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 07:08
Start Date: 24/Nov/20 07:08
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1676:
URL: https://github.com/apache/hive/pull/1676#discussion_r529245143
##
File path:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java
##
@@ -46,12 +46,4 @@ public IntWritable getWritableConstantValue() {
return value;
}
- @Override
- public int precision() {
Review comment:
Yes, by overriding precision constant values has the precision of the
value itself.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515935)
Time Spent: 1.5h (was: 1h 20m)
> Trailing zeros of constant decimal numbers are removed
> --
>
> Key: HIVE-24389
> URL: https://issues.apache.org/jira/browse/HIVE-24389
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In some case Hive removes trailing zeros of constant decimal numbers
> {code}
> select cast(1.1 as decimal(22, 2))
> 1.1
> {code}
> In this case *WritableConstantHiveDecimalObjectInspector* is used and this
> object inspector takes it's wrapped HiveDecimal scale instead of the scale
> specified in the wrapped typeinfo:
> {code}
> this = {WritableConstantHiveDecimalObjectInspector@14415}
> value = {HiveDecimalWritable@14426} "1.1"
> typeInfo = {DecimalTypeInfo@14421} "decimal(22,2)"{code}
> However in case of an expression with an aggregate function
> *WritableHiveDecimalObjectInspector* is used
> {code}
> select cast(sum(1.1) as decimal(22, 2))
> 1.10
> {code}
> {code}
> o = {HiveDecimalWritable@16633} "1.1"
> oi = {WritableHiveDecimalObjectInspector@16634}
> typeInfo = {DecimalTypeInfo@16640} "decimal(22,2)"
> {code}
> Casting the expressions to string
> {code:java}
> select cast(cast(1.1 as decimal(22, 2)) as string), cast(cast(sum(1.1) as
> decimal(22, 2)) as string)
> 1.1 1.10
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24389) Trailing zeros of constant decimal numbers are removed
[
https://issues.apache.org/jira/browse/HIVE-24389?focusedWorklogId=515934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515934
]
ASF GitHub Bot logged work on HIVE-24389:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 07:07
Start Date: 24/Nov/20 07:07
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1676:
URL: https://github.com/apache/hive/pull/1676#discussion_r529244528
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rewrite_window.q.out
##
@@ -166,7 +166,7 @@ POSTHOOK: Input: arc_view@wealth
A masked pattern was here
CBO PLAN:
HiveSortLimit(sort0=[$0], dir0=[ASC])
- HiveProject(quartile=[$0], total=[$1])
+ HiveProject(quartile=[CAST($0):DECIMAL(12, 1)], total=[$1])
Review comment:
Reverted the commit that removes precision override. Change in scale is
expected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515934)
Time Spent: 1h 20m (was: 1h 10m)
> Trailing zeros of constant decimal numbers are removed
> --
>
> Key: HIVE-24389
> URL: https://issues.apache.org/jira/browse/HIVE-24389
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In some case Hive removes trailing zeros of constant decimal numbers
> {code}
> select cast(1.1 as decimal(22, 2))
> 1.1
> {code}
> In this case *WritableConstantHiveDecimalObjectInspector* is used and this
> object inspector takes it's wrapped HiveDecimal scale instead of the scale
> specified in the wrapped typeinfo:
> {code}
> this = {WritableConstantHiveDecimalObjectInspector@14415}
> value = {HiveDecimalWritable@14426} "1.1"
> typeInfo = {DecimalTypeInfo@14421} "decimal(22,2)"{code}
> However in case of an expression with an aggregate function
> *WritableHiveDecimalObjectInspector* is used
> {code}
> select cast(sum(1.1) as decimal(22, 2))
> 1.10
> {code}
> {code}
> o = {HiveDecimalWritable@16633} "1.1"
> oi = {WritableHiveDecimalObjectInspector@16634}
> typeInfo = {DecimalTypeInfo@16640} "decimal(22,2)"
> {code}
> Casting the expressions to string
> {code:java}
> select cast(cast(1.1 as decimal(22, 2)) as string), cast(cast(sum(1.1) as
> decimal(22, 2)) as string)
> 1.1 1.10
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24389) Trailing zeros of constant decimal numbers are removed
[
https://issues.apache.org/jira/browse/HIVE-24389?focusedWorklogId=515933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515933
]
ASF GitHub Bot logged work on HIVE-24389:
-
Author: ASF GitHub Bot
Created on: 24/Nov/20 07:06
Start Date: 24/Nov/20 07:06
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1676:
URL: https://github.com/apache/hive/pull/1676#discussion_r529244214
##
File path:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveDecimalObjectInspector.java
##
@@ -62,13 +62,4 @@ public int precision() {
}
Review comment:
Seems that removing precision override has unwanted side effects. I
reverted that part.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515933)
Time Spent: 1h 10m (was: 1h)
> Trailing zeros of constant decimal numbers are removed
> --
>
> Key: HIVE-24389
> URL: https://issues.apache.org/jira/browse/HIVE-24389
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In some case Hive removes trailing zeros of constant decimal numbers
> {code}
> select cast(1.1 as decimal(22, 2))
> 1.1
> {code}
> In this case *WritableConstantHiveDecimalObjectInspector* is used and this
> object inspector takes it's wrapped HiveDecimal scale instead of the scale
> specified in the wrapped typeinfo:
> {code}
> this = {WritableConstantHiveDecimalObjectInspector@14415}
> value = {HiveDecimalWritable@14426} "1.1"
> typeInfo = {DecimalTypeInfo@14421} "decimal(22,2)"{code}
> However in case of an expression with an aggregate function
> *WritableHiveDecimalObjectInspector* is used
> {code}
> select cast(sum(1.1) as decimal(22, 2))
> 1.10
> {code}
> {code}
> o = {HiveDecimalWritable@16633} "1.1"
> oi = {WritableHiveDecimalObjectInspector@16634}
> typeInfo = {DecimalTypeInfo@16640} "decimal(22,2)"
> {code}
> Casting the expressions to string
> {code:java}
> select cast(cast(1.1 as decimal(22, 2)) as string), cast(cast(sum(1.1) as
> decimal(22, 2)) as string)
> 1.1 1.10
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24415) HiveSplitGenerator blocks Tez dispatcher
[ https://issues.apache.org/jira/browse/HIVE-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa İman reassigned HIVE-24415: --- > HiveSplitGenerator blocks Tez dispatcher > > > Key: HIVE-24415 > URL: https://issues.apache.org/jira/browse/HIVE-24415 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > > HiveSplitGenerator does a lot of heavyweight operations in its constructor. > These operations block AsyncDispatcher in Tez > [https://github.com/apache/tez/blob/989d286d09cac7c4e4c5a0e06dd75ea5a6f15478/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L141] > . We should move bulk of initialization out of constructor. > The only reason of setting up everything in constructor is > DynamicPartitionPruner. We can buffer incoming events in HiveSplitGenerator > until dynamic partition pruner is initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?focusedWorklogId=515817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515817 ] ASF GitHub Bot logged work on HIVE-24411: - Author: ASF GitHub Bot Created on: 23/Nov/20 23:17 Start Date: 23/Nov/20 23:17 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1695: URL: https://github.com/apache/hive/pull/1695 …fMemoryError ### What changes were proposed in this pull request? Check the root cause of failed tasks to see if that is an instance of OutOfMemoryError. ### Why are the changes needed? Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the HiveServer2 in case of OutOfMemoryError when executing the tasks. The exception is obtained by calling method `future.get()`, however the exception may never be an instance of OutOfMemoryError, as the exception is wrapped in ExecutionException, see the method report in class FutureTask. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515817) Time Spent: 0.5h (was: 20m) > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method _future.get()_, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?focusedWorklogId=515816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515816 ] ASF GitHub Bot logged work on HIVE-24411: - Author: ASF GitHub Bot Created on: 23/Nov/20 23:16 Start Date: 23/Nov/20 23:16 Worklog Time Spent: 10m Work Description: dengzhhu653 closed pull request #1695: URL: https://github.com/apache/hive/pull/1695 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515816) Time Spent: 20m (was: 10m) > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method _future.get()_, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24412) Extract common parts of HiveDeCorrelator
[ https://issues.apache.org/jira/browse/HIVE-24412?focusedWorklogId=515812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515812 ] ASF GitHub Bot logged work on HIVE-24412: - Author: ASF GitHub Bot Created on: 23/Nov/20 23:01 Start Date: 23/Nov/20 23:01 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #1697: URL: https://github.com/apache/hive/pull/1697#issuecomment-732475769 +1 pending tests. (I was under the impression that some of the hive rel operators have special logic, but if that turns out not to be the case we should go ahead with this pull request). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515812) Time Spent: 20m (was: 10m) > Extract common parts of HiveDeCorrelator > > > Key: HIVE-24412 > URL: https://issues.apache.org/jira/browse/HIVE-24412 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a > Hive version and the implementation is the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24389) Trailing zeros of constant decimal numbers are removed
[
https://issues.apache.org/jira/browse/HIVE-24389?focusedWorklogId=515807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515807
]
ASF GitHub Bot logged work on HIVE-24389:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 22:53
Start Date: 23/Nov/20 22:53
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1676:
URL: https://github.com/apache/hive/pull/1676#discussion_r529046059
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rewrite_window.q.out
##
@@ -293,24 +293,24 @@ STAGE PLANS:
Statistics: Num rows: 4 Data size: 480 Basic stats: COMPLETE
Column stats: COMPLETE
Group By Operator
aggregations: max(_col1)
- keys: _col2 (type: decimal(12,1))
+ keys: _col2 (type: decimal(19,4))
Review comment:
Difference in precision and scale for column. Why has this changed (same
below)?
##
File path: ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
##
@@ -377,7 +377,7 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5,
_col6, _col7, _col8
Statistics: Num rows: 1 Data size: 520 Basic stats: COMPLETE
Column stats: NONE
Select Operator
- expressions: _col0 (type: bigint), '1' (type: string), _col1
(type: bigint), _col2 (type: decimal(11,1)), 2 (type: int), _col0 (type:
bigint), _col3 (type: bigint), 7 (type: decimal(2,0)), _col4 (type: bigint),
_col5 (type: bigint), _col6 (type: bigint), _col7 (type: int), _col8 (type:
bigint)
+ expressions: _col0 (type: bigint), '1' (type: string), _col1
(type: bigint), _col2 (type: decimal(11,1)), 2 (type: int), _col0 (type:
bigint), _col3 (type: bigint), 7 (type: decimal(11,0)), _col4 (type: bigint),
_col5 (type: bigint), _col6 (type: bigint), _col7 (type: int), _col8 (type:
bigint)
Review comment:
Difference in precision and scale for column. Why has this changed?
##
File path:
ql/src/test/results/clientpositive/llap/materialized_view_rewrite_window.q.out
##
@@ -166,7 +166,7 @@ POSTHOOK: Input: arc_view@wealth
A masked pattern was here
CBO PLAN:
HiveSortLimit(sort0=[$0], dir0=[ASC])
- HiveProject(quartile=[$0], total=[$1])
+ HiveProject(quartile=[CAST($0):DECIMAL(12, 1)], total=[$1])
Review comment:
Difference in precision and scale for column. Why has this changed?
##
File path: ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out
##
@@ -180,7 +180,7 @@ STAGE PLANS:
native: true
nativeConditionsMet:
hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine
tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true,
BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 257 Data size: 40092 Basic
stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2
(type: bigint), _col3 (type: double), _col4 (type: bigint), _col5 (type:
decimal(12,0)), _col6 (type: bigint)
+ value expressions: _col1 (type: bigint), _col2
(type: bigint), _col3 (type: double), _col4 (type: bigint), _col5 (type:
decimal(20,0)), _col6 (type: bigint)
Review comment:
Difference in precision and scale for column. Why has this changed?
##
File path:
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java
##
@@ -46,12 +46,4 @@ public IntWritable getWritableConstantValue() {
return value;
}
- @Override
- public int precision() {
Review comment:
I think this could lead to a change of behavior, e.g., all integers will
have same precision by default, but constant integer would have the precision
of the constant value?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515807)
Time Spent: 1h (was: 50m)
> Trailing zeros of constant decimal numbers are removed
> --
>
> Key: HIVE-24389
> URL: https://issues.apache.org/jira/browse/HIVE-24389
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Krisztian Kasa
>Assignee:
[jira] [Work logged] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?focusedWorklogId=515773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515773
]
ASF GitHub Bot logged work on HIVE-18728:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 21:47
Start Date: 23/Nov/20 21:47
Worklog Time Spent: 10m
Work Description: HunterL closed pull request #1689:
URL: https://github.com/apache/hive/pull/1689
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515773)
Time Spent: 0.5h (was: 20m)
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237685#comment-17237685
]
Oleksiy Sayankin commented on HIVE-18728:
-
[~hunterl] wrote:
{quote}
Unfortunately there's no way for me to change the PR on github to your
authorship, which is why I suggested you open your own PR and I close mine. If
you do chose to do this just leave a comment here or on the github link I
posted above and I'll be sure to close mine out.
{quote}
You can easily change the ownership if existing commit is on top of the PR.
{code}
git commit --amend --author="Name "
git push origin +master --force
{code}
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237684#comment-17237684
]
Oleksiy Sayankin commented on HIVE-18728:
-
Created https://github.com/apache/hive/pull/1699
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?focusedWorklogId=515748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515748
]
ASF GitHub Bot logged work on HIVE-18728:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 20:17
Start Date: 23/Nov/20 20:17
Worklog Time Spent: 10m
Work Description: oleksiy-sayankin opened a new pull request #1699:
URL: https://github.com/apache/hive/pull/1699
Signed-off-by: Oleksiy Sayankin
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515748)
Time Spent: 20m (was: 10m)
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237615#comment-17237615
]
Hunter Logan commented on HIVE-18728:
-
[~osayankin] Any user can create a PR on github, no permissions are required.
However a reviewer will still have to commit it to the master branch.
Unfortunately there's no way for me to change the PR on github to your
authorship, which is why I suggested you open your own PR and I close mine. If
you do chose to do this just leave a comment here or on the github link I
posted above and I'll be sure to close mine out.
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24397) Add the projection specification to the table request object and add placeholders in ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-24397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24397: -- Labels: pull-request-available (was: ) > Add the projection specification to the table request object and add > placeholders in ObjectStore.java > - > > Key: HIVE-24397 > URL: https://issues.apache.org/jira/browse/HIVE-24397 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Narayanan Venkateswaran >Assignee: Narayanan Venkateswaran >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24397) Add the projection specification to the table request object and add placeholders in ObjectStore.java
[
https://issues.apache.org/jira/browse/HIVE-24397?focusedWorklogId=515709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515709
]
ASF GitHub Bot logged work on HIVE-24397:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 18:30
Start Date: 23/Nov/20 18:30
Worklog Time Spent: 10m
Work Description: vihangk1 commented on a change in pull request #1681:
URL: https://github.com/apache/hive/pull/1681#discussion_r528908297
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1855,43 +1869,81 @@ private MTable getMTable(String catName, String db,
String table) {
for (String t : tbl_names) {
lowered_tbl_names.add(normalizeIdentifier(t));
}
+
query = pm.newQuery(MTable.class);
query.setFilter("database.name == db && database.catalogName == cat &&
tbl_names.contains(tableName)");
query.declareParameters("java.lang.String db, java.lang.String cat,
java.util.Collection tbl_names");
- Collection mtables = (Collection) query.execute(db, catName,
lowered_tbl_names);
- if (mtables == null || mtables.isEmpty()) {
-// Need to differentiate between an unmatched pattern and a
non-existent database
-dbExistsQuery = pm.newQuery(MDatabase.class, "name == db &&
catalogName == cat");
-dbExistsQuery.declareParameters("java.lang.String db, java.lang.String
cat");
-dbExistsQuery.setUnique(true);
-dbExistsQuery.setResult("name");
-String dbNameIfExists = (String) dbExistsQuery.execute(db, catName);
-if (org.apache.commons.lang3.StringUtils.isEmpty(dbNameIfExists)) {
- throw new UnknownDBException("Could not find database " +
- DatabaseName.getQualified(catName, db));
+
+ if (projectionSpec == null) {
+mtables = (List) query.execute(db, catName, lowered_tbl_names);
+ }
+ else if(projectionSpec.getFieldList() != null &&
projectionSpec.getFieldList().size() > 1) {
Review comment:
why do we need to code blockes for projectionSpec fieldList size > 1 and
== 1? the code looks very similar.
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
##
@@ -733,6 +733,27 @@ Table getTable(String catName, String dbName, String
tableName,
List getTableObjectsByName(String dbName, List tableNames)
throws MetaException, InvalidOperationException, UnknownDBException,
TException;
+
+ /**
+ * Get tables as objects (rather than just fetching their names). This is
more expensive and
+ * should only be used if you actually need all the information about the
tables.
+ * @param request GetTablesRequest Object.
+ * @return A list of objects representing the tables.
+ * Only the tables that can be retrieved from the database are
returned. For example,
+ * if none of the requested tables could be retrieved, an empty
list is returned.
+ * There is no guarantee of ordering of the returned tables.
+ * @throws InvalidOperationException
+ * The input to this operation is invalid (e.g., the list of tables
names is null)
+ * @throws UnknownDBException
+ * The requested database could not be fetched.
+ * @throws TException
+ * A thrift communication error occurred
+ * @throws MetaException
+ * Any other errors
+ */
+ List getTableObjectsByRequest(GetTablesRequest request)
Review comment:
Does the method name have to include byRequest? Its self-evident by the
signature right? How about getTables(GetTablesRequest request). Also, can we
keep the return type as GetTablesResponse here so that the interface is more
extendable in the future?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1855,43 +1869,81 @@ private MTable getMTable(String catName, String db,
String table) {
for (String t : tbl_names) {
lowered_tbl_names.add(normalizeIdentifier(t));
}
+
query = pm.newQuery(MTable.class);
query.setFilter("database.name == db && database.catalogName == cat &&
tbl_names.contains(tableName)");
query.declareParameters("java.lang.String db, java.lang.String cat,
java.util.Collection tbl_names");
- Collection mtables = (Collection) query.execute(db, catName,
lowered_tbl_names);
- if (mtables == null || mtables.isEmpty()) {
-// Need to differentiate between an unmatched pattern and a
non-existent database
-dbExistsQuery = pm.newQuery(MDatabase.class, "name == db &&
catalogName == cat");
-dbExistsQuery.declareParameters("java.lang.String db, java.lang.String
cat");
-dbExistsQuery.setUnique(true);
-
[jira] [Assigned] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hunter Logan reassigned HIVE-18728:
---
Assignee: Oleksiy Sayankin (was: Hunter Logan)
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237553#comment-17237553
]
Oleksiy Sayankin commented on HIVE-18728:
-
I have no enough permissions to create a PR in Hive so that's why I added the
files with patches. If you are able to set me as an author in the PR, then the
rest is ok from my side.
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Hunter Logan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237552#comment-17237552
]
Hunter Logan commented on HIVE-18728:
-
[~osayankin] I actually had a user inquire about the ability to do this
recently, I think it would be a useful feature.
I've copied your code and created a pull request here
[https://github.com/apache/hive/pull/1689] which I think will help this get
attention and into a release.
Now that I know this was not abandoned, if you'd like me to close the pull
request I've created so you may open your own I will do so, please just let me
know. Thanks!
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Hunter Logan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24414) Backport HIVE-19662 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24414: -- Labels: pull-request-available (was: ) > Backport HIVE-19662 to branch-3.1 > - > > Key: HIVE-24414 > URL: https://issues.apache.org/jira/browse/HIVE-24414 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This JIRA proposes to backport HIVE-19662 to branch-3.1 and upgrade Avro to > 1.8.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24414) Backport HIVE-19662 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24414?focusedWorklogId=515686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515686 ] ASF GitHub Bot logged work on HIVE-24414: - Author: ASF GitHub Bot Created on: 23/Nov/20 17:46 Start Date: 23/Nov/20 17:46 Worklog Time Spent: 10m Work Description: sunchao opened a new pull request #1698: URL: https://github.com/apache/hive/pull/1698 ### What changes were proposed in this pull request? This backports HIVE-19662 to branch-3.1 ### Why are the changes needed? Hive currently use deprecated Avro APIs in Avro version 1.8.x and below, which are removed since 1.9. Therefore if downstream applications use Hive and want to upgrade its own Avro dependency to 1.9.x or up, there could be incompatibility issues. In order to change this, we'll need HIVE-24324 which removes the deprecated API usage with some others in 1.8.x. This is a pre-requisite for that. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515686) Remaining Estimate: 0h Time Spent: 10m > Backport HIVE-19662 to branch-3.1 > - > > Key: HIVE-24414 > URL: https://issues.apache.org/jira/browse/HIVE-24414 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This JIRA proposes to backport HIVE-19662 to branch-3.1 and upgrade Avro to > 1.8.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24414) Backport HIVE-19662 to branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned HIVE-24414: --- > Backport HIVE-19662 to branch-3.1 > - > > Key: HIVE-24414 > URL: https://issues.apache.org/jira/browse/HIVE-24414 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > > This JIRA proposes to backport HIVE-19662 to branch-3.1 and upgrade Avro to > 1.8.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24379) Backport HIVE-19662 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-24379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HIVE-24379. - Resolution: Fixed > Backport HIVE-19662 to branch-2.3 > - > > Key: HIVE-24379 > URL: https://issues.apache.org/jira/browse/HIVE-24379 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > In order to backport HIVE-24324 to branch-2.3 to remove deprecated Avro API > so that downstream applications who depend on Hive 2.3.x can upgrade their > Avro version, we'll need to first backport HIVE-19662 and bump Avro version > in branch-2.3 first as it is currently on 1.7.x, while HIVE-24324 requires > Avro 1.8.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237539#comment-17237539
]
Oleksiy Sayankin commented on HIVE-18728:
-
For some reason my solution was ignored by the community. I am still here and I
wish to commit this to Hive
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Hunter Logan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515642
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 16:21
Start Date: 23/Nov/20 16:21
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #1693:
URL: https://github.com/apache/hive/pull/1693#discussion_r528829672
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -590,6 +587,39 @@ public Object run() throws Exception {
return true;
}
+ private void failCompactionIfSetForTest() {
+if(conf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST) &&
conf.getBoolVar(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION)) {
+ throw new
RuntimeException(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION.name() + "=true");
+}
+ }
+
+ private void runCompactionViaMrJob(CompactionInfo ci, Table t, Partition p,
StorageDescriptor sd,
+ ValidCompactorWriteIdList tblValidWriteIds, StringBuilder jobName,
AcidUtils.Directory dir, StatsUpdater su)
+ throws IOException, HiveException, InterruptedException {
+final CompactorMR mr = new CompactorMR();
+if (runJobAsSelf(ci.runAs)) {
+ mr.run(conf, jobName.toString(), t, p, sd, tblValidWriteIds, ci, su,
msc, dir);
+} else {
+ UserGroupInformation ugi = UserGroupInformation.createProxyUser(ci.runAs,
+ UserGroupInformation.getLoginUser());
+ final Partition fp = p;
+ final CompactionInfo fci = ci;
+ ugi.doAs(new PrivilegedExceptionAction() {
Review comment:
Noted, will do! Thanks for taking a look at this!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515642)
Time Spent: 1.5h (was: 1h 20m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Work logged] (HIVE-24245) Vectorized PTF with count and distinct over partition producing incorrect results.
[
https://issues.apache.org/jira/browse/HIVE-24245?focusedWorklogId=515635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515635
]
ASF GitHub Bot logged work on HIVE-24245:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 16:09
Start Date: 23/Nov/20 16:09
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1649:
URL: https://github.com/apache/hive/pull/1649#discussion_r528820635
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorBytesCountDistinct.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+import org.apache.hive.common.util.Murmur3;
+
+/**
+ * Bytes (String) implementation for VectorPTFEvaluatorCountDistinct.
+ */
+public class VectorPTFEvaluatorBytesCountDistinct extends
VectorPTFEvaluatorCountDistinct {
+
+ public VectorPTFEvaluatorBytesCountDistinct(WindowFrameDef windowFrameDef,
+ VectorExpression inputVecExpr, int outputColumnNum) {
+super(windowFrameDef, inputVecExpr, outputColumnNum);
+resetEvaluator();
+ }
+
+ protected Object getValue(ColumnVector colVector, int i) {
+BytesColumnVector inV = (BytesColumnVector) colVector;
+return Murmur3.hash32(inV.vector[i], inV.start[i], inV.length[i],
Murmur3.DEFAULT_SEED);
Review comment:
What is the benefit of returning Murmur hash directly here? Adding a
String to the HashSet will also hash the input (so essentially we are hashing
twice)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515635)
Time Spent: 0.5h (was: 20m)
> Vectorized PTF with count and distinct over partition producing incorrect
> results.
> --
>
> Key: HIVE-24245
> URL: https://issues.apache.org/jira/browse/HIVE-24245
> Project: Hive
> Issue Type: Bug
> Components: Hive, PTF-Windowing, Vectorization
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Chiran Ravani
>Assignee: László Bodor
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Vectorized PTF for count and distinct over partition is broken. It produces
> incorrect results.
> Below is the test case.
> {code}
> CREATE TABLE bigd781b_new (
> id int,
> txt1 string,
> txt2 string,
> cda_date int,
> cda_job_name varchar(12));
> INSERT INTO bigd781b_new VALUES
> (1,'2010005759','7164335675012038',20200528,'load1'),
> (2,'2010005759','7164335675012038',20200528,'load2');
> {code}
> Running below query produces incorrect results
> {code}
> SELECT
> txt1,
> txt2,
> count(distinct txt1) over(partition by txt1) as n,
> count(distinct txt2) over(partition by txt2) as m
> FROM bigd781b_new
> {code}
> as below.
> {code}
> +-+---+++
> |txt1 | txt2| n | m |
> +-+---+++
> | 2010005759 | 7164335675012038 | 2 | 2 |
> | 2010005759 | 7164335675012038 | 2 | 2 |
> +-+---+++
> {code}
> While the correct output would be
> {code}
> +-+---+++
> |txt1 | txt2| n | m |
> +-+---+++
> | 2010005759 | 7164335675012038 | 1 | 1 |
> | 2010005759 | 7164335675012038 | 1
[jira] [Work logged] (HIVE-24245) Vectorized PTF with count and distinct over partition producing incorrect results.
[
https://issues.apache.org/jira/browse/HIVE-24245?focusedWorklogId=515627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515627
]
ASF GitHub Bot logged work on HIVE-24245:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 15:50
Start Date: 23/Nov/20 15:50
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1649:
URL: https://github.com/apache/hive/pull/1649#discussion_r528797991
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorCountDistinct.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+import org.apache.hive.common.util.Murmur3;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * This class evaluates count(column) for a PTF group where a distinct keyword
is applied to the
+ * partitioning column itself, e.g.:
+ *
+ * SELECT
+ * txt1,
+ * txt2,
+ * count(distinct txt1) over(partition by txt1) as n,
+ * count(distinct txt2) over(partition by txt2) as m
+ * FROM example;
+ *
+ * In this case, the framework is still supposed to ensure sorting
+ * on the key (let's say txt1 for the first Reducer stage), but the original
+ * VectorPTFEvaluatorCount is not aware that a distinct keyword was applied
+ * to the key column. This case would be simple, because such function should
+ * return 1 every time. However, that's just a corner-case, a real scenario is
+ * when the partitioning column is not the same. In such cases, a real count
+ * distinct implementation is needed:
+ *
+ * SELECT
+ * txt1,
+ * txt2,
+ * count(distinct txt2) over(partition by txt1) as n,
+ * count(distinct txt1) over(partition by txt2) as m
+ * FROM example;
+ */
+public abstract class VectorPTFEvaluatorCountDistinct extends
VectorPTFEvaluatorCount {
+
+ protected Set uniqueObjects;
+
+ public VectorPTFEvaluatorCountDistinct(WindowFrameDef windowFrameDef,
+ VectorExpression inputVecExpr, int outputColumnNum) {
+super(windowFrameDef, inputVecExpr, outputColumnNum);
+resetEvaluator();
+ }
+
+ @Override
+ public void evaluateGroupBatch(VectorizedRowBatch batch) throws
HiveException {
+
+evaluateInputExpr(batch);
+
+// We do not filter when PTF is in reducer.
+Preconditions.checkState(!batch.selectedInUse);
+
+final int size = batch.size;
+if (size == 0) {
+ return;
+}
+ColumnVector colVector = batch.cols[inputColumnNum];
+if (colVector.isRepeating) {
+ if (colVector.noNulls || !colVector.isNull[0]) {
+countValue(colVector, 0);
+ }
+} else {
+ boolean[] batchIsNull = colVector.isNull;
+ for (int i = 0; i < size; i++) {
+if (!batchIsNull[i]) {
+ countValue(colVector, i);
+}
+ }
+}
+ }
+
+ protected void countValue(ColumnVector colVector, int i) {
+Object value = getValue(colVector, i);
+if (!uniqueObjects.contains(value)) {
+ uniqueObjects.add(value);
Review comment:
We do check if the element is present twice here -- add method already
checks if the element is part of the Set so we dont need to call contains. I
would just add a comment here instead.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorCountDistinct.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
[jira] [Updated] (HIVE-24412) Extract common parts of HiveDeCorrelator
[ https://issues.apache.org/jira/browse/HIVE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24412: -- Labels: pull-request-available (was: ) > Extract common parts of HiveDeCorrelator > > > Key: HIVE-24412 > URL: https://issues.apache.org/jira/browse/HIVE-24412 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a > Hive version and the implementation is the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24412) Extract common parts of HiveDeCorrelator
[
https://issues.apache.org/jira/browse/HIVE-24412?focusedWorklogId=515619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515619
]
ASF GitHub Bot logged work on HIVE-24412:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 15:37
Start Date: 23/Nov/20 15:37
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #1697:
URL: https://github.com/apache/hive/pull/1697
### What changes were proposed in this pull request?
1. When methods which shares the same implementation one of the method's
implementation is changed to call its pair. Ex.
```
public Frame decorrelateRel(HiveSortLimit rel) {
return decorrelateRel((Sort) rel);
}
public Frame decorrelateRel(Sort rel) {
...
```
2. Variations of `removeCorrelationExpr` has the same implementation only
difference was the number of parameters of the methods. This patch removes the
duplicates and variants which has fewer parameters calls the most specific and
pass default values for missing parameters.
### Why are the changes needed?
Decrease the amount of redundant code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No functionality was altered. Execute some q tests which has subquery with
`in`, `exists` like `subquery_exists.q`, `subquery_in.q`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515619)
Remaining Estimate: 0h
Time Spent: 10m
> Extract common parts of HiveDeCorrelator
>
>
> Key: HIVE-24412
> URL: https://issues.apache.org/jira/browse/HIVE-24412
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a
> Hive version and the implementation is the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24412) Extract common parts of HiveDeCorrelator
[ https://issues.apache.org/jira/browse/HIVE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-24412: - > Extract common parts of HiveDeCorrelator > > > Key: HIVE-24412 > URL: https://issues.apache.org/jira/browse/HIVE-24412 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Fix For: 4.0.0 > > > Some methods in *HiveRelDecorrelator.decorrelateRel()* has a Calcite and a > Hive version and the implementation is the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515599
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 15:16
Start Date: 23/Nov/20 15:16
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1693:
URL: https://github.com/apache/hive/pull/1693#discussion_r528777347
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -590,6 +587,39 @@ public Object run() throws Exception {
return true;
}
+ private void failCompactionIfSetForTest() {
+if(conf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST) &&
conf.getBoolVar(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION)) {
+ throw new
RuntimeException(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION.name() + "=true");
+}
+ }
+
+ private void runCompactionViaMrJob(CompactionInfo ci, Table t, Partition p,
StorageDescriptor sd,
+ ValidCompactorWriteIdList tblValidWriteIds, StringBuilder jobName,
AcidUtils.Directory dir, StatsUpdater su)
+ throws IOException, HiveException, InterruptedException {
+final CompactorMR mr = new CompactorMR();
+if (runJobAsSelf(ci.runAs)) {
+ mr.run(conf, jobName.toString(), t, p, sd, tblValidWriteIds, ci, su,
msc, dir);
+} else {
+ UserGroupInformation ugi = UserGroupInformation.createProxyUser(ci.runAs,
+ UserGroupInformation.getLoginUser());
+ final Partition fp = p;
+ final CompactionInfo fci = ci;
+ ugi.doAs(new PrivilegedExceptionAction() {
Review comment:
You can use lambda like this:
ugi.doAs((PrivilegedExceptionAction) () -> {
mr.run(conf, jobName.toString(), t, fp, sd, tblValidWriteIds,
fci, su, msc, dir);
return null;
});
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515599)
Time Spent: 1h 20m (was: 1h 10m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515597
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 15:15
Start Date: 23/Nov/20 15:15
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1693:
URL: https://github.com/apache/hive/pull/1693#discussion_r528776095
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
##
@@ -219,23 +219,6 @@ void run(HiveConf conf, String jobName, Table t, Partition
p, StorageDescriptor
CompactionInfo ci, Worker.StatsUpdater su, IMetaStoreClient msc,
Directory dir)
throws IOException, HiveException {
-if(conf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST) &&
conf.getBoolVar(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION)) {
- throw new
RuntimeException(HiveConf.ConfVars.HIVETESTMODEFAILCOMPACTION.name() + "=true");
-}
-
-/*
- Try to run compaction via HiveQL queries.
- Compaction for MM tables happens here, or run compaction for Crud tables
if query-based compaction is enabled.
- todo Find a more generic approach to collecting files in the same logical
bucket to compact within the same task
- (currently we're using Tez split grouping).
- */
-QueryCompactor queryCompactor = QueryCompactorFactory.getQueryCompactor(t,
conf, ci);
Review comment:
This is so much cleaner, +100 for this :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515597)
Time Spent: 1h 10m (was: 1h)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515546
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:43
Start Date: 23/Nov/20 13:43
Worklog Time Spent: 10m
Work Description: klcopp opened a new pull request #1693:
URL: https://github.com/apache/hive/pull/1693
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
See HIVE-24410
### How was this patch tested?
Manually on a secure cluster
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515546)
Time Spent: 1h (was: 50m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515545
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:43
Start Date: 23/Nov/20 13:43
Worklog Time Spent: 10m
Work Description: klcopp closed pull request #1693:
URL: https://github.com/apache/hive/pull/1693
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515545)
Time Spent: 50m (was: 40m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:336)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:146)
> at
>
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515539
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:25
Start Date: 23/Nov/20 13:25
Worklog Time Spent: 10m
Work Description: klcopp opened a new pull request #1693:
URL: https://github.com/apache/hive/pull/1693
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
See HIVE-24410
### How was this patch tested?
Manually on a secure cluster
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515539)
Time Spent: 40m (was: 0.5h)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515537
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:24
Start Date: 23/Nov/20 13:24
Worklog Time Spent: 10m
Work Description: klcopp commented on pull request #1693:
URL: https://github.com/apache/hive/pull/1693#issuecomment-732159751
closing and reopening so precommit tests run
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515537)
Time Spent: 20m (was: 10m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:336)
> at
>
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515538
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:24
Start Date: 23/Nov/20 13:24
Worklog Time Spent: 10m
Work Description: klcopp closed pull request #1693:
URL: https://github.com/apache/hive/pull/1693
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515538)
Time Spent: 0.5h (was: 20m)
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:336)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:146)
> at
>
[jira] [Updated] (HIVE-24249) Create View fails if a materialized view exists with the same query
[
https://issues.apache.org/jira/browse/HIVE-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24249:
--
Labels: pull-request-available (was: )
> Create View fails if a materialized view exists with the same query
> ---
>
> Key: HIVE-24249
> URL: https://issues.apache.org/jira/browse/HIVE-24249
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> create table t1(col0 int) STORED AS ORC
> TBLPROPERTIES ('transactional'='true');
> create materialized view mv1 as
> select * from t1 where col0 > 2;
> create view v1 as
> select sub.* from (select * from t1 where col0 > 2) sub
> where sub.col0 = 10;
> {code}
> The planner realize that the view definition has a subquery which match the
> materialized view query and replaces it to the materialized view scan.
> {code:java}
> HiveProject($f0=[CAST(10):INTEGER])
> HiveFilter(condition=[=(10, $0)])
> HiveTableScan(table=[[default, mv1]], table:alias=[default.mv1])
> {code}
> Then exception is thrown:
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException: View definition
> references materialized view default.mv1
> at
> org.apache.hadoop.hive.ql.ddl.view.create.CreateViewAnalyzer.validateCreateView(CreateViewAnalyzer.java:211)
> at
> org.apache.hadoop.hive.ql.ddl.view.create.CreateViewAnalyzer.analyzeInternal(CreateViewAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:301)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:174)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:415)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:364)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:358)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> at
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at
[jira] [Work logged] (HIVE-24249) Create View fails if a materialized view exists with the same query
[
https://issues.apache.org/jira/browse/HIVE-24249?focusedWorklogId=515531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515531
]
ASF GitHub Bot logged work on HIVE-24249:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 13:08
Start Date: 23/Nov/20 13:08
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #1696:
URL: https://github.com/apache/hive/pull/1696
### What changes were proposed in this pull request?
Disable materialized view rewrite at view creation.
### Why are the changes needed?
Hive doesn't support view creation when one of the source tables is a
materialized view. When the `create view` command is executed the select
statement is compiled and the optimizer applies materialized view rewrites if
any suitable materialized view exists. A successful rewrite leads to Exception:
```
org.apache.hadoop.hive.ql.parse.SemanticException: View definition
references materialized view default.mv1
```
See the jira for full stacktrace.
### Does this PR introduce _any_ user-facing change?
Users can create views even if its query is matching with an existing
materialized view query. No SemanticException should be thrown because of that.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=create_view_when_mv_exists.q -pl
itests/qtest -Pitests
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515531)
Remaining Estimate: 0h
Time Spent: 10m
> Create View fails if a materialized view exists with the same query
> ---
>
> Key: HIVE-24249
> URL: https://issues.apache.org/jira/browse/HIVE-24249
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> create table t1(col0 int) STORED AS ORC
> TBLPROPERTIES ('transactional'='true');
> create materialized view mv1 as
> select * from t1 where col0 > 2;
> create view v1 as
> select sub.* from (select * from t1 where col0 > 2) sub
> where sub.col0 = 10;
> {code}
> The planner realize that the view definition has a subquery which match the
> materialized view query and replaces it to the materialized view scan.
> {code:java}
> HiveProject($f0=[CAST(10):INTEGER])
> HiveFilter(condition=[=(10, $0)])
> HiveTableScan(table=[[default, mv1]], table:alias=[default.mv1])
> {code}
> Then exception is thrown:
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException: View definition
> references materialized view default.mv1
> at
> org.apache.hadoop.hive.ql.ddl.view.create.CreateViewAnalyzer.validateCreateView(CreateViewAnalyzer.java:211)
> at
> org.apache.hadoop.hive.ql.ddl.view.create.CreateViewAnalyzer.analyzeInternal(CreateViewAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:301)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:174)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:415)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:364)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:358)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170)
> at
>
[jira] [Work logged] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?focusedWorklogId=515524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515524 ] ASF GitHub Bot logged work on HIVE-24411: - Author: ASF GitHub Bot Created on: 23/Nov/20 12:49 Start Date: 23/Nov/20 12:49 Worklog Time Spent: 10m Work Description: dengzhhu653 opened a new pull request #1695: URL: https://github.com/apache/hive/pull/1695 …fMemoryError ### What changes were proposed in this pull request? Check the root cause of failed tasks to see if that is an instance of OutOfMemoryError. ### Why are the changes needed? Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the HiveServer2 in case of OutOfMemoryError when executing the tasks. The exception is obtained by calling method `future.get()`, however the exception may never be an instance of OutOfMemoryError, as the exception is wrapped in ExecutionException, see the method report in class FutureTask. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515524) Remaining Estimate: 0h Time Spent: 10m > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method _future.get()_, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24411: -- Labels: pull-request-available (was: ) > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method _future.get()_, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-24411: --- Description: Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the HiveServer2 in case of OutOfMemoryError when executing the tasks. The exception is obtained by calling method _future.get()_, however the exception may never be an instance of OutOfMemoryError, as the exception is wrapped in ExecutionException, see the method report in class FutureTask. (was: Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the HiveServer2 in case of OutOfMemoryError when executing the tasks. The exception is obtained by calling method `future.get()`, however the exception may never be an instance of OutOfMemoryError, as the exception is wrapped in ExecutionException, see the method report in class FutureTask.) > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method _future.get()_, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24411) Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError
[ https://issues.apache.org/jira/browse/HIVE-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-24411: -- > Make ThreadPoolExecutorWithOomHook more awareness of OutOfMemoryError > - > > Key: HIVE-24411 > URL: https://issues.apache.org/jira/browse/HIVE-24411 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > > Now the ThreadPoolExecutorWithOomHook invokes some oom hooks and stops the > HiveServer2 in case of OutOfMemoryError when executing the tasks. The > exception is obtained by calling method `future.get()`, however the exception > may never be an instance of OutOfMemoryError, as the exception is wrapped in > ExecutionException, see the method report in class FutureTask. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24138) Llap external client flow is broken due to netty shading
[
https://issues.apache.org/jira/browse/HIVE-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-24138.
-
Fix Version/s: 4.0.0
Resolution: Fixed
> Llap external client flow is broken due to netty shading
>
>
> Key: HIVE-24138
> URL: https://issues.apache.org/jira/browse/HIVE-24138
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We shaded netty in hive-exec in -
> https://issues.apache.org/jira/browse/HIVE-23073
> This breaks LLAP external client flow on LLAP daemon side -
> LLAP daemon stacktrace -
> {code}
> 2020-09-09T18:22:13,413 INFO [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] llap.LlapOutputFormat: Returning
> writer for: attempt_497418324441977_0004_0_00_00_0
> 2020-09-09T18:22:13,419 ERROR [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] tez.MapRecordSource:
> java.lang.NoSuchMethodError:
> org.apache.arrow.memory.BufferAllocator.buffer(I)Lorg/apache/hive/io/netty/buffer/ArrowBuf;
> at
> org.apache.hadoop.hive.llap.WritableByteChannelAdapter.write(WritableByteChannelAdapter.java:96)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:74)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:57)
> at
> org.apache.arrow.vector.ipc.WriteChannel.writeIntLittleEndian(WriteChannel.java:89)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:88)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:130)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:102)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:85)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:842)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Arrow method signature mismatch mainly happens due to the fact that arrow
> contains some classes which are packaged under {{io.netty.buffer.*}} -
> {code}
> io.netty.buffer.ArrowBuf
>
[jira] [Commented] (HIVE-24138) Llap external client flow is broken due to netty shading
[
https://issues.apache.org/jira/browse/HIVE-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237322#comment-17237322
]
László Bodor commented on HIVE-24138:
-
merged to master, thanks [~ayushtkn] for the patch
> Llap external client flow is broken due to netty shading
>
>
> Key: HIVE-24138
> URL: https://issues.apache.org/jira/browse/HIVE-24138
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We shaded netty in hive-exec in -
> https://issues.apache.org/jira/browse/HIVE-23073
> This breaks LLAP external client flow on LLAP daemon side -
> LLAP daemon stacktrace -
> {code}
> 2020-09-09T18:22:13,413 INFO [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] llap.LlapOutputFormat: Returning
> writer for: attempt_497418324441977_0004_0_00_00_0
> 2020-09-09T18:22:13,419 ERROR [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] tez.MapRecordSource:
> java.lang.NoSuchMethodError:
> org.apache.arrow.memory.BufferAllocator.buffer(I)Lorg/apache/hive/io/netty/buffer/ArrowBuf;
> at
> org.apache.hadoop.hive.llap.WritableByteChannelAdapter.write(WritableByteChannelAdapter.java:96)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:74)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:57)
> at
> org.apache.arrow.vector.ipc.WriteChannel.writeIntLittleEndian(WriteChannel.java:89)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:88)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:130)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:102)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:85)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:842)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Arrow method signature mismatch mainly happens due to the fact that arrow
> contains some classes which are packaged under {{io.netty.buffer.*}} -
> {code}
>
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515517
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 12:24
Start Date: 23/Nov/20 12:24
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528666168
##
File path: service/src/java/org/apache/hive/service/cli/operation/Operation.java
##
@@ -88,8 +89,14 @@ protected Operation(HiveSession parentSession, OperationType
opType) {
}
protected Operation(HiveSession parentSession,
- Map confOverlay, OperationType opType) {
+ Map confOverlay, OperationType opType) {
+this(parentSession, confOverlay, opType, false);
+ }
+
+ protected Operation(HiveSession parentSession,
+ Map confOverlay, OperationType opType, boolean embedded)
{
Review comment:
The SqlOperation is called with embedded=true when it is instantiated
from a running hplsql script.
```
new SQLOperation(parentSession, statement, confOverlay, runAsync,
queryTimeout, hplSqlMode());
```
Could elaborate more on that idea of having it as a first class citizen?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515517)
Time Spent: 2h 50m (was: 2h 40m)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline while we'er in hplsql mode. Hive will create a hplsql
> interpreter and store it in the session state. A new HplSqlOperation is
> created to run the script on the interpreter.
> HPLSQL knows how to execute the for loop, but i'll call Hive to run the
> select expression. The HplSqlOperation is
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515516
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 12:21
Start Date: 23/Nov/20 12:21
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528664244
##
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##
@@ -18,18 +18,67 @@
package org.apache.hive.jdbc;
-import com.google.common.annotations.VisibleForTesting;
+import static org.apache.hadoop.hive.conf.Constants.HPLSQL_MODE;
+import java.io.BufferedReader;
+import java.io.DataInputStream;
+import java.io.File;
Review comment:
I'll try to do something about it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515516)
Time Spent: 2h 40m (was: 2.5h)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline while we'er in hplsql mode. Hive will create a hplsql
> interpreter and store it in the session state. A new HplSqlOperation is
> created to run the script on the interpreter.
> HPLSQL knows how to execute the for loop, but i'll call Hive to run the
> select expression. The HplSqlOperation is notified when the select reads a
> row and accumulates the rows into a RowSet (memory consumption need to be
> considered here) which can be retrieved via thrift from the client side.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515515
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 12:20
Start Date: 23/Nov/20 12:20
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528663855
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/executor/JdbcQueryExecutor.java
##
@@ -0,0 +1,101 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hive.hplsql.executor;
+
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.antlr.v4.runtime.ParserRuleContext;
+import org.apache.hive.hplsql.Exec;
+import org.apache.hive.hplsql.Query;
+
+public class JdbcQueryExecutor implements QueryExecutor {
+ private final Exec exec;
+
+ public JdbcQueryExecutor(Exec exec) {
+this.exec = exec;
+ }
+
+ @Override
+ public QueryResult executeQuery(String sql, ParserRuleContext ctx) {
+String conn = exec.getStatementConnection();
+Query query = exec.executeQuery(ctx, new Query(sql), conn);
+ResultSet resultSet = query.getResultSet();
+if (resultSet == null) { // offline mode
+ return new QueryResult(null, () -> new
Metadata(Collections.emptyList()), query.getException());
+} else {
+ return new QueryResult(new JdbcRowResult(resultSet), () ->
metadata(resultSet), query.getException());
+}
+ }
+
+ private static Metadata metadata(ResultSet resultSet) {
+try {
+ ResultSetMetaData meta = resultSet.getMetaData();
+ List colMetas = new ArrayList<>();
+ for (int i = 1; i <= meta.getColumnCount(); i++) {
+colMetas.add(new ColumnMeta(
+meta.getColumnName(i), meta.getColumnTypeName(i),
meta.getColumnType(i)));
+ }
+ return new Metadata(colMetas);
+} catch (SQLException e) {
+ throw new QueryException(e);
+}
+ }
+
+ private static class JdbcRowResult implements RowResult {
+private final ResultSet resultSet;
+
+private JdbcRowResult(ResultSet resultSet) {
+ this.resultSet = resultSet;
+}
+
+@Override
+public boolean next() {
+ try {
+return resultSet.next();
+ } catch (SQLException e) {
+throw new QueryException(e);
+ }
+}
+
+@Override
+public T get(int columnIndex, Class type) {
+ try {
+return resultSet.getObject(columnIndex, type);
Review comment:
Right, this is off by one. the Jdbc adaptor was added for compatibility,
I'm not sure for how long we'll want to support it but I fixed indexing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515515)
Time Spent: 2.5h (was: 2h 20m)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515513
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 12:19
Start Date: 23/Nov/20 12:19
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528663216
##
File path: hplsql/src/main/java/org/apache/hive/hplsql/Copy.java
##
@@ -233,16 +219,16 @@ void copyToFile(HplsqlParser.Copy_stmtContext ctx, Query
query) throws Exception
sql = "INSERT INTO " + sqlInsertName + " VALUES (";
rowdel = ");\n".getBytes();
}
- while (rs.next()) {
+ while (query.next()) {
if (sqlInsert) {
out.write(sql.getBytes());
}
-for (int i = 1; i <= cols; i++) {
- if (i > 1) {
+for (int i = 0; i < cols; i++) {
Review comment:
I supposed to be 0 based everywhere.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515513)
Time Spent: 2h 20m (was: 2h 10m)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline while we'er in hplsql mode. Hive will create a hplsql
> interpreter and store it in the session state. A new HplSqlOperation is
> created to run the script on the interpreter.
> HPLSQL knows how to execute the for loop, but i'll call Hive to run the
> select expression. The HplSqlOperation is notified when the select reads a
> row and accumulates the rows into a RowSet (memory consumption need to be
> considered here) which can be retrieved via thrift from the client side.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515512
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 12:18
Start Date: 23/Nov/20 12:18
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528662805
##
File path: hplsql/src/main/java/org/apache/hive/hplsql/Cmp.java
##
@@ -138,28 +147,26 @@ else if (query2.error()) {
exec.signal(query2);
return null;
}
-ResultSet rs1 = query1.getResultSet();
-ResultSet rs2 = query2.getResultSet();
-if (rs1 == null || rs2 == null) {
+if (query1 == null || query2 == null) {
exec.setSqlCode(-1);
return null;
}
boolean equal = true;
tests = 0;
failedTests = 0;
try {
- ResultSetMetaData rm1 = rs1.getMetaData();
- ResultSetMetaData rm2 = rs2.getMetaData();
- int cnt1 = rm1.getColumnCount();
- int cnt2 = rm2.getColumnCount();
+ Metadata rm1 = query1.metadata();
+ Metadata rm2 = query2.metadata();
+ int cnt1 = rm1.columnCount();
+ int cnt2 = rm2.columnCount();
tests = cnt1;
- while (rs1.next() && rs2.next()) {
+ while (query1.next() && query2.next()) {
for (int i = 1; i <= tests; i++) {
Review comment:
Yes, I missed this. It should have been 0 based. Fixed it. I also
removed the 2 threads since Tez doesn't support concurrent queries within the
same session.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515512)
Time Spent: 2h 10m (was: 2h)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline
[jira] [Work logged] (HIVE-24138) Llap external client flow is broken due to netty shading
[
https://issues.apache.org/jira/browse/HIVE-24138?focusedWorklogId=515502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515502
]
ASF GitHub Bot logged work on HIVE-24138:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 11:56
Start Date: 23/Nov/20 11:56
Worklog Time Spent: 10m
Work Description: abstractdog merged pull request #1491:
URL: https://github.com/apache/hive/pull/1491
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515502)
Time Spent: 1h (was: 50m)
> Llap external client flow is broken due to netty shading
>
>
> Key: HIVE-24138
> URL: https://issues.apache.org/jira/browse/HIVE-24138
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Shubham Chaurasia
>Assignee: Ayush Saxena
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We shaded netty in hive-exec in -
> https://issues.apache.org/jira/browse/HIVE-23073
> This breaks LLAP external client flow on LLAP daemon side -
> LLAP daemon stacktrace -
> {code}
> 2020-09-09T18:22:13,413 INFO [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] llap.LlapOutputFormat: Returning
> writer for: attempt_497418324441977_0004_0_00_00_0
> 2020-09-09T18:22:13,419 ERROR [TezTR-222977_4_0_0_0_0
> (497418324441977_0004_0_00_00_0)] tez.MapRecordSource:
> java.lang.NoSuchMethodError:
> org.apache.arrow.memory.BufferAllocator.buffer(I)Lorg/apache/hive/io/netty/buffer/ArrowBuf;
> at
> org.apache.hadoop.hive.llap.WritableByteChannelAdapter.write(WritableByteChannelAdapter.java:96)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:74)
> at org.apache.arrow.vector.ipc.WriteChannel.write(WriteChannel.java:57)
> at
> org.apache.arrow.vector.ipc.WriteChannel.writeIntLittleEndian(WriteChannel.java:89)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:88)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:130)
> at
> org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:102)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:85)
> at
> org.apache.hadoop.hive.llap.LlapArrowRecordWriter.write(LlapArrowRecordWriter.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:842)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
> at
[jira] [Work logged] (HIVE-24386) Add builder methods for GetTablesRequest and GetPartitionsRequest to HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-24386?focusedWorklogId=515483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515483 ] ASF GitHub Bot logged work on HIVE-24386: - Author: ASF GitHub Bot Created on: 23/Nov/20 11:25 Start Date: 23/Nov/20 11:25 Worklog Time Spent: 10m Work Description: vnhive opened a new pull request #1694: URL: https://github.com/apache/hive/pull/1694 HIVE-24386 : Add builder methods for GetTablesRequest and GetPartitionsRequest to HiveMetaStoreClient This patch builds over the patch for HIVE-24397 and adds builder methods for the request and the projection specification classes of Tables and Partitions. The relevant unit tests have also been updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515483) Remaining Estimate: 0h Time Spent: 10m > Add builder methods for GetTablesRequest and GetPartitionsRequest to > HiveMetaStoreClient > > > Key: HIVE-24386 > URL: https://issues.apache.org/jira/browse/HIVE-24386 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Narayanan Venkateswaran >Assignee: Narayanan Venkateswaran >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Builder methods for GetTablesRequest and GetPartitionsRequest should be added > to the HiveMetaStoreClient class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24386) Add builder methods for GetTablesRequest and GetPartitionsRequest to HiveMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24386: -- Labels: pull-request-available (was: ) > Add builder methods for GetTablesRequest and GetPartitionsRequest to > HiveMetaStoreClient > > > Key: HIVE-24386 > URL: https://issues.apache.org/jira/browse/HIVE-24386 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Narayanan Venkateswaran >Assignee: Narayanan Venkateswaran >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Builder methods for GetTablesRequest and GetPartitionsRequest should be added > to the HiveMetaStoreClient class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?focusedWorklogId=515449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515449
]
ASF GitHub Bot logged work on HIVE-24410:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 10:23
Start Date: 23/Nov/20 10:23
Worklog Time Spent: 10m
Work Description: klcopp opened a new pull request #1693:
URL: https://github.com/apache/hive/pull/1693
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
See HIVE-24410
### How was this patch tested?
Manually on a secure cluster
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515449)
Remaining Estimate: 0h
Time Spent: 10m
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
>
[jira] [Updated] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24410:
--
Labels: pull-request-available (was: )
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:336)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:146)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> at
>
[jira] [Work logged] (HIVE-24353) performance: Refactor TimestampTZ parsing
[
https://issues.apache.org/jira/browse/HIVE-24353?focusedWorklogId=515448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515448
]
ASF GitHub Bot logged work on HIVE-24353:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 10:19
Start Date: 23/Nov/20 10:19
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1650:
URL: https://github.com/apache/hive/pull/1650#discussion_r528596005
##
File path:
common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java
##
@@ -84,6 +86,18 @@ public void testVariations() {
TimestampTZUtil.parse("2017-05-08 07:45:00-3:00");
}
+ @Test
+ public void testPerformance() {
+for (int i = 0; i < 100; i++) {
Review comment:
I see -- usually we test the correctness with Junit tests and the
performance sensitive parts with jmh tests (where you could for example test
and compare the old behaviour with the new) -- check itests->hive-jmh package.
Not critical for this change though
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515448)
Time Spent: 40m (was: 0.5h)
> performance: Refactor TimestampTZ parsing
> -
>
> Key: HIVE-24353
> URL: https://issues.apache.org/jira/browse/HIVE-24353
> Project: Hive
> Issue Type: Improvement
>Reporter: Vincenz Priesnitz
>Assignee: Vincenz Priesnitz
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I found that for datasets that contain a lot of timestamps (without
> timezones) hive spends the majority of time in TimestampTZUtil.parse, in
> particular constructing stractraces for the try-catch blocks.
> When parsing TimestampTZ we are currently using a fallback chain with several
> try-catch blocks. For a common timestamp string without a timezone, we
> currently throw and catch 2 exceptions, and actually parse the string twice.
> I propose a refactor, that parses the string once and then expresses the
> fallback chain with queries to the parsed TemporalAccessor.
>
> Update: I added a PR that resolves this issue:
> [https://github.com/apache/hive/pull/1650]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24410) Query-based compaction hangs because of doAs
[
https://issues.apache.org/jira/browse/HIVE-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage reassigned HIVE-24410:
> Query-based compaction hangs because of doAs
>
>
> Key: HIVE-24410
> URL: https://issues.apache.org/jira/browse/HIVE-24410
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Fix For: 4.0.0
>
>
> QB compaction runs within a doas +and+ hive.server2.enable.doAs is set to
> true (as of HIVE-24089). On a secure cluster with Worker threads running in
> HS2, this results in HMS client not receiving a login context during
> compaction queries, so kerberos prompts for a login via stdin which causes
> the worker thread to hang until it times out:
> {code:java}
> "node-x.com-44_executor" #47 daemon prio=1 os_prio=0 tid=0x01506000
> nid=0x1348 runnable [0x7f1beea95000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x9fa38c90> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x8c7d5010> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at
> com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:862)
> at
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:708)
> at
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getInitialTicket(Krb5Util.java:175)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:341)
> at
> sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:337)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:336)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:146)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> at
>
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515439
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 09:33
Start Date: 23/Nov/20 09:33
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528568530
##
File path: hplsql/src/main/java/org/apache/hive/hplsql/Var.java
##
@@ -258,44 +256,35 @@ public void setValue(Object value) {
this.value = value;
}
}
-
- /**
- * Set the new value from the result set
- */
- public Var setValue(ResultSet rs, ResultSetMetaData rsm, int idx) throws
SQLException {
-int type = rsm.getColumnType(idx);
+
+ public Var setValue(QueryResult queryResult, int idx) {
+int type = queryResult.jdbcType(idx);
if (type == java.sql.Types.CHAR || type == java.sql.Types.VARCHAR) {
- cast(new Var(rs.getString(idx)));
-}
-else if (type == java.sql.Types.INTEGER || type == java.sql.Types.BIGINT ||
-type == java.sql.Types.SMALLINT || type == java.sql.Types.TINYINT) {
- cast(new Var(Long.valueOf(rs.getLong(idx;
-}
-else if (type == java.sql.Types.DECIMAL || type == java.sql.Types.NUMERIC)
{
- cast(new Var(rs.getBigDecimal(idx)));
-}
-else if (type == java.sql.Types.FLOAT || type == java.sql.Types.DOUBLE) {
- cast(new Var(Double.valueOf(rs.getDouble(idx;
+ cast(new Var(queryResult.column(idx, String.class)));
+} else if (type == java.sql.Types.INTEGER || type == java.sql.Types.BIGINT
||
+type == java.sql.Types.SMALLINT || type == java.sql.Types.TINYINT)
{
+ cast(new Var(Long.valueOf(queryResult.column(idx, Long.class;
+} else if (type == java.sql.Types.DECIMAL || type ==
java.sql.Types.NUMERIC) {
+ cast(new Var(queryResult.column(idx, BigDecimal.class)));
+} else if (type == java.sql.Types.FLOAT || type == java.sql.Types.DOUBLE) {
+ cast(new Var(Double.valueOf(queryResult.column(idx, Double.class;
}
return this;
}
-
- /**
- * Set ROW values from the result set
- */
- public Var setValues(ResultSet rs, ResultSetMetaData rsm) throws
SQLException {
+
+ public Var setValues(QueryResult queryResult) {
Row row = (Row)this.value;
-int idx = 1;
+int idx = 0;
Review comment:
I added a javadoc explaining the difference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515439)
Time Spent: 2h (was: 1h 50m)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515438
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 09:32
Start Date: 23/Nov/20 09:32
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528568269
##
File path: beeline/src/java/org/apache/hive/beeline/BeeLine.java
##
@@ -892,8 +893,12 @@ private boolean connectUsingArgs(BeelineParser
beelineParser, CommandLine cl) {
getOpts().setInitFiles(cl.getOptionValues("i"));
getOpts().setScriptFile(cl.getOptionValue("f"));
-
if (url != null) {
+ String hplSqlMode = Utils.parsePropertyFromUrl(url,
Constants.HPLSQL_MODE);
Review comment:
Ok, modified it.
##
File path: hplsql/pom.xml
##
@@ -46,6 +46,11 @@
commons-io
+ org.apache.hive
+ hive-standalone-metastore-common
Review comment:
Right, I removed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515438)
Time Spent: 1h 50m (was: 1h 40m)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline while we'er in hplsql mode. Hive will create a hplsql
> interpreter and store it in the session state. A new HplSqlOperation is
> created to run the script on the interpreter.
> HPLSQL knows how to execute the for loop, but i'll call Hive to run the
> select expression. The HplSqlOperation is notified when the select reads a
> row and accumulates the rows into a RowSet (memory consumption need to be
> considered here) which can be retrieved via thrift from the client side.
>
--
This message was
[jira] [Work logged] (HIVE-24230) Integrate HPL/SQL into HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-24230?focusedWorklogId=515436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515436
]
ASF GitHub Bot logged work on HIVE-24230:
-
Author: ASF GitHub Bot
Created on: 23/Nov/20 09:30
Start Date: 23/Nov/20 09:30
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1633:
URL: https://github.com/apache/hive/pull/1633#discussion_r528567059
##
File path: service/src/java/org/apache/hive/service/cli/operation/Operation.java
##
@@ -102,7 +109,7 @@ protected Operation(HiveSession parentSession,
MetricsConstant.COMPLETED_OPERATION_PREFIX, state);
queryState = new QueryState.Builder()
.withConfOverlay(confOverlay)
- .withGenerateNewQueryId(true)
+ .withGenerateNewQueryId(!embedded)
Review comment:
An hplsql script contains both declarative (for example select) and
imperative statements (for example a for loop). The script is executed by an
HplSqlOperation but this will start SQLOperations when it sees hive ql stuffs.
So this flag indicates that one operation was started by an other operation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 515436)
Time Spent: 1h 40m (was: 1.5h)
> Integrate HPL/SQL into HiveServer2
> --
>
> Key: HIVE-24230
> URL: https://issues.apache.org/jira/browse/HIVE-24230
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, hpl/sql
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> HPL/SQL is a standalone command line program that can store and load scripts
> from text files, or from Hive Metastore (since HIVE-24217). Currently HPL/SQL
> depends on Hive and not the other way around.
> Changing the dependency order between HPL/SQL and HiveServer would open up
> some possibilities which are currently not feasable to implement. For example
> one might want to use a third party SQL tool to run selects on stored
> procedure (or rather function in this case) outputs.
> {code:java}
> SELECT * from myStoredProcedure(1, 2); {code}
> HPL/SQL doesn’t have a JDBC interface and it’s not a daemon so this would not
> work with the current architecture.
> Another important factor is performance. Declarative SQL commands are sent to
> Hive via JDBC by HPL/SQL. The integration would make it possible to drop JDBC
> and use HiveSever’s internal API for compilation and execution.
> The third factor is that existing tools like Beeline or Hue cannot be used
> with HPL/SQL since it has its own, separated CLI.
>
> To make it easier to implement, we keep things separated in the inside at
> first, by introducing a hive session level JDBC parameter.
> {code:java}
> jdbc:hive2://localhost:1/default;hplsqlMode=true {code}
>
> The hplsqlMode indicates that we are in procedural SQL mode where the user
> can create and call stored procedures. HPLSQL allows you to write any kind of
> procedural statement at the top level. This patch doesn't limit this but it
> might be better to eventually restrict what statements are allowed outside of
> stored procedures.
>
> Since HPLSQL and Hive are running in the same process there is no need to use
> the JDBC driver between them. The patch adds an abstraction with 2 different
> implementations, one for executing queries on JDBC (for keeping the existing
> behaviour) and another one for directly calling Hive's compiler. In HPLSQL
> mode the latter is used.
> In the inside a new operation (HplSqlOperation) and operation type
> (PROCEDURAL_SQL) was added which works similar to the SQLOperation but it
> uses the hplsql interpreter to execute arbitrary scripts. This operation
> might spawns new SQLOpertions.
> For example consider the following statement:
> {code:java}
> FOR i in 1..10 LOOP
> SELECT * FROM table
> END LOOP;{code}
> We send this to beeline while we'er in hplsql mode. Hive will create a hplsql
> interpreter and store it in the session state. A new HplSqlOperation is
> created to run the script on the interpreter.
> HPLSQL knows how to execute the for loop, but i'll call Hive to run the
> select expression. The HplSqlOperation is notified when the select reads a
> row and accumulates the rows into a RowSet (memory consumption need to
[jira] [Resolved] (HIVE-24329) Add HMS notification for compaction commit
[ https://issues.apache.org/jira/browse/HIVE-24329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga resolved HIVE-24329. Fix Version/s: 4.0.0 Resolution: Fixed > Add HMS notification for compaction commit > -- > > Key: HIVE-24329 > URL: https://issues.apache.org/jira/browse/HIVE-24329 > Project: Hive > Issue Type: New Feature >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > This could be used by file metadata caches, to invalidate the cache content -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24329) Add HMS notification for compaction commit
[ https://issues.apache.org/jira/browse/HIVE-24329?focusedWorklogId=515402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515402 ] ASF GitHub Bot logged work on HIVE-24329: - Author: ASF GitHub Bot Created on: 23/Nov/20 08:18 Start Date: 23/Nov/20 08:18 Worklog Time Spent: 10m Work Description: deniskuzZ merged pull request #1647: URL: https://github.com/apache/hive/pull/1647 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515402) Time Spent: 3h 10m (was: 3h) > Add HMS notification for compaction commit > -- > > Key: HIVE-24329 > URL: https://issues.apache.org/jira/browse/HIVE-24329 > Project: Hive > Issue Type: New Feature >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > This could be used by file metadata caches, to invalidate the cache content -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24377) Instant deallocation option for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-24377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-24377: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks for the review [~asinkovits] and [~pvary]. > Instant deallocation option for proactive cache eviction > > > Key: HIVE-24377 > URL: https://issues.apache.org/jira/browse/HIVE-24377 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24377) Instant deallocation option for proactive cache eviction
[ https://issues.apache.org/jira/browse/HIVE-24377?focusedWorklogId=515398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-515398 ] ASF GitHub Bot logged work on HIVE-24377: - Author: ASF GitHub Bot Created on: 23/Nov/20 08:01 Start Date: 23/Nov/20 08:01 Worklog Time Spent: 10m Work Description: szlta merged pull request #1687: URL: https://github.com/apache/hive/pull/1687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 515398) Time Spent: 20m (was: 10m) > Instant deallocation option for proactive cache eviction > > > Key: HIVE-24377 > URL: https://issues.apache.org/jira/browse/HIVE-24377 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
