[jira] [Updated] (HIVE-24622) Invalid column ref error in order by when using group by
[ https://issues.apache.org/jira/browse/HIVE-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaolong updated HIVE-24622: Description: create table tbl1(id int,name string,company string) row format delimited fields terminated by ',' stored as textfile; insert into tbl1 values(1,'xxx','xxx'); insert into tbl1 values(2,'yyy','yyy'); set hive.cbo.enable=false; select count(id) FROM tbl1 GROUP BY id,name ORDER BY id desc; !image-2021-01-12-15-41-06-394.png! when cbo is true it works fine. when execute “select count(id),id FROM tbl1 GROUP BY id,name ORDER BY id desc“，it works fine. was: create table tbl1(id int,name string,company string) row format delimited fields terminated by ',' stored as textfile; insert into tbl1 values(1,'xxx','xxx'); insert into tbl1 values(2,'yyy','yyy'); set hive.cbo.enable=false; select count(id) FROM tbl1 GROUP BY id,name ORDER BY id desc; !image-2021-01-12-15-41-06-394.png! > Invalid column ref error in order by when using group by > > > Key: HIVE-24622 > URL: https://issues.apache.org/jira/browse/HIVE-24622 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 4.0.0 >Reporter: zhaolong >Priority: Major > Attachments: image-2021-01-12-15-41-06-394.png > > > create table tbl1(id int,name string,company string) row format delimited > fields terminated by ',' stored as textfile; > insert into tbl1 values(1,'xxx','xxx'); > insert into tbl1 values(2,'yyy','yyy'); > set hive.cbo.enable=false; > select count(id) FROM tbl1 GROUP BY id,name ORDER BY id desc; > !image-2021-01-12-15-41-06-394.png! > > when cbo is true it works fine. > when execute “select count(id),id FROM tbl1 GROUP BY id,name ORDER BY id > desc“，it works fine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24622) Invalid column ref error in order by when using group by
[ https://issues.apache.org/jira/browse/HIVE-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaolong updated HIVE-24622: Attachment: image-2021-01-12-15-41-06-394.png > Invalid column ref error in order by when using group by > > > Key: HIVE-24622 > URL: https://issues.apache.org/jira/browse/HIVE-24622 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 4.0.0 >Reporter: zhaolong >Priority: Major > Attachments: image-2021-01-12-15-41-06-394.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24622) Invalid column ref error in order by when using group by
[ https://issues.apache.org/jira/browse/HIVE-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaolong updated HIVE-24622: Description: create table tbl1(id int,name string,company string) row format delimited fields terminated by ',' stored as textfile; insert into tbl1 values(1,'xxx','xxx'); insert into tbl1 values(2,'yyy','yyy'); set hive.cbo.enable=false; select count(id) FROM tbl1 GROUP BY id,name ORDER BY id desc; !image-2021-01-12-15-41-06-394.png! > Invalid column ref error in order by when using group by > > > Key: HIVE-24622 > URL: https://issues.apache.org/jira/browse/HIVE-24622 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 4.0.0 >Reporter: zhaolong >Priority: Major > Attachments: image-2021-01-12-15-41-06-394.png > > > create table tbl1(id int,name string,company string) row format delimited > fields terminated by ',' stored as textfile; > insert into tbl1 values(1,'xxx','xxx'); > insert into tbl1 values(2,'yyy','yyy'); > set hive.cbo.enable=false; > select count(id) FROM tbl1 GROUP BY id,name ORDER BY id desc; > !image-2021-01-12-15-41-06-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24613) Support Values clause without Insert
[
https://issues.apache.org/jira/browse/HIVE-24613?focusedWorklogId=534676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534676
]
ASF GitHub Bot logged work on HIVE-24613:
-
Author: ASF GitHub Bot
Created on: 12/Jan/21 06:12
Start Date: 12/Jan/21 06:12
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1847:
URL: https://github.com/apache/hive/pull/1847#discussion_r34669
##
File path: ql/src/test/queries/clientpositive/values.q
##
@@ -0,0 +1,10 @@
+set hive.cli.print.header=true;
+
Review comment:
Can we add `explain cbo` for some of these queries?
##
File path: parser/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g
##
@@ -306,7 +316,7 @@ valuesTableConstructor
@init { gParent.pushMsg("values table constructor", state); }
@after { gParent.popMsg(state); }
:
-valueRowConstructor (COMMA! valueRowConstructor)*
+valueRowConstructor (options{greedy=true;}: COMMA! valueRowConstructor)*
Review comment:
Can you leave a comment about the `greedy` option behavior?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534676)
Time Spent: 20m (was: 10m)
> Support Values clause without Insert
>
>
> Key: HIVE-24613
> URL: https://issues.apache.org/jira/browse/HIVE-24613
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Standalone:
> {code}
> VALUES(1,2,3),(4,5,6);
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
> In subquery:
> {code}
> SELECT * FROM (VALUES(1,2,3),(4,5,6)) as FOO;
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24621) TEXT and varchar datatype does not support unicode encoding in MSSQL
[
https://issues.apache.org/jira/browse/HIVE-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Gupta reassigned HIVE-24621:
---
> TEXT and varchar datatype does not support unicode encoding in MSSQL
>
>
> Key: HIVE-24621
> URL: https://issues.apache.org/jira/browse/HIVE-24621
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Critical
>
> Why Unicode is required?
> In following example the Chinese character cannot be properly interpreted.
> {noformat}
> CREATE VIEW `test_view` AS select `test_tbl_char`.`col1` from
> `test_db5`.`test_tbl_char` where `test_tbl_char`.`col1`='你好';
> show create table test_view;
> ++
> | createtab_stmt |
> ++
> | CREATE VIEW `test_view` AS select `test_tbl_char`.`col1` from
> `test_db5`.`test_tbl_char` where `test_tbl_char`.`col1`='??' |
> ++ {noformat}
>
> This issue comes because TBLS is defined as follows:
>
> CREATE TABLE TBLS
> (
> TBL_ID bigint NOT NULL,
> CREATE_TIME int NOT NULL,
> DB_ID bigint NULL,
> LAST_ACCESS_TIME int NOT NULL,
> OWNER nvarchar(767) NULL,
> OWNER_TYPE nvarchar(10) NULL,
> RETENTION int NOT NULL,
> SD_ID bigint NULL,
> TBL_NAME nvarchar(256) NULL,
> TBL_TYPE nvarchar(128) NULL,
> VIEW_EXPANDED_TEXT text NULL,
> VIEW_ORIGINAL_TEXT text NULL,
> IS_REWRITE_ENABLED bit NOT NULL DEFAULT 0,
> WRITE_ID bigint NOT NULL DEFAULT 0
> );
> Text data type does not support unicode encoding irrespective of collation
> varchar data type does not support unicode encoding prior to SQL Server 2019.
> Also UTF8 enabled Collation needs to be defined for use of unicode characters.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24511) Fix typo in SerDeStorageSchemaReader
[ https://issues.apache.org/jira/browse/HIVE-24511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-24511: --- Description: 1, Close the created classloader to release resources. 2, More detail error messages on MetaException when throwing. 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such tables. was: 1, Close the created classloader to release resources. 2, More detail error messages on MetaException when throwing. 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such tables. > Fix typo in SerDeStorageSchemaReader > > > Key: HIVE-24511 > URL: https://issues.apache.org/jira/browse/HIVE-24511 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > 1, Close the created classloader to release resources. > 2, More detail error messages on MetaException when throwing. > 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such > tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24511) Fix typo in SerDeStorageSchemaReader
[ https://issues.apache.org/jira/browse/HIVE-24511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-24511: --- Description: 1, Close the created classloader to release resources. 2, More detail error messages on MetaException when throwing. 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such tables. was: 1, Close the created classloader to release resources. 2, More detail error messages on MetaException when throwing. > Fix typo in SerDeStorageSchemaReader > > > Key: HIVE-24511 > URL: https://issues.apache.org/jira/browse/HIVE-24511 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > 1, Close the created classloader to release resources. > 2, More detail error messages on MetaException when throwing. > 3, Skip JsonSerDe/RegexSerDe creation when get columns/schemas of such tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24620) Reverse not dealing with unicode grapheme clusters correctly
[
https://issues.apache.org/jira/browse/HIVE-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated HIVE-24620:
--
Description:
Reverse() deal with unicode code points instead of grapheme clusters:
{code:java}
hive> select 'abc\u0303def', reverse('abc\u0303def');
abc̃def fed̃cba
hive> select 'abc̃def', reverse('abc̃def');
abc̃def fed̃cba{code}
Note that "~" should on top of "c" in the result.
was:
Reverse() deal with unicode code points instead of grapheme clusters:
{code:java}
hive> select 'abc\u0303def', reverse('abc\u0303def');
OK
abc̃def fed̃cba{code}
Note that "~" should on top of "c" in the result.
> Reverse not dealing with unicode grapheme clusters correctly
>
>
> Key: HIVE-24620
> URL: https://issues.apache.org/jira/browse/HIVE-24620
> Project: Hive
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
>
> Reverse() deal with unicode code points instead of grapheme clusters:
> {code:java}
> hive> select 'abc\u0303def', reverse('abc\u0303def');
> abc̃def fed̃cba
> hive> select 'abc̃def', reverse('abc̃def');
> abc̃def fed̃cba{code}
> Note that "~" should on top of "c" in the result.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[
https://issues.apache.org/jira/browse/HIVE-24543?focusedWorklogId=534652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534652
]
ASF GitHub Bot logged work on HIVE-24543:
-
Author: ASF GitHub Bot
Created on: 12/Jan/21 03:35
Start Date: 12/Jan/21 03:35
Worklog Time Spent: 10m
Work Description: vihangk1 commented on a change in pull request #1791:
URL: https://github.com/apache/hive/pull/1791#discussion_r555489284
##
File path:
service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java
##
@@ -76,6 +77,9 @@ public static PasswdAuthenticationProvider
getAuthenticationProvider(AuthMethods
return new CustomAuthenticationProviderImpl((conf == null) ?
AuthMethods.CUSTOM.getConf() : conf);
} else if (authMethod == AuthMethods.NONE) {
return new AnonymousAuthenticationProviderImpl();
+} else if (authMethod == AuthMethods.SAML) {
+ //TODO right thing to do?
+ return new AnonymousAuthenticationProviderImpl();
Review comment:
I don't think it is necessary. I removed this from the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534652)
Time Spent: 4h 50m (was: 4h 40m)
> Support SAML 2.0 as an authentication mechanism
> ---
>
> Key: HIVE-24543
> URL: https://issues.apache.org/jira/browse/HIVE-24543
> Project: Hive
> Issue Type: New Feature
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> With cloud based deployments, having a SAML 2.0 based authentication support
> in HS2 will be greatly useful in case of federated or external identity
> providers like Okta, PingIdentity or Azure AD.
> This authentication mechanism can initially be only supported on http
> transport mode in HiveServer2 since the SAML 2.0 protocol is primarily
> designed for web clients.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[ https://issues.apache.org/jira/browse/HIVE-24543?focusedWorklogId=534650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534650 ] ASF GitHub Bot logged work on HIVE-24543: - Author: ASF GitHub Bot Created on: 12/Jan/21 03:35 Start Date: 12/Jan/21 03:35 Worklog Time Spent: 10m Work Description: vihangk1 commented on a change in pull request #1791: URL: https://github.com/apache/hive/pull/1791#discussion_r555489128 ## File path: service/pom.xml ## @@ -179,6 +179,37 @@ + + org.pac4j + pac4j-saml-opensamlv3 + 4.0.3 Review comment: Created HIVE-24619 to track this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534650) Time Spent: 4h 40m (was: 4.5h) > Support SAML 2.0 as an authentication mechanism > --- > > Key: HIVE-24543 > URL: https://issues.apache.org/jira/browse/HIVE-24543 > Project: Hive > Issue Type: New Feature >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > With cloud based deployments, having a SAML 2.0 based authentication support > in HS2 will be greatly useful in case of federated or external identity > providers like Okta, PingIdentity or Azure AD. > This authentication mechanism can initially be only supported on http > transport mode in HiveServer2 since the SAML 2.0 protocol is primarily > designed for web clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[
https://issues.apache.org/jira/browse/HIVE-24543?focusedWorklogId=534649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534649
]
ASF GitHub Bot logged work on HIVE-24543:
-
Author: ASF GitHub Bot
Created on: 12/Jan/21 03:34
Start Date: 12/Jan/21 03:34
Worklog Time Spent: 10m
Work Description: vihangk1 commented on a change in pull request #1791:
URL: https://github.com/apache/hive/pull/1791#discussion_r555489042
##
File path: jdbc/src/java/org/apache/hive/jdbc/saml/HiveJdbcBrowserClient.java
##
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.jdbc.saml;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import java.awt.Desktop;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketTimeoutException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.hive.jdbc.Utils.JdbcConnectionParams;
+import org.apache.hive.service.auth.saml.HiveSamlUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class is used to execute a browser based SSO workflow with the
authentication mode
+ * is browser.
+ */
+public class HiveJdbcBrowserClient implements IJdbcBrowserClient {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveJdbcBrowserClient.class);
+ // error message when the socket times out.
+ @VisibleForTesting
+ public static final String TIMEOUT_ERROR_MSG = "Timed out while waiting for
server response";
+ private final ServerSocket serverSocket;
+ private HiveJdbcBrowserServerResponse serverResponse;
+ protected JdbcBrowserClientContext clientContext;
+ // By default we wait for 2 min unless overridden by a JDBC connection param
+ // browserResponseTimeout
+ private static final int DEFAULT_SOCKET_TIMEOUT_SECS = 120;
+ private final ExecutorService serverResponseThread =
Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setNameFormat("Hive-Jdbc-Browser-Client-%d")
+ .setDaemon(true).build());
+
+ HiveJdbcBrowserClient(JdbcConnectionParams connectionParams)
+ throws HiveJdbcBrowserException {
+serverSocket = getServerSocket(connectionParams.getSessionVars());
+ }
+
+ private ServerSocket getServerSocket(Map sessionConf)
+ throws HiveJdbcBrowserException {
+final ServerSocket serverSocket;
+int port = Integer.parseInt(sessionConf
+.getOrDefault(JdbcConnectionParams.AUTH_BROWSER_RESPONSE_PORT, "0"));
+int timeout = Integer.parseInt(
+
sessionConf.getOrDefault(JdbcConnectionParams.AUTH_BROWSER_RESPONSE_TIMEOUT_SECS,
+String.valueOf(DEFAULT_SOCKET_TIMEOUT_SECS)));
+try {
+ serverSocket = new ServerSocket(port, 0,
+ InetAddress.getByName(HiveSamlUtils.LOOP_BACK_INTERFACE));
+ LOG.debug("Browser response timeout is set to {} seconds", timeout);
+ serverSocket.setSoTimeout(timeout * 1000);
+} catch (IOException e) {
+ throw new HiveJdbcBrowserException("Unable to bind to the localhost");
+}
+return serverSocket;
+ }
+
+ public Integer getPort() {
+return serverSocket.getLocalPort();
+ }
+
+ @Override
+ public void close() throws IOException {
+if (serverSocket != null) {
+ serverSocket.close();
+}
+ }
+
+ public void init(JdbcBrowserClientContext clientContext) {
+// everytime we
[jira] [Assigned] (HIVE-24619) Exclude unnecessary dependencies from pac4j
[ https://issues.apache.org/jira/browse/HIVE-24619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-24619: -- > Exclude unnecessary dependencies from pac4j > --- > > Key: HIVE-24619 > URL: https://issues.apache.org/jira/browse/HIVE-24619 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > HIVE-24543 introduces pac4j dependency which pulls in multiple other > dependencies. It would be great to exclude as many dependencies as possible. > This JIRA is used to track this effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24310) Allow specified number of deserialize errors to be ignored
[ https://issues.apache.org/jira/browse/HIVE-24310?focusedWorklogId=534639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534639 ] ASF GitHub Bot logged work on HIVE-24310: - Author: ASF GitHub Bot Created on: 12/Jan/21 03:10 Start Date: 12/Jan/21 03:10 Worklog Time Spent: 10m Work Description: dengzhhu653 removed a comment on pull request #1607: URL: https://github.com/apache/hive/pull/1607#issuecomment-724406733 @sershe-apache @prasanthj any thoughts or comments on this? thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534639) Time Spent: 1h (was: 50m) > Allow specified number of deserialize errors to be ignored > -- > > Key: HIVE-24310 > URL: https://issues.apache.org/jira/browse/HIVE-24310 > Project: Hive > Issue Type: Improvement > Components: Operators >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Sometimes we see some corrupted records in user's raw data, like one > corrupted in a file which contains over thousands of records, user has to > either give up all records or replay the whole data in order to run > successfully on hive, we should provide a way to ignore such corrupted > records. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24588) Run tests using specific log4j2 configuration conveniently
[
https://issues.apache.org/jira/browse/HIVE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez resolved HIVE-24588.
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master, thanks [~zabetak]!
> Run tests using specific log4j2 configuration conveniently
> --
>
> Key: HIVE-24588
> URL: https://issues.apache.org/jira/browse/HIVE-24588
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In order to reproduce a problem (e.g., HIVE-24569) or validate that a log4j2
> configuration is working as expected it is necessary to run a test and
> explicitly specify which configuration should be used. Moreover, after the
> end of the test in question it is desirable to restore the old logging
> configuration that was used before launching the test to avoid affecting the
> overall logging output.
> The goal of this issue is to introduce a convenient & declarative way of
> running tests with log4j2 configurations based on Jupiter extensions and
> annotations. The test could like below:
> {code:java}
> @Test
> @Log4jConfig("test-log4j2.properties")
> void testUseExplicitConfig() {
> // Do something and assert
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24588) Run tests using specific log4j2 configuration conveniently
[
https://issues.apache.org/jira/browse/HIVE-24588?focusedWorklogId=534637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534637
]
ASF GitHub Bot logged work on HIVE-24588:
-
Author: ASF GitHub Bot
Created on: 12/Jan/21 03:04
Start Date: 12/Jan/21 03:04
Worklog Time Spent: 10m
Work Description: jcamachor merged pull request #1842:
URL: https://github.com/apache/hive/pull/1842
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534637)
Time Spent: 0.5h (was: 20m)
> Run tests using specific log4j2 configuration conveniently
> --
>
> Key: HIVE-24588
> URL: https://issues.apache.org/jira/browse/HIVE-24588
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In order to reproduce a problem (e.g., HIVE-24569) or validate that a log4j2
> configuration is working as expected it is necessary to run a test and
> explicitly specify which configuration should be used. Moreover, after the
> end of the test in question it is desirable to restore the old logging
> configuration that was used before launching the test to avoid affecting the
> overall logging output.
> The goal of this issue is to introduce a convenient & declarative way of
> running tests with log4j2 configurations based on Jupiter extensions and
> annotations. The test could like below:
> {code:java}
> @Test
> @Log4jConfig("test-log4j2.properties")
> void testUseExplicitConfig() {
> // Do something and assert
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24363) Current order of transactional event listeners is prone to deadlock in backend DB connections
[
https://issues.apache.org/jira/browse/HIVE-24363?focusedWorklogId=534619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534619
]
ASF GitHub Bot logged work on HIVE-24363:
-
Author: ASF GitHub Bot
Created on: 12/Jan/21 01:18
Start Date: 12/Jan/21 01:18
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #1657:
URL: https://github.com/apache/hive/pull/1657#issuecomment-758329009
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534619)
Time Spent: 2h 10m (was: 2h)
> Current order of transactional event listeners is prone to deadlock in
> backend DB connections
> -
>
> Key: HIVE-24363
> URL: https://issues.apache.org/jira/browse/HIVE-24363
> Project: Hive
> Issue Type: Bug
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-24363.01.patch, HIVE-24363.02.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently the AcidEventListener is added to the end of list of transactional
> event listeners. When DbNotificationListener is configured as
> 'hive.metastore.transactional.event.listeners'. The final list will be formed
> as :
> {"DbNotificationListener" , "AcidEventListener"}
> This will result in backend DB lock acquisition in this order:
> {code:java}
> lock(a) {
> // perform some op on a
> lock(b) {
> // perform some op on b
> }
> }
> {code}
> On the other hand, there are some HMS API say for example commit_txn(), which
> calls the TxnHandler method directly, followed by DbNotificationListener
> processing. Which will result in the lock acquisition in reverse order:
> {code:java}
> lock(b) {
> // perform some op on b
> lock(a) {
> // perform some op on a
> }
> }
> {code}
> Note: 'a' and 'b' above are backend DB lock and not jvm locks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24371) Ranger Replication fallback to updateIfExists
[ https://issues.apache.org/jira/browse/HIVE-24371?focusedWorklogId=534618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534618 ] ASF GitHub Bot logged work on HIVE-24371: - Author: ASF GitHub Bot Created on: 12/Jan/21 01:18 Start Date: 12/Jan/21 01:18 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1663: URL: https://github.com/apache/hive/pull/1663#issuecomment-758329004 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534618) Time Spent: 20m (was: 10m) > Ranger Replication fallback to updateIfExists > - > > Key: HIVE-24371 > URL: https://issues.apache.org/jira/browse/HIVE-24371 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24371.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Ranger Replication fallback to updateIfExists > Add dummy resource as workaround while creating the deny policy to avoid it > from overriding the actual policy -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive JDBC should use java.time.LocalDateTime instead of timestamp
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Summary: Hive JDBC should use java.time.LocalDateTime instead of timestamp (was: Hive JDBC should use java.time.LocalDateTime ) > Hive JDBC should use java.time.LocalDateTime instead of timestamp > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive JDBC should use java.time.LocalDateTime
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Summary: Hive JDBC should use java.time.LocalDateTime (was: Hive JDBC should use timestamp in UTC) > Hive JDBC should use java.time.LocalDateTime > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive JDBC should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Description: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive JDBC handler should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . was: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive JDBC handler should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. Also, it might be helpful if the user can set the user timezone in Beeline session, lets say "user.timezone”, which should be honored while showing the beeline output of a particular timestamp field. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . > Hive JDBC should use timestamp in UTC > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24618) Hive JDBC should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24618 started by Kishen Das. - > Hive JDBC should use timestamp in UTC > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > Also, it might be helpful if the user can set the user timezone in Beeline > session, lets say "user.timezone”, which should be honored while showing the > beeline output of a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24618) Hive JDBC should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das reassigned HIVE-24618: - Assignee: Kishen Das > Hive JDBC should use timestamp in UTC > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > Also, it might be helpful if the user can set the user timezone in Beeline > session, lets say "user.timezone”, which should be honored while showing the > beeline output of a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[ https://issues.apache.org/jira/browse/HIVE-24543?focusedWorklogId=534560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534560 ] ASF GitHub Bot logged work on HIVE-24543: - Author: ASF GitHub Bot Created on: 11/Jan/21 21:49 Start Date: 11/Jan/21 21:49 Worklog Time Spent: 10m Work Description: vihangk1 commented on a change in pull request #1791: URL: https://github.com/apache/hive/pull/1791#discussion_r555362138 ## File path: service/pom.xml ## @@ -179,6 +179,37 @@ + + org.pac4j + pac4j-saml-opensamlv3 + 4.0.3 Review comment: I found on dependency which can potentially cause problems (commons-collections). I excluded it here. However, finding all the dependencies which are unnecessary may be a bit more complex. I will create a separate JIRA for that if that is okay with you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534560) Time Spent: 4h 20m (was: 4h 10m) > Support SAML 2.0 as an authentication mechanism > --- > > Key: HIVE-24543 > URL: https://issues.apache.org/jira/browse/HIVE-24543 > Project: Hive > Issue Type: New Feature >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > With cloud based deployments, having a SAML 2.0 based authentication support > in HS2 will be greatly useful in case of federated or external identity > providers like Okta, PingIdentity or Azure AD. > This authentication mechanism can initially be only supported on http > transport mode in HiveServer2 since the SAML 2.0 protocol is primarily > designed for web clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[
https://issues.apache.org/jira/browse/HIVE-24543?focusedWorklogId=534559=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534559
]
ASF GitHub Bot logged work on HIVE-24543:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 21:48
Start Date: 11/Jan/21 21:48
Worklog Time Spent: 10m
Work Description: vihangk1 commented on a change in pull request #1791:
URL: https://github.com/apache/hive/pull/1791#discussion_r555361475
##
File path: jdbc/src/java/org/apache/hive/jdbc/saml/HiveJdbcBrowserClient.java
##
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.jdbc.saml;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import java.awt.Desktop;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketTimeoutException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.hive.jdbc.Utils.JdbcConnectionParams;
+import org.apache.hive.service.auth.saml.HiveSamlUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class is used to execute a browser based SSO workflow with the
authentication mode
+ * is browser.
+ */
+public class HiveJdbcBrowserClient implements IJdbcBrowserClient {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HiveJdbcBrowserClient.class);
+ // error message when the socket times out.
+ @VisibleForTesting
+ public static final String TIMEOUT_ERROR_MSG = "Timed out while waiting for
server response";
+ private final ServerSocket serverSocket;
+ private HiveJdbcBrowserServerResponse serverResponse;
+ protected JdbcBrowserClientContext clientContext;
+ // By default we wait for 2 min unless overridden by a JDBC connection param
+ // browserResponseTimeout
+ private static final int DEFAULT_SOCKET_TIMEOUT_SECS = 120;
+ private final ExecutorService serverResponseThread =
Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setNameFormat("Hive-Jdbc-Browser-Client-%d")
+ .setDaemon(true).build());
+
+ HiveJdbcBrowserClient(JdbcConnectionParams connectionParams)
+ throws HiveJdbcBrowserException {
+serverSocket = getServerSocket(connectionParams.getSessionVars());
+ }
+
+ private ServerSocket getServerSocket(Map sessionConf)
+ throws HiveJdbcBrowserException {
+final ServerSocket serverSocket;
+int port = Integer.parseInt(sessionConf
+.getOrDefault(JdbcConnectionParams.AUTH_BROWSER_RESPONSE_PORT, "0"));
+int timeout = Integer.parseInt(
+
sessionConf.getOrDefault(JdbcConnectionParams.AUTH_BROWSER_RESPONSE_TIMEOUT_SECS,
+String.valueOf(DEFAULT_SOCKET_TIMEOUT_SECS)));
+try {
+ serverSocket = new ServerSocket(port, 0,
+ InetAddress.getByName(HiveSamlUtils.LOOP_BACK_INTERFACE));
+ LOG.debug("Browser response timeout is set to {} seconds", timeout);
+ serverSocket.setSoTimeout(timeout * 1000);
+} catch (IOException e) {
+ throw new HiveJdbcBrowserException("Unable to bind to the localhost");
+}
+return serverSocket;
+ }
+
+ public Integer getPort() {
+return serverSocket.getLocalPort();
Review comment:
serverSocket is a final field and it will always be not-null. If there
is an exception during getServerSocket, the HiveJdbcBrowserClient's constructor
will error out.
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=534533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534533
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 20:57
Start Date: 11/Jan/21 20:57
Worklog Time Spent: 10m
Work Description: belugabehr commented on pull request #1833:
URL: https://github.com/apache/hive/pull/1833#issuecomment-758218546
@nrg4878 beeline uses Log4J as the backend logger. Log4J is a pretty
robust, heavy, logging implementation that plugs into SLF4J. I have proposed
here that we swap out Log4J and use the 'Simple' logger that is produced by
SLF4J team. This will make configuring beeline debug logging much easier.
Sure, we won't be able to forward beeline log statements to JMS and Kafka
anymore, but who would want to do that anyway? :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534533)
Time Spent: 1h 20m (was: 1h 10m)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=534529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534529
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 20:53
Start Date: 11/Jan/21 20:53
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1833:
URL: https://github.com/apache/hive/pull/1833#discussion_r555330360
##
File path: beeline/pom.xml
##
@@ -105,19 +117,40 @@
super-csv
${super-csv.version}
+
+ org.slf4j
+ slf4j-api
Review comment:
It is being used, but the dependencies have been transitive for a long
time (d'oh).
No scope required. Technically don't need it for compile, but it's no harm.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534529)
Time Spent: 1h 10m (was: 1h)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=534528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534528
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 20:51
Start Date: 11/Jan/21 20:51
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1833:
URL: https://github.com/apache/hive/pull/1833#discussion_r555329597
##
File path: bin/ext/cli.sh
##
@@ -28,7 +28,7 @@ updateCli() {
CLASS=org.apache.hadoop.hive.cli.CliDriver
JAR=hive-cli-*.jar
else
-export HADOOP_CLIENT_OPTS=" -Dproc_beeline $HADOOP_CLIENT_OPTS
-Dlog4j.configurationFile=beeline-log4j2.properties"
+export HADOOP_CLIENT_OPTS="-Dproc_beeline
-Dorg.slf4j.simpleLogger.defaultLogLevel=warn $HADOOP_CLIENT_OPTS"
Review comment:
Not a dumb question. I think I included it in the original JIRA, but
here it is:
```
export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
beeline ...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534528)
Time Spent: 1h (was: 50m)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive JDBC should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Description: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive JDBC handler should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. Also, it might be helpful if the user can set the user timezone in Beeline session, lets say "user.timezone”, which should be honored while showing the beeline output of a particular timestamp field. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . was: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive Beeline should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. Also, it might be helpful if the user can set the user timezone in Beeline session, lets say "user.timezone”, which should be honored while showing the beeline output of a particular timestamp field. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . > Hive JDBC should use timestamp in UTC > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive JDBC handler should follow suit and use > LocalDateTime ( A date-time without a time-zone ), when doing the conversion. > Also, it might be helpful if the user can set the user timezone in Beeline > session, lets say "user.timezone”, which should be honored while showing the > beeline output of a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive Beeline should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Component/s: (was: Beeline) JDBC > Hive Beeline should use timestamp in UTC > > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive Beeline should follow suit and use LocalDateTime ( > A date-time without a time-zone ), when doing the conversion. Also, it might > be helpful if the user can set the user timezone in Beeline session, lets say > "user.timezone”, which should be honored while showing the beeline output of > a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive JDBC should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Summary: Hive JDBC should use timestamp in UTC (was: Hive Beeline should use timestamp in UTC) > Hive JDBC should use timestamp in UTC > - > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive Beeline should follow suit and use LocalDateTime ( > A date-time without a time-zone ), when doing the conversion. Also, it might > be helpful if the user can set the user timezone in Beeline session, lets say > "user.timezone”, which should be honored while showing the beeline output of > a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24618) Hive Beeline should use timestamp in UTC
[ https://issues.apache.org/jira/browse/HIVE-24618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-24618: -- Description: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive Beeline should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. Also, it might be helpful if the user can set the user timezone in Beeline session, lets say "user.timezone”, which should be honored while showing the beeline output of a particular timestamp field. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . was: Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( https://issues.apache.org/jira/browse/HIVE-12192, in order to be consistent w.r.t timestamp field irrespective of the current time zone. For example: for time zones like America/Los_Angeles that alternate between PST and PDT, time can be shown based on effective current time zone, which is set in the current SQL session. Hive Beeline should follow suit and use LocalDateTime ( A date-time without a time-zone ), when doing the conversion. Also, it might be helpful if the user can set the user timezone in Beeline session, lets say "user.timezone”, which should be honored while showing the beeline output of a particular timestamp field. This issue is also more pronounced because of this bug in Java -> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . > Hive Beeline should use timestamp in UTC > > > Key: HIVE-24618 > URL: https://issues.apache.org/jira/browse/HIVE-24618 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Kishen Das >Priority: Major > > Hive moved away from java.sql.Timestamp to java.time.LocalDateTime ( > https://issues.apache.org/jira/browse/HIVE-12192 ), in order to be consistent > w.r.t timestamp field irrespective of the current time zone. For example: for > time zones like America/Los_Angeles that alternate between PST and PDT, time > can be shown based on effective current time zone, which is set in the > current SQL session. Hive Beeline should follow suit and use LocalDateTime ( > A date-time without a time-zone ), when doing the conversion. Also, it might > be helpful if the user can set the user timezone in Beeline session, lets say > "user.timezone”, which should be honored while showing the beeline output of > a particular timestamp field. > > This issue is also more pronounced because of this bug in Java -> > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8258586] . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=534460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534460
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 19:13
Start Date: 11/Jan/21 19:13
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #1833:
URL: https://github.com/apache/hive/pull/1833#issuecomment-758163429
@belugabehr Could you add a bit more details on the rationale behind these
changes in the jira, so I/we can understand what user facing impact, this fix
has. I am not sure I understand what is broken or needs to be enhanced. It is
my impression beeline already uses slf4j.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534460)
Time Spent: 50m (was: 40m)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=534454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534454
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 19:10
Start Date: 11/Jan/21 19:10
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #1833:
URL: https://github.com/apache/hive/pull/1833#discussion_r555273362
##
File path: bin/ext/cli.sh
##
@@ -28,7 +28,7 @@ updateCli() {
CLASS=org.apache.hadoop.hive.cli.CliDriver
JAR=hive-cli-*.jar
else
-export HADOOP_CLIENT_OPTS=" -Dproc_beeline $HADOOP_CLIENT_OPTS
-Dlog4j.configurationFile=beeline-log4j2.properties"
+export HADOOP_CLIENT_OPTS="-Dproc_beeline
-Dorg.slf4j.simpleLogger.defaultLogLevel=warn $HADOOP_CLIENT_OPTS"
Review comment:
dumb question? how would users enable debug logging in beeline with this
approach ?
##
File path: beeline/pom.xml
##
@@ -105,19 +117,40 @@
super-csv
${super-csv.version}
+
+ org.slf4j
+ slf4j-api
Review comment:
arent we already using slf4j libraries in beeline today? How has this
been compiling without explicit dependencies? transitive?
also we should specify a scope for these dependencies.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534454)
Time Spent: 40m (was: 0.5h)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24617) Review beeline Driver Scanning Code
[
https://issues.apache.org/jira/browse/HIVE-24617?focusedWorklogId=534433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534433
]
ASF GitHub Bot logged work on HIVE-24617:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 18:51
Start Date: 11/Jan/21 18:51
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1853:
URL: https://github.com/apache/hive/pull/1853#discussion_r555265817
##
File path: beeline/src/java/org/apache/hive/beeline/BeeLine.java
##
@@ -2298,23 +2293,17 @@ public void addLocalDriverClazz(String driverClazz) {
supportedLocalDriver.add(driverClazz);
}
- Driver[] scanDrivers(String line) throws IOException {
-return scanDrivers(false);
- }
-
- Driver[] scanDrivers(boolean knownOnly) throws IOException {
Review comment:
This `knownOnly` flag is never used, so remove it. By extension, the
`scanDrivers(String line)` is also defunct.
Calling code turns the array into a Java Collection, so might as well just
change the return type here to return a Collection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534433)
Time Spent: 0.5h (was: 20m)
> Review beeline Driver Scanning Code
> ---
>
> Key: HIVE-24617
> URL: https://issues.apache.org/jira/browse/HIVE-24617
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There seems to be quite a few code artifacts laying around the area of the
> code that are no longer valid. Remove and improve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24617) Review beeline Driver Scanning Code
[
https://issues.apache.org/jira/browse/HIVE-24617?focusedWorklogId=534430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534430
]
ASF GitHub Bot logged work on HIVE-24617:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 18:50
Start Date: 11/Jan/21 18:50
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1853:
URL: https://github.com/apache/hive/pull/1853#discussion_r555265817
##
File path: beeline/src/java/org/apache/hive/beeline/BeeLine.java
##
@@ -2298,23 +2293,17 @@ public void addLocalDriverClazz(String driverClazz) {
supportedLocalDriver.add(driverClazz);
}
- Driver[] scanDrivers(String line) throws IOException {
-return scanDrivers(false);
- }
-
- Driver[] scanDrivers(boolean knownOnly) throws IOException {
Review comment:
This `knownOnly` flag is never used, so remove it. By extension, the
`scanDrivers(String line)` is also defunct.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534430)
Time Spent: 20m (was: 10m)
> Review beeline Driver Scanning Code
> ---
>
> Key: HIVE-24617
> URL: https://issues.apache.org/jira/browse/HIVE-24617
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There seems to be quite a few code artifacts laying around the area of the
> code that are no longer valid. Remove and improve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24617) Review beeline Driver Scanning Code
[ https://issues.apache.org/jira/browse/HIVE-24617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24617: -- Labels: pull-request-available (was: ) > Review beeline Driver Scanning Code > --- > > Key: HIVE-24617 > URL: https://issues.apache.org/jira/browse/HIVE-24617 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There seems to be quite a few code artifacts laying around the area of the > code that are no longer valid. Remove and improve. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24617) Review beeline Driver Scanning Code
[ https://issues.apache.org/jira/browse/HIVE-24617?focusedWorklogId=534426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534426 ] ASF GitHub Bot logged work on HIVE-24617: - Author: ASF GitHub Bot Created on: 11/Jan/21 18:49 Start Date: 11/Jan/21 18:49 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1853: URL: https://github.com/apache/hive/pull/1853 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534426) Remaining Estimate: 0h Time Spent: 10m > Review beeline Driver Scanning Code > --- > > Key: HIVE-24617 > URL: https://issues.apache.org/jira/browse/HIVE-24617 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > There seems to be quite a few code artifacts laying around the area of the > code that are no longer valid. Remove and improve. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24617) Review beeline Driver Scanning Code
[ https://issues.apache.org/jira/browse/HIVE-24617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-24617: - > Review beeline Driver Scanning Code > --- > > Key: HIVE-24617 > URL: https://issues.apache.org/jira/browse/HIVE-24617 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > > There seems to be quite a few code artifacts laying around the area of the > code that are no longer valid. Remove and improve. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24616) Add Logging to Track Query Status
[ https://issues.apache.org/jira/browse/HIVE-24616?focusedWorklogId=534385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534385 ] ASF GitHub Bot logged work on HIVE-24616: - Author: ASF GitHub Bot Created on: 11/Jan/21 18:07 Start Date: 11/Jan/21 18:07 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1851: URL: https://github.com/apache/hive/pull/1851 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534385) Time Spent: 0.5h (was: 20m) > Add Logging to Track Query Status > - > > Key: HIVE-24616 > URL: https://issues.apache.org/jira/browse/HIVE-24616 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Add additional logging to JDBC to all for tracking the status of a query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24616) Add Logging to Track Query Status
[ https://issues.apache.org/jira/browse/HIVE-24616?focusedWorklogId=534384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534384 ] ASF GitHub Bot logged work on HIVE-24616: - Author: ASF GitHub Bot Created on: 11/Jan/21 18:06 Start Date: 11/Jan/21 18:06 Worklog Time Spent: 10m Work Description: belugabehr closed pull request #1851: URL: https://github.com/apache/hive/pull/1851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534384) Time Spent: 20m (was: 10m) > Add Logging to Track Query Status > - > > Key: HIVE-24616 > URL: https://issues.apache.org/jira/browse/HIVE-24616 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Add additional logging to JDBC to all for tracking the status of a query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?focusedWorklogId=534370=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534370 ] ASF GitHub Bot logged work on HIVE-24348: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:46 Start Date: 11/Jan/21 17:46 Worklog Time Spent: 10m Work Description: nrg4878 commented on a change in pull request #1852: URL: https://github.com/apache/hive/pull/1852#discussion_r555229243 ## File path: beeline/pom.xml ## @@ -197,6 +197,25 @@ + +org.apache.maven.plugins +maven-assembly-plugin +3.1.1 Review comment: Thanks David. These changes are not final. I am having a new team member look at these changes. There is more work to be done in terms of building a beeline standalone distributable (.zip file). I am just transitioning these code changes for him to take a look. But your comment is noted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534370) Time Spent: 50m (was: 40m) > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?focusedWorklogId=534365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534365 ] ASF GitHub Bot logged work on HIVE-24348: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:36 Start Date: 11/Jan/21 17:36 Worklog Time Spent: 10m Work Description: belugabehr commented on a change in pull request #1852: URL: https://github.com/apache/hive/pull/1852#discussion_r555222070 ## File path: beeline/pom.xml ## @@ -197,6 +197,25 @@ + +org.apache.maven.plugins +maven-assembly-plugin +3.1.1 Review comment: I don't think you need to include an explicit version number here. It should just inherit from parent POM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534365) Time Spent: 40m (was: 0.5h) > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?focusedWorklogId=534364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534364 ] ASF GitHub Bot logged work on HIVE-24348: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:35 Start Date: 11/Jan/21 17:35 Worklog Time Spent: 10m Work Description: belugabehr commented on a change in pull request #1852: URL: https://github.com/apache/hive/pull/1852#discussion_r555222070 ## File path: beeline/pom.xml ## @@ -197,6 +197,25 @@ + +org.apache.maven.plugins +maven-assembly-plugin +3.1.1 Review comment: I don't think you need to include an explicit version number here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534364) Time Spent: 0.5h (was: 20m) > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?focusedWorklogId=534362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534362 ] ASF GitHub Bot logged work on HIVE-24348: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:33 Start Date: 11/Jan/21 17:33 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #1852: URL: https://github.com/apache/hive/pull/1852#issuecomment-758106837 @nrg4878 Can you also take a look at my PR #1833 since you seem to be looking at beeline improvements? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534362) Time Spent: 20m (was: 10m) > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24348: -- Labels: pull-request-available (was: ) > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24348) Beeline: Isolating dependencies and execution with java
[ https://issues.apache.org/jira/browse/HIVE-24348?focusedWorklogId=534359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534359 ] ASF GitHub Bot logged work on HIVE-24348: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:29 Start Date: 11/Jan/21 17:29 Worklog Time Spent: 10m Work Description: nrg4878 opened a new pull request #1852: URL: https://github.com/apache/hive/pull/1852 1) Changes to use java to execute beeline if HADOOP_HOME is not set 2) POM changes to build an uber beeline-with-dependencies jar 3) Eliminates some of the not required jars in classpath for beeline. ### What changes were proposed in this pull request? 1) Changes to use java to execute beeline if HADOOP_HOME is not set 2) POM changes to build an uber beeline-with-dependencies jar 3) Eliminates some of the not required jars in classpath for beeline. ### Why are the changes needed? Currently beeline has a bunch of dependencies on other jars, some required and some unnecessary. beeline script also uses HADOOP_HOME to execute beeline class. This adds some other jars to the classpath that makes it hard to run beeline on an edge node. This fix is a first pass at building a standalone jar with all needed dependencies and also make it work using "java" when HADOOP_HOME is not set. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Manually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534359) Remaining Estimate: 0h Time Spent: 10m > Beeline: Isolating dependencies and execution with java > --- > > Key: HIVE-24348 > URL: https://issues.apache.org/jira/browse/HIVE-24348 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently, beeline code, binaries and executables are somewhat tightly > coupled with the hive product. To be able to execute beeline from a node with > just JRE installed and some jars in classpath is impossible. > * beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to > use "hadoop" executable to run beeline. > * Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be > enough but sadly they arent. The latter jar adds more problems than it solves > because all the classfiles are shaded some dependencies cannot be resolved. > * Beeline has many other dependencies like hive-exec, hive-common. > hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. > While it may not be possible to eliminate some of these, we should atleast > have a self-contains jar that contains all these to be able to make it work. > * the underlying script used to run beeline should use JAVA as an alternate > means to execute if HADOOP_HOME is not set -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24616) Add Logging to Track Query Status
[ https://issues.apache.org/jira/browse/HIVE-24616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24616: -- Labels: pull-request-available (was: ) > Add Logging to Track Query Status > - > > Key: HIVE-24616 > URL: https://issues.apache.org/jira/browse/HIVE-24616 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add additional logging to JDBC to all for tracking the status of a query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24616) Add Logging to Track Query Status
[ https://issues.apache.org/jira/browse/HIVE-24616?focusedWorklogId=534358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534358 ] ASF GitHub Bot logged work on HIVE-24616: - Author: ASF GitHub Bot Created on: 11/Jan/21 17:26 Start Date: 11/Jan/21 17:26 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1851: URL: https://github.com/apache/hive/pull/1851 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534358) Remaining Estimate: 0h Time Spent: 10m > Add Logging to Track Query Status > - > > Key: HIVE-24616 > URL: https://issues.apache.org/jira/browse/HIVE-24616 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Add additional logging to JDBC to all for tracking the status of a query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24589) Drop catalog failing with deadlock error for Oracle backend dbms.
[
https://issues.apache.org/jira/browse/HIVE-24589?focusedWorklogId=534356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534356
]
ASF GitHub Bot logged work on HIVE-24589:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 17:18
Start Date: 11/Jan/21 17:18
Worklog Time Spent: 10m
Work Description: maheshk114 opened a new pull request #1850:
URL: https://github.com/apache/hive/pull/1850
…
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534356)
Remaining Estimate: 0h
Time Spent: 10m
> Drop catalog failing with deadlock error for Oracle backend dbms.
> -
>
> Key: HIVE-24589
> URL: https://issues.apache.org/jira/browse/HIVE-24589
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we do a drop catalog we drop the catalog from the CTLGS table. The DBS
> table has a foreign key reference on CTLGS for CTLG_NAME. This is causing the
> DBS table to be locked exclusively and causing deadlocks. This can be avoided
> by creating an index in the DBS table on CTLG_NAME.
> {code:java}
> CREATE INDEX CTLG_NAME_DBS ON DBS(CTLG_NAME); {code}
> {code:java}
> Oracle Database maximizes the concurrency control of parent keys in relation
> to dependent foreign keys.Locking behaviour depends on whether foreign key
> columns are indexed. If foreign keys are not indexed, then the child table
> will probably be locked more frequently, deadlocks will occur, and
> concurrency will be decreased. For this reason foreign keys should almost
> always be indexed. The only exception is when the matching unique or primary
> key is never updated or deleted.{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24616) Add Logging to Track Query Status
[ https://issues.apache.org/jira/browse/HIVE-24616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-24616: - > Add Logging to Track Query Status > - > > Key: HIVE-24616 > URL: https://issues.apache.org/jira/browse/HIVE-24616 > Project: Hive > Issue Type: Improvement > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > > Add additional logging to JDBC to all for tracking the status of a query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24589) Drop catalog failing with deadlock error for Oracle backend dbms.
[
https://issues.apache.org/jira/browse/HIVE-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24589:
--
Labels: pull-request-available (was: )
> Drop catalog failing with deadlock error for Oracle backend dbms.
> -
>
> Key: HIVE-24589
> URL: https://issues.apache.org/jira/browse/HIVE-24589
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we do a drop catalog we drop the catalog from the CTLGS table. The DBS
> table has a foreign key reference on CTLGS for CTLG_NAME. This is causing the
> DBS table to be locked exclusively and causing deadlocks. This can be avoided
> by creating an index in the DBS table on CTLG_NAME.
> {code:java}
> CREATE INDEX CTLG_NAME_DBS ON DBS(CTLG_NAME); {code}
> {code:java}
> Oracle Database maximizes the concurrency control of parent keys in relation
> to dependent foreign keys.Locking behaviour depends on whether foreign key
> columns are indexed. If foreign keys are not indexed, then the child table
> will probably be locked more frequently, deadlocks will occur, and
> concurrency will be decreased. For this reason foreign keys should almost
> always be indexed. The only exception is when the matching unique or primary
> key is never updated or deleted.{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262761#comment-17262761 ] László Bodor commented on HIVE-24539: - PR merged to master, thanks [~pgaref] for the patch > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-24539. - Resolution: Fixed > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-24539: Fix Version/s: 4.0.0 > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?focusedWorklogId=534313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534313 ] ASF GitHub Bot logged work on HIVE-24539: - Author: ASF GitHub Bot Created on: 11/Jan/21 16:25 Start Date: 11/Jan/21 16:25 Worklog Time Spent: 10m Work Description: abstractdog merged pull request #1783: URL: https://github.com/apache/hive/pull/1783 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534313) Time Spent: 2h (was: 1h 50m) > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262748#comment-17262748
]
Stamatis Zampetakis edited comment on HIVE-24590 at 1/11/21, 4:08 PM:
--
Hey [~euigeun_chung], can you try out the patch in
[https://github.com/apache/hive/pull/1849] and see what it gives on your
setting?
was (Author: zabetak):
Hey [~euigeun_chung], can you try out the patch in
[https://github.com/apache/hive/pull/1849] and see what it gives?
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> Screen Shot 2021-01-08 at 21.01.40.png, add_debug_log_and_trace.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262748#comment-17262748
]
Stamatis Zampetakis commented on HIVE-24590:
Hey [~euigeun_chung], can you try out the patch in
[https://github.com/apache/hive/pull/1849] and see what it gives?
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> Screen Shot 2021-01-08 at 21.01.40.png, add_debug_log_and_trace.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?focusedWorklogId=534304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534304
]
ASF GitHub Bot logged work on HIVE-24590:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 16:06
Start Date: 11/Jan/21 16:06
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #1849:
URL: https://github.com/apache/hive/pull/1849
### What changes were proposed in this pull request?
1. Handle dynamic routing appenders via Log4j's `IdlePurgePolicy`
2. Remove in-house management of appenders.
### Why are the changes needed?
1. Solve descriptor leak as shown in HIVE-24590
2. Delegate burden of managing appenders to Log4j.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
`mvn test -Dtest=TestOperationLoggingLayout`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534304)
Remaining Estimate: 0h
Time Spent: 10m
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> Screen Shot 2021-01-08 at 21.01.40.png, add_debug_log_and_trace.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24590:
--
Labels: pull-request-available (was: )
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> Screen Shot 2021-01-08 at 21.01.40.png, add_debug_log_and_trace.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24615) Remove unnecessary FileSystem listing from Initiator
[
https://issues.apache.org/jira/browse/HIVE-24615?focusedWorklogId=534260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534260
]
ASF GitHub Bot logged work on HIVE-24615:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 14:22
Start Date: 11/Jan/21 14:22
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1848:
URL: https://github.com/apache/hive/pull/1848#discussion_r555079627
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -943,7 +947,7 @@ public String toString() {
* not needed {@link ParsedDeltaLight} should be used.
*/
@Immutable
- public static final class ParsedDelta extends ParsedDeltaLight {
+ public static final class ParsedDelta extends ParsedDeltaLight implements
ParsedDirectory{
Review comment:
nit. missing space before {
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534260)
Time Spent: 20m (was: 10m)
> Remove unnecessary FileSystem listing from Initiator
> -
>
> Key: HIVE-24615
> URL: https://issues.apache.org/jira/browse/HIVE-24615
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> AcidUtils already returns the file list in base and delta directories if it
> does recursive listing on S3, listing those directories can be removed from
> the Initiator
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24615) Remove unnecessary FileSystem listing from Initiator
[ https://issues.apache.org/jira/browse/HIVE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24615: -- Labels: pull-request-available (was: ) > Remove unnecessary FileSystem listing from Initiator > - > > Key: HIVE-24615 > URL: https://issues.apache.org/jira/browse/HIVE-24615 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > AcidUtils already returns the file list in base and delta directories if it > does recursive listing on S3, listing those directories can be removed from > the Initiator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24615) Remove unnecessary FileSystem listing from Initiator
[ https://issues.apache.org/jira/browse/HIVE-24615?focusedWorklogId=534251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534251 ] ASF GitHub Bot logged work on HIVE-24615: - Author: ASF GitHub Bot Created on: 11/Jan/21 14:12 Start Date: 11/Jan/21 14:12 Worklog Time Spent: 10m Work Description: pvargacl opened a new pull request #1848: URL: https://github.com/apache/hive/pull/1848 ### What changes were proposed in this pull request? AcidUtils already returns the file list in base and delta directories if it does recursive listing on S3, listing those directories can be removed from the Initiator ### Why are the changes needed? Performance improvement ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534251) Remaining Estimate: 0h Time Spent: 10m > Remove unnecessary FileSystem listing from Initiator > - > > Key: HIVE-24615 > URL: https://issues.apache.org/jira/browse/HIVE-24615 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > AcidUtils already returns the file list in base and delta directories if it > does recursive listing on S3, listing those directories can be removed from > the Initiator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24615) Remove unnecessary FileSystem listing from Initiator
[ https://issues.apache.org/jira/browse/HIVE-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga reassigned HIVE-24615: -- > Remove unnecessary FileSystem listing from Initiator > - > > Key: HIVE-24615 > URL: https://issues.apache.org/jira/browse/HIVE-24615 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > > AcidUtils already returns the file list in base and delta directories if it > does recursive listing on S3, listing those directories can be removed from > the Initiator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24610) Remove superfluous throws IOException from Context
[
https://issues.apache.org/jira/browse/HIVE-24610?focusedWorklogId=534237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534237
]
ASF GitHub Bot logged work on HIVE-24610:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 13:47
Start Date: 11/Jan/21 13:47
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1845:
URL: https://github.com/apache/hive/pull/1845#discussion_r555057154
##
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
##
@@ -196,7 +196,7 @@ public void tearDown() throws Exception {
}
@Test
- public void testBuildDag() throws IllegalArgumentException, IOException,
Exception {
+ public void testBuildDag() throws IllegalArgumentException, Exception {
Review comment:
Nice catch, thanks, fixed.
##
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
##
@@ -217,7 +217,7 @@ public void testBuildDag() throws IllegalArgumentException,
IOException, Excepti
}
@Test
- public void testEmptyWork() throws IllegalArgumentException, IOException,
Exception {
+ public void testEmptyWork() throws IllegalArgumentException, Exception {
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534237)
Time Spent: 40m (was: 0.5h)
> Remove superfluous throws IOException from Context
> --
>
> Key: HIVE-24610
> URL: https://issues.apache.org/jira/browse/HIVE-24610
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23553) Upgrade ORC version to 1.6.X
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262640#comment-17262640 ] Panagiotis Garefalakis edited comment on HIVE-23553 at 1/11/21, 1:36 PM: - Hey [~yisong] – yes ORC 1.6 will have ORC-14 and thus the column-level encryption feature but there is still some work missing on the HIVE side AFAIK. For example, there was a community discussion as part of HIVE-21848 to unify encryption table properties (at least for ORC and Parquet). However, we also need to cary over these properties to the derived Tables, and this is still in progress (HIVE-21849). was (Author: pgaref): Hey [~yisong] – yes ORC 1.6 will have ORC-14 and thus the column-level encryption feature but there is still some work missing in HIVE. For example, there was a community discussion as part of HIVE-21848 to unify encryption table properties (at least for ORC and Parquet). However, we also need to cary over these properties to the derived Tables, and this is still in progress (HIVE-21849). > Upgrade ORC version to 1.6.X > > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Apache Hive is currently on 1.5.X version and in order to take advantage of > the latest ORC improvements such as column encryption we have to bump to > 1.6.X. > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288==12318320=Create_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin > Even though ORC reader could work out of the box, HIVE LLAP is heavily > depending on internal ORC APIs e.g., to retrieve and store File Footers, > Tails, streams – un/compress RG data etc. As there ware many internal changes > from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the > upgrade is not straightforward. > This Umbrella Jira tracks this upgrade effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23553) Upgrade ORC version to 1.6.X
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262640#comment-17262640 ] Panagiotis Garefalakis commented on HIVE-23553: --- Hey [~yisong] – yes ORC 1.6 will have ORC-14 and thus the column-level encryption feature but there is still some work missing in HIVE. For example, there was a community discussion as part of HIVE-21848 to unify encryption table properties (at least for ORC and Parquet). However, we also need to cary over these properties to the derived Tables, and this is still in progress (HIVE-21849). > Upgrade ORC version to 1.6.X > > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Apache Hive is currently on 1.5.X version and in order to take advantage of > the latest ORC improvements such as column encryption we have to bump to > 1.6.X. > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288==12318320=Create_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin > Even though ORC reader could work out of the box, HIVE LLAP is heavily > depending on internal ORC APIs e.g., to retrieve and store File Footers, > Tails, streams – un/compress RG data etc. As there ware many internal changes > from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the > upgrade is not straightforward. > This Umbrella Jira tracks this upgrade effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24614) using coalesce via vector,when date type of column are different between source and target,the result of target is zero
[
https://issues.apache.org/jira/browse/HIVE-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
taoyuyin updated HIVE-24614:
Description:
set hive.vectorized.execution.enabled=true;
CREATE TABLE `tmp.tmp_test_vectorization_source`(
`rn` string,
`val_1` int,
`val_2` bigint)
stored as parquet;
insert into table `tmp.tmp_test_vectorization_source`
values('line1',1000,10001),('line1',2000,20001);
select rn,val_1,val_2 from tmp.tmp_test_vectorization_source t;
+-+---++
|rn|val_1|val_2|
+-+---++
|line1|1000|10001|
|line1|2000|20001|
+-+---++
CREATE TABLE `tmp.tmp_test_vectorization_target`(
`rn` string,
`val_1` bigint,
`val_2` int)
stored as parquet;
insert into table tmp.tmp_test_vectorization_target
select
rn,
coalesce(val_1,0),
coalesce(val_2,0)
from tmp.tmp_test_vectorization_source;
select rn,val_1,val_2 from tmp.tmp_test_vectorization_target t;
+-+---++
|rn|val_1|val_2|
+-+---++
|line1|0|0|
|line1|0|0|
+-+---++
> using coalesce via vector,when date type of column are different between
> source and target,the result of target is zero
> ---
>
> Key: HIVE-24614
> URL: https://issues.apache.org/jira/browse/HIVE-24614
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.2
>Reporter: taoyuyin
>Priority: Major
>
> set hive.vectorized.execution.enabled=true;
>
> CREATE TABLE `tmp.tmp_test_vectorization_source`(
> `rn` string,
> `val_1` int,
> `val_2` bigint)
> stored as parquet;
>
> insert into table `tmp.tmp_test_vectorization_source`
> values('line1',1000,10001),('line1',2000,20001);
>
> select rn,val_1,val_2 from tmp.tmp_test_vectorization_source t;
>
> +-+---++
> |rn|val_1|val_2|
> +-+---++
> |line1|1000|10001|
> |line1|2000|20001|
> +-+---++
>
> CREATE TABLE `tmp.tmp_test_vectorization_target`(
> `rn` string,
> `val_1` bigint,
> `val_2` int)
> stored as parquet;
>
> insert into table tmp.tmp_test_vectorization_target
> select
> rn,
> coalesce(val_1,0),
> coalesce(val_2,0)
> from tmp.tmp_test_vectorization_source;
>
> select rn,val_1,val_2 from tmp.tmp_test_vectorization_target t;
>
> +-+---++
> |rn|val_1|val_2|
> +-+---++
> |line1|0|0|
> |line1|0|0|
> +-+---++
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24614) using coalesce via vector,when date type of column are different between source and target,the result of target is zero
[
https://issues.apache.org/jira/browse/HIVE-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
taoyuyin updated HIVE-24614:
Environment: (was: set hive.vectorized.execution.enabled=true;
CREATE TABLE `tmp.tmp_test_vectorization_source`(
`rn` string,
`val_1` int,
`val_2` bigint)
stored as parquet;
insert into table `tmp.tmp_test_vectorization_source`
values('line1',1000,10001),('line1',2000,20001);
select rn,val_1,val_2 from tmp.tmp_test_vectorization_source t;
++++
| rn | val_1 | val_2 |
++++
| line1 | 1000 | 10001 |
| line1 | 2000 | 20001 |
++++
CREATE TABLE `tmp.tmp_test_vectorization_target`(
`rn` string,
`val_1` bigint,
`val_2` int)
stored as parquet;
insert into table tmp.tmp_test_vectorization_target
select
rn,
coalesce(val_1,0),
coalesce(val_2,0)
from tmp.tmp_test_vectorization_source;
select rn,val_1,val_2 from tmp.tmp_test_vectorization_target t;
++++
| rn | val_1 | val_2 |
++++
| line1 | 0 | 0 |
| line1 | 0 | 0 |
++++)
> using coalesce via vector,when date type of column are different between
> source and target,the result of target is zero
> ---
>
> Key: HIVE-24614
> URL: https://issues.apache.org/jira/browse/HIVE-24614
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.1.2
>Reporter: taoyuyin
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23553) Upgrade ORC version to 1.6.X
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23553: -- Summary: Upgrade ORC version to 1.6.X (was: Upgrade ORC version to 1.6.6) > Upgrade ORC version to 1.6.X > > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Apache Hive is currently on 1.5.X version and in order to take advantage of > the latest ORC improvements such as column encryption we have to bump to > 1.6.X. > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288==12318320=Create_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin > Even though ORC reader could work out of the box, HIVE LLAP is heavily > depending on internal ORC APIs e.g., to retrieve and store File Footers, > Tails, streams – un/compress RG data etc. As there ware many internal changes > from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the > upgrade is not straightforward. > This Umbrella Jira tracks this upgrade effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24610) Remove superfluous throws IOException from Context
[
https://issues.apache.org/jira/browse/HIVE-24610?focusedWorklogId=534224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534224
]
ASF GitHub Bot logged work on HIVE-24610:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 13:07
Start Date: 11/Jan/21 13:07
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1845:
URL: https://github.com/apache/hive/pull/1845#discussion_r555030681
##
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
##
@@ -196,7 +196,7 @@ public void tearDown() throws Exception {
}
@Test
- public void testBuildDag() throws IllegalArgumentException, IOException,
Exception {
+ public void testBuildDag() throws IllegalArgumentException, Exception {
Review comment:
Listing `IllegalArgumentException` is not necessary if `Exception` can
be thrown.
Or this can be tackled in a follow-up
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534224)
Time Spent: 0.5h (was: 20m)
> Remove superfluous throws IOException from Context
> --
>
> Key: HIVE-24610
> URL: https://issues.apache.org/jira/browse/HIVE-24610
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24610) Remove superfluous throws IOException from Context
[
https://issues.apache.org/jira/browse/HIVE-24610?focusedWorklogId=534222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534222
]
ASF GitHub Bot logged work on HIVE-24610:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 13:05
Start Date: 11/Jan/21 13:05
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1845:
URL: https://github.com/apache/hive/pull/1845#discussion_r555030681
##
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
##
@@ -196,7 +196,7 @@ public void tearDown() throws Exception {
}
@Test
- public void testBuildDag() throws IllegalArgumentException, IOException,
Exception {
+ public void testBuildDag() throws IllegalArgumentException, Exception {
Review comment:
Listing `IllegalArgumentException` is not necessary if `Exception` can
be thrown.
##
File path: ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
##
@@ -217,7 +217,7 @@ public void testBuildDag() throws IllegalArgumentException,
IOException, Excepti
}
@Test
- public void testEmptyWork() throws IllegalArgumentException, IOException,
Exception {
+ public void testEmptyWork() throws IllegalArgumentException, Exception {
Review comment:
Listing `IllegalArgumentException` is not necessary if `Exception` can
be thrown.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534222)
Time Spent: 20m (was: 10m)
> Remove superfluous throws IOException from Context
> --
>
> Key: HIVE-24610
> URL: https://issues.apache.org/jira/browse/HIVE-24610
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24511) Fix typo in SerDeStorageSchemaReader
[
https://issues.apache.org/jira/browse/HIVE-24511?focusedWorklogId=534217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534217
]
ASF GitHub Bot logged work on HIVE-24511:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 12:45
Start Date: 11/Jan/21 12:45
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1757:
URL: https://github.com/apache/hive/pull/1757#discussion_r555021517
##
File path:
metastore/src/java/org/apache/hadoop/hive/metastore/SerDeStorageSchemaReader.java
##
@@ -47,10 +48,10 @@
Deserializer s = HiveMetaStoreUtils.getDeserializer(conf, tbl, false);
return HiveMetaStoreUtils.getFieldsFromDeserializer(tbl.getTableName(),
s);
} catch (Exception e) {
- StringUtils.stringifyException(e);
- throw new MetaException(e.getMessage());
+ throw new MetaException(StringUtils.stringifyException(e));
Review comment:
Hey @belugabehr, could you please take another look? thanks a lot.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534217)
Time Spent: 1h (was: 50m)
> Fix typo in SerDeStorageSchemaReader
>
>
> Key: HIVE-24511
> URL: https://issues.apache.org/jira/browse/HIVE-24511
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> 1, Close the created classloader to release resources.
> 2, More detail error messages on MetaException when throwing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24613) Support Values clause without Insert
[
https://issues.apache.org/jira/browse/HIVE-24613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24613:
--
Labels: pull-request-available (was: )
> Support Values clause without Insert
>
>
> Key: HIVE-24613
> URL: https://issues.apache.org/jira/browse/HIVE-24613
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Standalone:
> {code}
> VALUES(1,2,3),(4,5,6);
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
> In subquery:
> {code}
> SELECT * FROM (VALUES(1,2,3),(4,5,6)) as FOO;
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24613) Support Values clause without Insert
[
https://issues.apache.org/jira/browse/HIVE-24613?focusedWorklogId=534216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534216
]
ASF GitHub Bot logged work on HIVE-24613:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 12:40
Start Date: 11/Jan/21 12:40
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #1847:
URL: https://github.com/apache/hive/pull/1847
### What changes were proposed in this pull request?
Prior this patch for defining a row of constant values Select clause could
be used:
```
SELECT * FROM t1 foo
JOIN (select 1,'a') as bar ON foo.col1 = bar.col1
```
This enables the usage of Values clasuse:
```
SELECT * FROM t1 foo
JOIN (VALUES(1,'a'),(4,'b')) as bar ON foo.col1 = bar.col1
```
### Why are the changes needed?
With Select clause only one row could be defined. There is no such
limitation with Values clause.
### Does this PR introduce _any_ user-facing change?
Yes. From now the Values clause can be used in subqueries and can be a query
itself.
### How was this patch tested?
```
mvn test -Dtest=TestValuesClause -pl parser
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=values.q -pl itests/qtest -Pitests
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534216)
Remaining Estimate: 0h
Time Spent: 10m
> Support Values clause without Insert
>
>
> Key: HIVE-24613
> URL: https://issues.apache.org/jira/browse/HIVE-24613
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Standalone:
> {code}
> VALUES(1,2,3),(4,5,6);
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
> In subquery:
> {code}
> SELECT * FROM (VALUES(1,2,3),(4,5,6)) as FOO;
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24514) UpdateMDatabaseURI does not update managed location URI
[ https://issues.apache.org/jira/browse/HIVE-24514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262611#comment-17262611 ] Prasanth Jayachandran commented on HIVE-24514: -- [~ngangam] can you please take another look? addressed your review comment. > UpdateMDatabaseURI does not update managed location URI > --- > > Key: HIVE-24514 > URL: https://issues.apache.org/jira/browse/HIVE-24514 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When FS Root is updated using metatool, if the DB has managed location > defined, > updateMDatabaseURI API should update the managed location as well. Currently > it only updates location uri. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?focusedWorklogId=534207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534207 ] ASF GitHub Bot logged work on HIVE-24539: - Author: ASF GitHub Bot Created on: 11/Jan/21 12:20 Start Date: 11/Jan/21 12:20 Worklog Time Spent: 10m Work Description: pgaref edited a comment on pull request #1783: URL: https://github.com/apache/hive/pull/1783#issuecomment-757916044 Thanks for taking a look @abstractdog ! Shall we push this change before getting out-of date? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534207) Time Spent: 1h 50m (was: 1h 40m) > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24539) OrcInputFormat schema generation should respect column delimiter
[ https://issues.apache.org/jira/browse/HIVE-24539?focusedWorklogId=534206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534206 ] ASF GitHub Bot logged work on HIVE-24539: - Author: ASF GitHub Bot Created on: 11/Jan/21 12:18 Start Date: 11/Jan/21 12:18 Worklog Time Spent: 10m Work Description: pgaref commented on pull request #1783: URL: https://github.com/apache/hive/pull/1783#issuecomment-757916044 Shall we push this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534206) Time Spent: 1h 40m (was: 1.5h) > OrcInputFormat schema generation should respect column delimiter > > > Key: HIVE-24539 > URL: https://issues.apache.org/jira/browse/HIVE-24539 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > OrcInputFormat currently generates schema using the given configuration and > the default delimiter – that causes inconsistencies when names contain commas. > We should follow a similar approach to > [OrcOutputFormat|https://github.com/apache/hive/blob/9563dd63188280f4b7c307f36e1ea0c69aec/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java#L145] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24613) Support Values clause without Insert
[
https://issues.apache.org/jira/browse/HIVE-24613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-24613:
-
> Support Values clause without Insert
>
>
> Key: HIVE-24613
> URL: https://issues.apache.org/jira/browse/HIVE-24613
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>
> Standalone:
> {code}
> VALUES(1,2,3),(4,5,6);
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
> In subquery:
> {code}
> SELECT * FROM (VALUES(1,2,3),(4,5,6)) as FOO;
> {code}
> {code}
> 1 2 3
> 4 5 6
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24581) Remove AcidUtils call from OrcInputformat for non transactional tables
[ https://issues.apache.org/jira/browse/HIVE-24581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora resolved HIVE-24581. -- Fix Version/s: 4.0.0 Resolution: Fixed > Remove AcidUtils call from OrcInputformat for non transactional tables > -- > > Key: HIVE-24581 > URL: https://issues.apache.org/jira/browse/HIVE-24581 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently the split generation in OrcInputformat is tightly coupled with acid > and AcidUtils.getAcidState is called even if the table is not transactional. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24581) Remove AcidUtils call from OrcInputformat for non transactional tables
[ https://issues.apache.org/jira/browse/HIVE-24581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262504#comment-17262504 ] Marta Kuczora commented on HIVE-24581: -- Pushed to master. Thanks a lot [~pvargacl] for the fix. > Remove AcidUtils call from OrcInputformat for non transactional tables > -- > > Key: HIVE-24581 > URL: https://issues.apache.org/jira/browse/HIVE-24581 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently the split generation in OrcInputformat is tightly coupled with acid > and AcidUtils.getAcidState is called even if the table is not transactional. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24581) Remove AcidUtils call from OrcInputformat for non transactional tables
[ https://issues.apache.org/jira/browse/HIVE-24581?focusedWorklogId=534162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534162 ] ASF GitHub Bot logged work on HIVE-24581: - Author: ASF GitHub Bot Created on: 11/Jan/21 09:18 Start Date: 11/Jan/21 09:18 Worklog Time Spent: 10m Work Description: kuczoram merged pull request #1826: URL: https://github.com/apache/hive/pull/1826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 534162) Time Spent: 1h 10m (was: 1h) > Remove AcidUtils call from OrcInputformat for non transactional tables > -- > > Key: HIVE-24581 > URL: https://issues.apache.org/jira/browse/HIVE-24581 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently the split generation in OrcInputformat is tightly coupled with acid > and AcidUtils.getAcidState is called even if the table is not transactional. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24581) Remove AcidUtils call from OrcInputformat for non transactional tables
[
https://issues.apache.org/jira/browse/HIVE-24581?focusedWorklogId=534160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534160
]
ASF GitHub Bot logged work on HIVE-24581:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 09:13
Start Date: 11/Jan/21 09:13
Worklog Time Spent: 10m
Work Description: kuczoram commented on a change in pull request #1826:
URL: https://github.com/apache/hive/pull/1826#discussion_r554906173
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -1866,92 +1870,6 @@ private static boolean isDirUsable(Path child, long
visibilityTxnId, List
return true;
}
- public static class HdfsFileStatusWithoutId implements HdfsFileStatusWithId {
-private final FileStatus fs;
-
-public HdfsFileStatusWithoutId(FileStatus fs) {
- this.fs = fs;
-}
-
-@Override
-public FileStatus getFileStatus() {
- return fs;
-}
-
-@Override
-public Long getFileId() {
- return null;
-}
- }
-
- /**
- * Find the original files (non-ACID layout) recursively under the partition
directory.
- * @param fs the file system
- * @param dir the directory to add
- * @return the list of original files
- * @throws IOException
- */
- public static List findOriginals(FileSystem fs, Path
dir, Ref useFileIds,
- boolean ignoreEmptyFiles, boolean recursive) throws IOException {
-List originals = new ArrayList<>();
-List childrenWithId =
tryListLocatedHdfsStatus(useFileIds, fs, dir, hiddenFileFilter);
-if (childrenWithId != null) {
- for (HdfsFileStatusWithId child : childrenWithId) {
-if (child.getFileStatus().isDirectory()) {
- if (recursive) {
-originals.addAll(findOriginals(fs,
child.getFileStatus().getPath(), useFileIds,
-ignoreEmptyFiles, true));
- }
-} else {
- if (!ignoreEmptyFiles || child.getFileStatus().getLen() > 0) {
-originals.add(child);
- }
-}
- }
-} else {
- List children = HdfsUtils.listLocatedStatus(fs, dir,
hiddenFileFilter);
- for (FileStatus child : children) {
-if (child.isDirectory()) {
- if (recursive) {
-originals.addAll(findOriginals(fs, child.getPath(), useFileIds,
ignoreEmptyFiles, true));
- }
-} else {
- if (!ignoreEmptyFiles || child.getLen() > 0) {
Review comment:
Oh, ok, I see. Thanks for the details.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534160)
Time Spent: 1h (was: 50m)
> Remove AcidUtils call from OrcInputformat for non transactional tables
> --
>
> Key: HIVE-24581
> URL: https://issues.apache.org/jira/browse/HIVE-24581
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently the split generation in OrcInputformat is tightly coupled with acid
> and AcidUtils.getAcidState is called even if the table is not transactional.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24523) Vectorized read path for LazySimpleSerde does not honor the SERDEPROPERTIES for timestamp
[
https://issues.apache.org/jira/browse/HIVE-24523?focusedWorklogId=534141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-534141
]
ASF GitHub Bot logged work on HIVE-24523:
-
Author: ASF GitHub Bot
Created on: 11/Jan/21 08:18
Start Date: 11/Jan/21 08:18
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1825:
URL: https://github.com/apache/hive/pull/1825#discussion_r554879974
##
File path:
serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/LazySimpleDeserializeRead.java
##
@@ -350,7 +350,12 @@ public LazySimpleDeserializeRead(TypeInfo[] typeInfos,
throw new RuntimeException("serialization.last.column.takes.rest not
supported");
}
-timestampParser = new TimestampParser();
+List timestampFormats = lazyParams.getTimestampFormats();
+if (timestampFormats != null) {
Review comment:
minor: i would use ternary operator here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 534141)
Time Spent: 20m (was: 10m)
> Vectorized read path for LazySimpleSerde does not honor the SERDEPROPERTIES
> for timestamp
> -
>
> Key: HIVE-24523
> URL: https://issues.apache.org/jira/browse/HIVE-24523
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 3.2.0, 4.0.0
>Reporter: Rajkumar Singh
>Assignee: Naresh P R
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Steps to repro:
> {code:java}
> create external table tstable(date_created timestamp) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES (
> 'timestamp.formats'='MMddHHmmss') stored as textfile;
> cat sampledata
> 2020120517
> hdfs dfs -put sampledata /warehouse/tablespace/external/hive/tstable
> {code}
> disable fetch task conversion and run select * from tstable which produce no
> results, disabling the set
> hive.vectorized.use.vector.serde.deserialize=false; return the expected
> output.
> while parsing the string to timestamp
> https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazy/fast/LazySimpleDeserializeRead.java#L812
> does not set the DateTimeFormatter which results IllegalArgumentException
> while parsing the timestamp through TimestampUtils.stringToTimestamp(strValue)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
