[jira] [Updated] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25895: -- Labels: pull-request-available (was: ) > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[
https://issues.apache.org/jira/browse/HIVE-25895?focusedWorklogId=719898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719898
]
ASF GitHub Bot logged work on HIVE-25895:
-
Author: ASF GitHub Bot
Created on: 03/Feb/22 05:40
Start Date: 03/Feb/22 05:40
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2980:
URL: https://github.com/apache/hive/pull/2980#discussion_r798225837
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/OptimisedBootstrapUtils.java
##
@@ -70,6 +71,8 @@
/** event ack file which contains the event id till which the cluster was
last loaded. */
public static final String EVENT_ACK_FILE = "event_ack";
+ public static final String BOOTSTRAP_TABLES_LIST = "bootstrap_table_list";
Review comment:
What would happen in rollback case , like we initiated the failover but
aborted the process in between. Theoretically, upto what point will we allow
that to happen?
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOptimisedBootstrap.java
##
@@ -734,4 +743,167 @@ public NotificationEventResponse apply(@Nullable
NotificationEventResponse event
InjectableBehaviourObjectStore.resetGetNextNotificationBehaviour(); //
reset the behaviour
}
}
+
+
+ @Test
+ public void testReverseBootstrap() throws Throwable {
+List withClause =
ReplicationTestUtils.includeExternalTableClause(true);
+withClause.add("'" + HiveConf.ConfVars.REPLDIR.varname + "'='" +
primary.repldDir + "'");
+
+// Do a bootstrap cycle.
+primary.dump(primaryDbName, withClause);
+replica.load(replicatedDbName, primaryDbName, withClause);
+
+// Create 4 managed tables and do a dump & load.
+WarehouseInstance.Tuple tuple = primary.run("use " + primaryDbName)
+.run("create table t1 (id int)")
+.run("insert into table t1 values (1)")
+.run("insert into table t1 values (2),(3),(4)")
+.run("create table t2 (place string) partitioned by (country string)")
+.run("insert into table t2 partition(country='india') values
('chennai')")
+.run("insert into table t2 partition(country='us') values ('new
york')")
+.run("create table t3 (id int)")
+.run("insert into table t3 values (10)")
+.run("insert into table t3 values (20),(31),(42)")
+.run("create table t4 (place string) partitioned by (country string)")
+.run("insert into table t4 partition(country='india') values
('bangalore')")
+.run("insert into table t4 partition(country='us') values ('austin')")
+.dump(primaryDbName, withClause);
+
+// Do the load and check all the external & managed tables are present.
+replica.load(replicatedDbName, primaryDbName, withClause)
+.run("repl status " + replicatedDbName)
+.verifyResult(tuple.lastReplicationId)
+.run("use " + replicatedDbName)
+.run("show tables like 't1'")
+.verifyResult("t1")
+.run("show tables like 't2'")
+.verifyResult("t2")
+.run("show tables like 't3'")
+.verifyResult("t3")
+.run("show tables like 't4'")
+.verifyResult("t4")
+.verifyReplTargetProperty(replicatedDbName);
+
+
+// Do some modifications on original source cluster. The diff
becomes(tnew_managed, t1, t2, t3)
+primary.run("use " + primaryDbName)
+.run("create table tnew_managed (id int)")
+.run("insert into table t1 values (25)")
+.run("insert into table tnew_managed values (110)")
+.run("insert into table t2 partition(country='france') values
('lyon')")
+.run("drop table t3");
+
+// Do some modifications on the target cluster. (t1, t2, t3: bootstrap &
t4, t5: incremental)
+replica.run("use " + replicatedDbName)
+.run("insert into table t1 values (101)")
+.run("insert into table t1 values (210),(321)")
+.run("insert into table t2 partition(country='india') values
('delhi')")
+.run("insert into table t3 values (11)")
+.run("insert into table t4 partition(country='india') values
('lucknow')")
+.run("create table t5 (place string) partitioned by (country string)")
+.run("insert into table t5 partition(country='china') values
('beejing')");
+
+// Prepare for reverse replication.
+DistributedFileSystem replicaFs = replica.miniDFSCluster.getFileSystem();
+Path newReplDir = new Path(replica.repldDir + "1");
+replicaFs.mkdirs(newReplDir);
+withClause = ReplicationTestUtils.includeExternalTableClause(true);
+withClause.add("'" + HiveConf.ConfVars.REPLDIR.varname + "'='" +
newReplDir + "'");
+
+// Do a reverse dump
+tuple = replica.dump(replicatedDbName, withClause);
+
+// Check the event ack file got created.
+
[jira] [Resolved] (HIVE-25923) CLONE - SchemaTool error: Unknown version specified for initialization: 3.1.0
[
https://issues.apache.org/jira/browse/HIVE-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOHAMMAD AAMIR resolved HIVE-25923.
---
Resolution: Fixed
> CLONE - SchemaTool error: Unknown version specified for initialization: 3.1.0
> -
>
> Key: HIVE-25923
> URL: https://issues.apache.org/jira/browse/HIVE-25923
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 3.1.1
>Reporter: MOHAMMAD AAMIR
>Assignee: MOHAMMAD AAMIR
>Priority: Critical
>
> {{While trying to initialise the schema using SchemaTool in Hive 3.1.1, it
> was failing with this message `}}
> {code:java}
> Unknown version specified for initialization: 3.1.0
> {code}
> {{`}}
> It looks to be a bug to me. I had to use Apache Hive 3.0.0
> {{}}
> {code:java}
> // ./schematool -dbType Derby -initSchema --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/rhel/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/rhel/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
> Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
> Metastore connection User: APP
> Starting metastore schema initialization to 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> at
> org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.generateInitFileName(MetaStoreSchemaInfo.java:137)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:585)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> *** schemaTool failed ***
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25924) CLONE - CLONE - SchemaTool error: Unknown version specified for initialization: 3.1.0
[
https://issues.apache.org/jira/browse/HIVE-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOHAMMAD AAMIR reassigned HIVE-25924:
-
> CLONE - CLONE - SchemaTool error: Unknown version specified for
> initialization: 3.1.0
> -
>
> Key: HIVE-25924
> URL: https://issues.apache.org/jira/browse/HIVE-25924
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 3.1.1
>Reporter: MOHAMMAD AAMIR
>Assignee: MOHAMMAD AAMIR
>Priority: Critical
>
> {{While trying to initialise the schema using SchemaTool in Hive 3.1.1, it
> was failing with this message `}}
> {code:java}
> Unknown version specified for initialization: 3.1.0
> {code}
> {{`}}
> It looks to be a bug to me. I had to use Apache Hive 3.0.0
> {{}}
> {code:java}
> // ./schematool -dbType Derby -initSchema --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/rhel/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/rhel/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
> Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
> Metastore connection User: APP
> Starting metastore schema initialization to 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> at
> org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.generateInitFileName(MetaStoreSchemaInfo.java:137)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:585)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> *** schemaTool failed ***
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25923) CLONE - SchemaTool error: Unknown version specified for initialization: 3.1.0
[
https://issues.apache.org/jira/browse/HIVE-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOHAMMAD AAMIR reassigned HIVE-25923:
-
> CLONE - SchemaTool error: Unknown version specified for initialization: 3.1.0
> -
>
> Key: HIVE-25923
> URL: https://issues.apache.org/jira/browse/HIVE-25923
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 3.1.1
>Reporter: MOHAMMAD AAMIR
>Assignee: MOHAMMAD AAMIR
>Priority: Critical
>
> {{While trying to initialise the schema using SchemaTool in Hive 3.1.1, it
> was failing with this message `}}
> {code:java}
> Unknown version specified for initialization: 3.1.0
> {code}
> {{`}}
> It looks to be a bug to me. I had to use Apache Hive 3.0.0
> {{}}
> {code:java}
> // ./schematool -dbType Derby -initSchema --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/rhel/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/rhel/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
> Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
> Metastore connection User: APP
> Starting metastore schema initialization to 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> at
> org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.generateInitFileName(MetaStoreSchemaInfo.java:137)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:585)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> *** schemaTool failed ***
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-21623) SchemaTool error: Unknown version specified for initialization: 3.1.0
[
https://issues.apache.org/jira/browse/HIVE-21623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOHAMMAD AAMIR reassigned HIVE-21623:
-
Assignee: MOHAMMAD AAMIR
> SchemaTool error: Unknown version specified for initialization: 3.1.0
> -
>
> Key: HIVE-21623
> URL: https://issues.apache.org/jira/browse/HIVE-21623
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 3.1.1
>Reporter: Namana H G
>Assignee: MOHAMMAD AAMIR
>Priority: Critical
> Attachments: Screenshot 2019-04-17 at 6.59.49 PM.png
>
>
> {{While trying to initialise the schema using SchemaTool in Hive 3.1.1, it
> was failing with this message `}}
> {code:java}
> Unknown version specified for initialization: 3.1.0
> {code}
> {{`}}
> It looks to be a bug to me. I had to use Apache Hive 3.0.0
> {{}}
> {code:java}
> // ./schematool -dbType Derby -initSchema --verbose
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/rhel/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/rhel/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
> Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
> Metastore connection User: APP
> Starting metastore schema initialization to 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> org.apache.hadoop.hive.metastore.HiveMetaException: Unknown version specified
> for initialization: 3.1.0
> at
> org.apache.hadoop.hive.metastore.MetaStoreSchemaInfo.generateInitFileName(MetaStoreSchemaInfo.java:137)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:585)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:567)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> *** schemaTool failed ***
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25875) Support multiple authentication mechanisms simultaneously
[
https://issues.apache.org/jira/browse/HIVE-25875?focusedWorklogId=719590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719590
]
ASF GitHub Bot logged work on HIVE-25875:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 19:26
Start Date: 02/Feb/22 19:26
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #2954:
URL: https://github.com/apache/hive/pull/2954#discussion_r797943031
##
File path:
service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java
##
@@ -214,9 +214,13 @@ protected void doPost(HttpServletRequest request,
HttpServletResponse response)
}
} else if (HiveSamlUtils.isSamlAuthMode(authType)) {
// check if this request needs a SAML redirect
-if (needsRedirect(request, response)) {
+String authHeader = request.getHeader(HttpAuthUtils.AUTHORIZATION);
+if ((authHeader == null || authHeader.isEmpty()) &&
needsRedirect(request, response)) {
doSamlRedirect(request, response);
return;
+} else
if(authHeader.toLowerCase().startsWith(HttpAuthUtils.BASIC.toLowerCase())) {
+ //LDAP Authentication if the header starts with Basic
+ clientUserName = doPasswdAuth(request,
HiveAuthConstants.AuthTypes.NONE.toString());
Review comment:
@hsnusonic to answer your question, yes. This should be the first check
in the authentication process. We dont expect this trust header to exist in
many scenarios. Sai will be looking to rearrange the code upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 719590)
Time Spent: 50m (was: 40m)
> Support multiple authentication mechanisms simultaneously
> --
>
> Key: HIVE-25875
> URL: https://issues.apache.org/jira/browse/HIVE-25875
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently, HS2 supports a single form of auth on any given instance of
> HiveServer2. Hive should be able to support multiple auth mechanisms on a
> single instance especially with http transport. for example, LDAP and SAML.
> In both cases, HS2 ends up with receiving an Authorization header in the
> request.
> Similarly, we could be able to support JWT support or other forms of boundary
> authentication that is done outside of Hive (This is not in the scope of this
> jira).
> *The current scope of the Jira:* Support SAML and LDAP authentication modes
> simultaneously when the transport mode is set to *HTTP.* This multi-auth will
> not work when transport mode is set to {_}binary or all{_}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25875) Support multiple authentication mechanisms simultaneously
[
https://issues.apache.org/jira/browse/HIVE-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala updated HIVE-25875:
-
Description:
Currently, HS2 supports a single form of auth on any given instance of
HiveServer2. Hive should be able to support multiple auth mechanisms on a
single instance especially with http transport. for example, LDAP and SAML. In
both cases, HS2 ends up with receiving an Authorization header in the request.
Similarly, we could be able to support JWT support or other forms of boundary
authentication that is done outside of Hive (This is not in the scope of this
jira).
*The current scope of the Jira:* Support SAML and LDAP authentication modes
simultaneously when the transport mode is set to *HTTP.* This multi-auth will
not work when transport mode is set to {_}binary or all{_}.
was:
Currently, HS2 supports a single form of auth on any given instance of
HiveServer2. Hive should be able to support multiple auth mechanisms on a
single instance especially with http transport. for example, LDAP and SAML. In
both cases, HS2 ends up with receiving an Authorization header in the request.
Similarly, we could be able to support JWT support or other forms of boundary
authentication that is done outside of Hive (This is not in the scope of this
jira).
> Support multiple authentication mechanisms simultaneously
> --
>
> Key: HIVE-25875
> URL: https://issues.apache.org/jira/browse/HIVE-25875
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, HS2 supports a single form of auth on any given instance of
> HiveServer2. Hive should be able to support multiple auth mechanisms on a
> single instance especially with http transport. for example, LDAP and SAML.
> In both cases, HS2 ends up with receiving an Authorization header in the
> request.
> Similarly, we could be able to support JWT support or other forms of boundary
> authentication that is done outside of Hive (This is not in the scope of this
> jira).
> *The current scope of the Jira:* Support SAML and LDAP authentication modes
> simultaneously when the transport mode is set to *HTTP.* This multi-auth will
> not work when transport mode is set to {_}binary or all{_}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25875) Support multiple authentication mechanisms simultaneously
[ https://issues.apache.org/jira/browse/HIVE-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-25875: - Description: Currently, HS2 supports a single form of auth on any given instance of HiveServer2. Hive should be able to support multiple auth mechanisms on a single instance especially with http transport. for example, LDAP and SAML. In both cases, HS2 ends up with receiving an Authorization header in the request. Similarly, we could be able to support JWT support or other forms of boundary authentication that is done outside of Hive (This is not in the scope of this jira). was:Currently, HS2 supports a single form of auth on any given instance of HiveServer2. Hive should be able to support multiple auth mechanisms on a single instance especially with http transport. for example, LDAP and SAML. In both cases, HS2 ends up with receiving an Authorization header in the request. Similarly, we could be able to support JWT support or other forms of boundary authentication that is done outside of Hive. > Support multiple authentication mechanisms simultaneously > -- > > Key: HIVE-25875 > URL: https://issues.apache.org/jira/browse/HIVE-25875 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Currently, HS2 supports a single form of auth on any given instance of > HiveServer2. Hive should be able to support multiple auth mechanisms on a > single instance especially with http transport. for example, LDAP and SAML. > In both cases, HS2 ends up with receiving an Authorization header in the > request. > Similarly, we could be able to support JWT support or other forms of boundary > authentication that is done outside of Hive (This is not in the scope of this > jira). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25875) Support multiple authentication mechanisms simultaneously
[ https://issues.apache.org/jira/browse/HIVE-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala updated HIVE-25875: - Description: Currently, HS2 supports a single form of auth on any given instance of HiveServer2. Hive should be able to support multiple auth mechanisms on a single instance especially with http transport. for example, LDAP and SAML. In both cases, HS2 ends up with receiving an Authorization header in the request. Similarly, we could be able to support JWT support or other forms of boundary authentication that is done outside of Hive. (was: Currently, HS2 supports a single form of auth on any given instance of HiveServer2. Hive should be able to support multiple auth mechanisms on a single instance especially with http transport. for example, LDAP and SAML. In both cases, HS2 ends up with receiving an Authorization Header in the request. Similarly we could be able to support JWT support or other forms of boundary authentication that is done outside of Hive.) > Support multiple authentication mechanisms simultaneously > -- > > Key: HIVE-25875 > URL: https://issues.apache.org/jira/browse/HIVE-25875 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Currently, HS2 supports a single form of auth on any given instance of > HiveServer2. Hive should be able to support multiple auth mechanisms on a > single instance especially with http transport. for example, LDAP and SAML. > In both cases, HS2 ends up with receiving an Authorization header in the > request. Similarly, we could be able to support JWT support or other forms of > boundary authentication that is done outside of Hive. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?focusedWorklogId=719515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719515
]
ASF GitHub Bot logged work on HIVE-25912:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 17:14
Start Date: 02/Feb/22 17:14
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2987:
URL: https://github.com/apache/hive/pull/2987#issuecomment-1028164028
Makes sense, but be aware that at lot of code doesn't like using / as a
destination of work, as it is a special directory.
* paths are wrong
* rm / doesn't delete it
even after this is fixed, there's a risk that jobs to / will fail for other
reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 719515)
Remaining Estimate: 95h 40m (was: 95h 50m)
Time Spent: 20m (was: 10m)
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Attachments: hive bugs.png
>
> Original Estimate: 96h
> Time Spent: 20m
> Remaining Estimate: 95h 40m
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HIVE-25912:
--
Summary: Drop external table at root of s3 bucket throws NPE (was: Drop
external table throw NPE)
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Attachments: hive bugs.png
>
> Original Estimate: 96h
> Time Spent: 10m
> Remaining Estimate: 95h 50m
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25922) Upgrade Iceberg dependency to 0.13.0
[
https://issues.apache.org/jira/browse/HIVE-25922?focusedWorklogId=719484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719484
]
ASF GitHub Bot logged work on HIVE-25922:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 16:36
Start Date: 02/Feb/22 16:36
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2994:
URL: https://github.com/apache/hive/pull/2994#discussion_r797800931
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergTimeTravel.java
##
@@ -49,8 +49,8 @@ public void testSelectAsOfTimestamp() throws IOException,
InterruptedException {
Assert.assertEquals(4, rows.size());
AssertHelpers.assertThrows("should throw exception",
IllegalArgumentException.class,
-"Cannot find a snapshot older than 1970-01-01 00:00:00", () -> {
- shell.executeStatement("SELECT * FROM customers FOR SYSTEM_TIME AS
OF '1970-01-01 00:00:00'");
+"java.lang.IllegalArgumentException: Cannot find a snapshot older than
1970-01-01", () -> {
+shell.executeStatement("SELECT * FROM customers FOR SYSTEM_TIME AS OF
'1970-01-01 00:00:00.000'");
Review comment:
Out of curiosity: does it not work without the `.000` part in the SQL
statement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 719484)
Time Spent: 20m (was: 10m)
> Upgrade Iceberg dependency to 0.13.0
>
>
> Key: HIVE-25922
> URL: https://issues.apache.org/jira/browse/HIVE-25922
> Project: Hive
> Issue Type: Improvement
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25922) Upgrade Iceberg dependency to 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25922: -- Labels: pull-request-available (was: ) > Upgrade Iceberg dependency to 0.13.0 > > > Key: HIVE-25922 > URL: https://issues.apache.org/jira/browse/HIVE-25922 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25922) Upgrade Iceberg dependency to 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-25922?focusedWorklogId=719481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719481 ] ASF GitHub Bot logged work on HIVE-25922: - Author: ASF GitHub Bot Created on: 02/Feb/22 16:35 Start Date: 02/Feb/22 16:35 Worklog Time Spent: 10m Work Description: marton-bod commented on a change in pull request #2994: URL: https://github.com/apache/hive/pull/2994#discussion_r797799573 ## File path: iceberg/iceberg-handler/pom.xml ## @@ -106,6 +106,12 @@ assertj-core test + +org.roaringbitmap Review comment: How come we need this here? I thought this would come transitively via iceberg-core -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 719481) Remaining Estimate: 0h Time Spent: 10m > Upgrade Iceberg dependency to 0.13.0 > > > Key: HIVE-25922 > URL: https://issues.apache.org/jira/browse/HIVE-25922 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25916) Optimise updateCompactionMetricsData
[
https://issues.apache.org/jira/browse/HIVE-25916?focusedWorklogId=719291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719291
]
ASF GitHub Bot logged work on HIVE-25916:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 10:59
Start Date: 02/Feb/22 10:59
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2989:
URL: https://github.com/apache/hive/pull/2989#discussion_r797489255
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -1584,38 +1584,17 @@ public boolean
updateCompactionMetricsData(CompactionMetricsData data) throws Me
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
boolean updateRes;
-CompactionMetricsData prevMetricsData = getCompactionMetricsData(data,
dbConn);
-if (prevMetricsData != null) {
- String query = UPDATE_COMPACTION_METRICS_CACHE_QUERY;
- if (data.getPartitionName() != null) {
-query += " AND \"CMC_PARTITION\" = ?";
+if (data.getMetricValue() >= data.getThreshold()) {
+ CompactionMetricsData prevMetricsData =
getCompactionMetricsData(data, dbConn);
+ if (prevMetricsData != null) {
+updateRes = updateCompactionMetricsData(dbConn, data,
prevMetricsData);
} else {
-query += " AND \"CMC_PARTITION\" IS NULL";
- }
- try (PreparedStatement pstmt = dbConn.prepareStatement(query)) {
-pstmt.setInt(1, data.getMetricValue());
-pstmt.setInt(2, prevMetricsData.getVersion() + 1);
-pstmt.setString(3, data.getDbName());
-pstmt.setString(4, data.getTblName());
-pstmt.setString(5, data.getMetricType().toString());
-pstmt.setInt(6, prevMetricsData.getVersion());
-if (data.getPartitionName() != null) {
- pstmt.setString(7, data.getPartitionName());
-}
-updateRes = pstmt.executeUpdate() > 0;
+updateRes = createCompactionMetricsData(dbConn, data);
}
} else {
- try (PreparedStatement pstmt =
dbConn.prepareStatement(INSERT_COMPACTION_METRICS_CACHE_QUERY)) {
-pstmt.setString(1, data.getDbName());
-pstmt.setString(2, data.getTblName());
-pstmt.setString(3, data.getPartitionName());
-pstmt.setString(4, data.getMetricType().toString());
-pstmt.setInt(5, data.getMetricValue());
-pstmt.setInt(6, 1);
-updateRes = pstmt.executeUpdate() > 0;
- }
+ updateRes = removeCompactionMetricsData(dbConn, data.getDbName(),
data.getTblName(), data.getPartitionName(),
Review comment:
Good point. I changed the logic to run remove in case the new metric is
below threshold and it is already present in the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 719291)
Time Spent: 0.5h (was: 20m)
> Optimise updateCompactionMetricsData
>
>
> Key: HIVE-25916
> URL: https://issues.apache.org/jira/browse/HIVE-25916
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TxnStore#updateCompactionMetricsData can be optimised to use the same DB
> connection during create/update/delete if the delta metric threshold
> evaluation is moved inside this method.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25887) Add external_table_concatenate.q to testconfiguration.properties
[ https://issues.apache.org/jira/browse/HIVE-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-25887. Resolution: Won't Fix Marking this as won't fix. > Add external_table_concatenate.q to testconfiguration.properties > > > Key: HIVE-25887 > URL: https://issues.apache.org/jira/browse/HIVE-25887 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > external_table_concatenate.q is not part of testconfiguration.properties. > Adding it to ensure that test runs in mini local llap driver. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HIVE-25887) Add external_table_concatenate.q to testconfiguration.properties
[ https://issues.apache.org/jira/browse/HIVE-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis reopened HIVE-25887: > Add external_table_concatenate.q to testconfiguration.properties > > > Key: HIVE-25887 > URL: https://issues.apache.org/jira/browse/HIVE-25887 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > external_table_concatenate.q is not part of testconfiguration.properties. > Adding it to ensure that test runs in mini local llap driver. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25887) Add external_table_concatenate.q to testconfiguration.properties
[ https://issues.apache.org/jira/browse/HIVE-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485713#comment-17485713 ] Stamatis Zampetakis commented on HIVE-25887: Reverted in https://github.com/apache/hive/commit/8a5be527737d57da79b659223e8f83cbec64ce54. > Add external_table_concatenate.q to testconfiguration.properties > > > Key: HIVE-25887 > URL: https://issues.apache.org/jira/browse/HIVE-25887 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > external_table_concatenate.q is not part of testconfiguration.properties. > Adding it to ensure that test runs in mini local llap driver. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-24283) Memory leak problem of hiveserver2 when compiling in parallel
[ https://issues.apache.org/jira/browse/HIVE-24283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485709#comment-17485709 ] Naveen Gangam commented on HIVE-24283: -- [~zhaolun7] would you still have the heap dumps from that run? If you do, please attach to this jira if it does not contain any sensitive info. > Memory leak problem of hiveserver2 when compiling in parallel > - > > Key: HIVE-24283 > URL: https://issues.apache.org/jira/browse/HIVE-24283 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 2.3.7 > Environment: CentOS 7.2 > openjdk 8 > Hadoop 2.9.2 > Hive 2.3.7 >Reporter: zhaolun7 >Priority: Critical > Fix For: 2.3.7 > > Attachments: image-2020-10-18-22-25-44-271.png, > image-2020-10-18-22-26-20-436.png > > Original Estimate: 504h > Remaining Estimate: 504h > > I used JDBC to connect to HIVESERVER2 and got about 25,000 SQL statements as > test data from the production environment to test parallel compilation. Then > save the memory snapshot of hiveserver2. Then run the test again and save the > memory snapshot. I found that the memory occupied became larger and the > second time running was more slower. > > This is the first time I have submitted an issue. If there is any incomplete > description, please point it out. If the format of the issue is incorrect, > please help me modify it, thank you > > !image-2020-10-18-22-26-20-436.png! > !image-2020-10-18-22-25-44-271.png! > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25887) Add external_table_concatenate.q to testconfiguration.properties
[ https://issues.apache.org/jira/browse/HIVE-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485708#comment-17485708 ] Stamatis Zampetakis commented on HIVE-25887: [~harishjp] No worries I just wanted to clarify the intention in case I was missing something. > Add external_table_concatenate.q to testconfiguration.properties > > > Key: HIVE-25887 > URL: https://issues.apache.org/jira/browse/HIVE-25887 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish JP >Assignee: Harish JP >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > external_table_concatenate.q is not part of testconfiguration.properties. > Adding it to ensure that test runs in mini local llap driver. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25851) Replace HiveRelMdPredicate with RelMdPredicate from Calcite
[ https://issues.apache.org/jira/browse/HIVE-25851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485707#comment-17485707 ] Stamatis Zampetakis commented on HIVE-25851: The one that came to my mind was CALCITE-2205 but yes its not the only one. > Replace HiveRelMdPredicate with RelMdPredicate from Calcite > --- > > Key: HIVE-25851 > URL: https://issues.apache.org/jira/browse/HIVE-25851 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Alessandro Solimando >Priority: Major > > `HiveRelMdPredicates` was copied from `RelMdPredicates` in Calcite long ago, > it has few differences which could be ported to the Calcite version, if > needed. > The goal of the ticket is to: > # ascertain which are the additional features in `HiveRelMdPredicates`, port > them to Calcite if needed > # drop `HiveRelMdPredicates` in favour of `RelMdPredicates` in order to > benefit from all the advances in such class -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25898) Compaction txn heartbeating after Worker timeout
[
https://issues.apache.org/jira/browse/HIVE-25898?focusedWorklogId=719271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719271
]
ASF GitHub Bot logged work on HIVE-25898:
-
Author: ASF GitHub Bot
Created on: 02/Feb/22 10:09
Start Date: 02/Feb/22 10:09
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2981:
URL: https://github.com/apache/hive/pull/2981#discussion_r797437477
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorUtil.java
##
@@ -88,4 +90,13 @@ static String getCompactorJobQueueName(HiveConf conf,
CompactionInfo ci, Table t
}
return null;
}
+
+ public static ThreadFactory createThreadFactory(String threadNameFormat) {
+return new ThreadFactoryBuilder()
+.setPriority(Thread.currentThread().getPriority())
+.setDaemon(Thread.currentThread().isDaemon())
Review comment:
Do we want the heartbeater to run with same options as the current
thread? From the code below it seems that before it was a daemon thread with
min priority.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -207,26 +206,16 @@ static void gatherStats(CompactionInfo ci, HiveConf conf,
String userName, Strin
}
}
- static final class CompactionHeartbeater extends Thread {
+ static final class CompactionHeartbeater implements Runnable {
static final private Logger LOG =
LoggerFactory.getLogger(CompactionHeartbeater.class);
private final CompactionTxn compactionTxn;
private final String tableName;
private final HiveConf conf;
-private final AtomicBoolean errorLogEnabled;
public CompactionHeartbeater(CompactionTxn compactionTxn, String
tableName, HiveConf conf) {
this.tableName = Objects.requireNonNull(tableName);
this.compactionTxn = Objects.requireNonNull(compactionTxn);
this.conf = Objects.requireNonNull(conf);
- this.errorLogEnabled = new AtomicBoolean(true);
-
- setDaemon(true);
- setPriority(MIN_PRIORITY);
- setName("CompactionHeartbeater-" + compactionTxn.getTxnId());
-}
-
-public void shouldLogError(boolean shouldLogError) {
- this.errorLogEnabled.set(shouldLogError);
Review comment:
Why it was necessary to have conditional error logging in HIVE-25740?
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
##
@@ -170,6 +174,73 @@ public void tearDown() {
}
}
+
+ @Test
+ public void testHeartbeatShutdownOnFailedCompaction() throws Exception {
+String dbName = "default";
+String tblName = "compaction_test";
+executeStatementOnDriver("drop table if exists " + tblName, driver);
+executeStatementOnDriver("CREATE TABLE " + tblName + "(a INT, b STRING) " +
+" PARTITIONED BY(bkt INT)" +
+" CLUSTERED BY(a) INTO 4 BUCKETS" + //currently ACID requires
table to be bucketed
+" STORED AS ORC TBLPROPERTIES ('transactional'='true')", driver);
+
+StrictDelimitedInputWriter writer = StrictDelimitedInputWriter.newBuilder()
+.withFieldDelimiter(',')
+.build();
+HiveStreamingConnection connection = HiveStreamingConnection.newBuilder()
+.withDatabase(dbName)
+.withTable(tblName)
+.withStaticPartitionValues(Arrays.asList("0"))
+.withAgentInfo("UT_" + Thread.currentThread().getName())
+.withHiveConf(conf)
+.withRecordWriter(writer)
+.connect();
+connection.beginTransaction();
+connection.write("55, 'London'".getBytes());
+connection.commitTransaction();
+connection.beginTransaction();
+connection.write("56, 'Paris'".getBytes());
+connection.commitTransaction();
+connection.close();
+
+executeStatementOnDriver("INSERT INTO TABLE " + tblName + "
PARTITION(bkt=1)" +
+" values(57, 'Budapest')", driver);
+executeStatementOnDriver("INSERT INTO TABLE " + tblName + "
PARTITION(bkt=1)" +
+" values(58, 'Milano')", driver);
+execSelectAndDumpData("select * from " + tblName, driver, "Dumping data
for " +
+tblName + " after load:");
+
+TxnStore txnHandler = TxnUtils.getTxnStore(conf);
+
+// Commit will throw an exception
+IMetaStoreClient mockedClient = Mockito.spy(new HiveMetaStoreClient(conf));
+doThrow(new RuntimeException("Simulating RuntimeException from
CompactionTxn.commit")).when(mockedClient).commitTxn(Mockito.anyLong());
+
+//Do a major compaction
+CompactionRequest rqst = new CompactionRequest(dbName, tblName,
CompactionType.MAJOR);
+rqst.setPartitionname("bkt=0");
+txnHandler.compact(rqst);
+
+Worker worker = Mockito.spy(new Worker());
+worker.setThreadId((int) worker.getId());
[jira] [Assigned] (HIVE-25922) Upgrade Iceberg dependency to 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita reassigned HIVE-25922: - > Upgrade Iceberg dependency to 0.13.0 > > > Key: HIVE-25922 > URL: https://issues.apache.org/jira/browse/HIVE-25922 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25922) Upgrade Iceberg dependency to 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25922 started by Ádám Szita. - > Upgrade Iceberg dependency to 0.13.0 > > > Key: HIVE-25922 > URL: https://issues.apache.org/jira/browse/HIVE-25922 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25898) Compaction txn heartbeating after Worker timeout
[ https://issues.apache.org/jira/browse/HIVE-25898?focusedWorklogId=719252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-719252 ] ASF GitHub Bot logged work on HIVE-25898: - Author: ASF GitHub Bot Created on: 02/Feb/22 08:32 Start Date: 02/Feb/22 08:32 Worklog Time Spent: 10m Work Description: deniskuzZ commented on a change in pull request #2981: URL: https://github.com/apache/hive/pull/2981#discussion_r797369739 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java ## @@ -22,6 +22,7 @@ import static org.apache.hadoop.hive.ql.TestTxnCommands2.runInitiator; import static org.apache.hadoop.hive.ql.TestTxnCommands2.runWorker; import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.*; Review comment: why wildcard import? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 719252) Time Spent: 2h (was: 1h 50m) > Compaction txn heartbeating after Worker timeout > > > Key: HIVE-25898 > URL: https://issues.apache.org/jira/browse/HIVE-25898 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > In some cases, when the compaction transaction is aborted, the hearbeater > thread is not shut down and keeps heartbeating. -- This message was sent by Atlassian Jira (v8.20.1#820001)
