[
https://issues.apache.org/jira/browse/HIVE-25659?focusedWorklogId=675771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675771
]
ASF GitHub Bot logged work on HIVE-25659:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Nov/21 01:27
Start Date: 04/Nov/21 01:27
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2758:
URL: https://github.com/apache/hive/pull/2758#discussion_r741678476
##########
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -680,6 +680,9 @@ public static ConfVars getMetaConf(String name) {
DIRECT_SQL_MAX_ELEMENTS_VALUES_CLAUSE("metastore.direct.sql.max.elements.values.clause",
"hive.direct.sql.max.elements.values.clause",
1000, "The maximum number of values in a VALUES clause for INSERT
statement."),
+ DIRECT_SQL_MAX_PARAMETERS("metastore.direct.sql.max.parameters",
+ "hive.direct.sql.max.parameters", 1000, "The maximum parameters the\n"
+
Review comment:
Rephrase: "... maximum query parameters..."
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
##########
@@ -316,7 +317,7 @@ public static String getFullTableName(String dbName, String
tableName) {
// Compute the size of a query when the 'nextValue' is added to the
current query.
int querySize = querySizeExpected(buf.length(), nextValue.length(),
suffix.length(), addParens);
- if (querySize > maxQueryLength * 1024) {
+ if (querySize > maxQueryLength * 1024 || currentCount >= maxParameters) {
Review comment:
nit: Keep each condition within ()
##########
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestTxnUtils.java
##########
@@ -150,12 +151,34 @@ public void testBuildQueryWithINClause() throws Exception
{
ret = TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
inList, "TXN_ID", false, false);
Assert.assertEquals(3, queries.size());
Assert.assertEquals(queries.size(), ret.size());
- Assert.assertEquals(2255L, ret.get(0).longValue());
- Assert.assertEquals(2033L, ret.get(1).longValue());
- Assert.assertEquals(33L, ret.get(2).longValue());
+ Assert.assertEquals(2000L, ret.get(0).longValue());
+ Assert.assertEquals(2000L, ret.get(1).longValue());
+ Assert.assertEquals(321L, ret.get(2).longValue());
runAgainstDerby(queries);
}
+ @Test(expected = IllegalArgumentException.class)
+ public void testBuildQueryWithNOTINClauseFailure() throws Exception {
Review comment:
Why would this test fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 675771)
Time Spent: 2h 20m (was: 2h 10m)
> Metastore direct sql queries with IN/(NOT IN) should be split based on max
> parameters allowed by SQL DB
> -------------------------------------------------------------------------------------------------------
>
> Key: HIVE-25659
> URL: https://issues.apache.org/jira/browse/HIVE-25659
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Affects Versions: 3.1.0, 4.0.0
> Reporter: Nikhil Gupta
> Assignee: Nikhil Gupta
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Function
> org.apache.hadoop.hive.metastore.txn.TxnUtils#buildQueryWithINClauseStrings
> can generate queries with huge number of parameters with very small value ofÂ
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE and DIRECT_SQL_MAX_QUERY_LENGTH while
> generating delete query for completed_compactions table
> Example:
> {code:java}
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE = 100
> DIRECT_SQL_MAX_QUERY_LENGTH = 10 (10 KB)
> Number of parameters in a single query = 4759
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
