[jira] [Work logged] (HIVE-25659) Divide IN/(NOT IN) queries based on number of max parameters SQL engine can support

Sun, 31 Oct 2021 21:16:56 -0700 


     [ 
https://issues.apache.org/jira/browse/HIVE-25659?focusedWorklogId=672532&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672532
 ]

ASF GitHub Bot logged work on HIVE-25659:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Nov/21 04:15
            Start Date: 01/Nov/21 04:15
    Worklog Time Spent: 10m 
      Work Description: guptanikhil007 commented on a change in pull request 
#2758:
URL: https://github.com/apache/hive/pull/2758#discussion_r739430648



##########
File path: 
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -680,6 +680,9 @@ public static ConfVars getMetaConf(String name) {
     
DIRECT_SQL_MAX_ELEMENTS_VALUES_CLAUSE("metastore.direct.sql.max.elements.values.clause",
         "hive.direct.sql.max.elements.values.clause",
         1000, "The maximum number of values in a VALUES clause for INSERT 
statement."),
+    DIRECT_SQL_MAX_PARAMETERS("metastore.direct.sql.max.parameters",
+        "hive.direct.sql.max.parameters", 1000, "The maximum parameters the\n" 
+

Review comment:
       MSSQL: 2100
   
   Oracle: 64,000. Source
   
   MySQL: The MySQL has a limit of 65,535 parameters (because num_params has to 
fit in two bytes).
   
   PostgreSql: 34464 for a query and 100 for a function
   
   SqlLite: 999 (SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999, but can be 
lowered at runtime) - And for functions default is 100 parameters.
   
   Source:
   
[StackOverflow](https://stackoverflow.com/questions/6581573/what-are-the-max-number-of-allowable-parameters-per-database-provider-type/49379324#:~:text=Thus%20the%20PostgreSQL%20protocol%20doesn,of%20which%20has%2065535%20parameters.)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Issue Time Tracking
-------------------

    Worklog Id:     (was: 672532)
    Time Spent: 50m  (was: 40m)

> Divide IN/(NOT IN) queries based on number of max parameters SQL engine can 
> support
> -----------------------------------------------------------------------------------
>
>                 Key: HIVE-25659
>                 URL: https://issues.apache.org/jira/browse/HIVE-25659
>             Project: Hive
>          Issue Type: Bug
>          Components: Standalone Metastore
>    Affects Versions: 3.1.0, 4.0.0
>            Reporter: Nikhil Gupta
>            Assignee: Nikhil Gupta
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> Function 
> org.apache.hadoop.hive.metastore.txn.TxnUtils#buildQueryWithINClauseStrings 
> can generate queries with huge number of parameters with very small value of 
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE and DIRECT_SQL_MAX_QUERY_LENGTH while 
> generating delete query for completed_compactions table
> Example:
> {code:java}
> DIRECT_SQL_MAX_ELEMENTS_IN_CLAUSE = 100
> DIRECT_SQL_MAX_QUERY_LENGTH = 10 (10 KB)
> Number of parameters in a single query = 4759
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to