[jira] [Work logged] (HIVE-21100) Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause
[ https://issues.apache.org/jira/browse/HIVE-21100?focusedWorklogId=705377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705377 ] ASF GitHub Bot logged work on HIVE-21100: - Author: ASF GitHub Bot Created on: 08/Jan/22 06:57 Start Date: 08/Jan/22 06:57 Worklog Time Spent: 10m Work Description: pvary edited a comment on pull request #2921: URL: https://github.com/apache/hive/pull/2921#issuecomment-1007898221 @hsnusonic: Does this work for ACID tables? If so, will this work with direct insert enabled? (I suspect not, as we are not moving files there) Is it possible to have a solution which could work even if the direct insert is enabled? Direct insert is a serious performance gain on S3, and it would be good to have a solution working for both ceses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705377) Time Spent: 40m (was: 0.5h) > Allow flattening of table subdirectories resulted when using TEZ engine and > UNION clause > > > Key: HIVE-21100 > URL: https://issues.apache.org/jira/browse/HIVE-21100 > Project: Hive > Issue Type: Improvement >Reporter: George Pachitariu >Assignee: George Pachitariu >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-21100.1.patch, HIVE-21100.2.patch, > HIVE-21100.3.patch, HIVE-21100.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Right now, when writing data into a table with Tez engine and the clause > UNION ALL is the last step of the query, Hive on Tez will create a > subdirectory for each branch of the UNION ALL. > With this patch the subdirectories are removed, and the files are renamed and > moved to the parent directory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-21100) Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause
[ https://issues.apache.org/jira/browse/HIVE-21100?focusedWorklogId=705376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705376 ] ASF GitHub Bot logged work on HIVE-21100: - Author: ASF GitHub Bot Created on: 08/Jan/22 06:53 Start Date: 08/Jan/22 06:53 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2921: URL: https://github.com/apache/hive/pull/2921#issuecomment-1007898221 @hsnusonic: will this work with direct insert enabled? (I suspect not, as we are not moving files there) Is it possible to have a solution which could work even if the direct insert is enabled? Direct insert is a serious performance gain on S3, and it would be good to have a solution working for both ceses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705376) Time Spent: 0.5h (was: 20m) > Allow flattening of table subdirectories resulted when using TEZ engine and > UNION clause > > > Key: HIVE-21100 > URL: https://issues.apache.org/jira/browse/HIVE-21100 > Project: Hive > Issue Type: Improvement >Reporter: George Pachitariu >Assignee: George Pachitariu >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-21100.1.patch, HIVE-21100.2.patch, > HIVE-21100.3.patch, HIVE-21100.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Right now, when writing data into a table with Tez engine and the clause > UNION ALL is the last step of the query, Hive on Tez will create a > subdirectory for each branch of the UNION ALL. > With this patch the subdirectories are removed, and the files are renamed and > moved to the parent directory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25824) Upgrade branch-2.3 to log4j 2.17.0
[ https://issues.apache.org/jira/browse/HIVE-25824?focusedWorklogId=705318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705318 ] ASF GitHub Bot logged work on HIVE-25824: - Author: ASF GitHub Bot Created on: 07/Jan/22 23:29 Start Date: 07/Jan/22 23:29 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #2908: URL: https://github.com/apache/hive/pull/2908#issuecomment-1007818622 Sorry for the late reply @elukey ! it seems the CI run failed due to timeout. Let me re-trigger it first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705318) Time Spent: 1h 10m (was: 1h) > Upgrade branch-2.3 to log4j 2.17.0 > -- > > Key: HIVE-25824 > URL: https://issues.apache.org/jira/browse/HIVE-25824 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: Luca Toscano >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Hi everybody, > I am wondering if there are any plans to upgrade branch-2.3 to log4j 2.17.0 > as it was done in HIVE-25795 (and related). > In Apache Bigtop we created https://github.com/apache/bigtop/pull/844, since > the one before the last release (Bigtop 1.5.0) shipped Hive 2.3.6. > I can try to file a pull request for branch-2.3 adapting what was done for > Bigtop (if the branch is still maintained), but I am currently experiencing > some mvn package failures (that seem unrelated to log4j) so I'd need some > help for you in case :) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25839) Upgrade master to log4j 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hankó Gergely updated HIVE-25839: - Attachment: HIVE-25839.1.patch Target Version/s: 4.0.0 Status: Patch Available (was: In Progress) > Upgrade master to log4j 2.17.1 > -- > > Key: HIVE-25839 > URL: https://issues.apache.org/jira/browse/HIVE-25839 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: weidong >Assignee: Hankó Gergely >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25839.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25839) Upgrade master to log4j 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25839 started by Hankó Gergely. > Upgrade master to log4j 2.17.1 > -- > > Key: HIVE-25839 > URL: https://issues.apache.org/jira/browse/HIVE-25839 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: weidong >Assignee: Hankó Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25839) Upgrade master to log4j 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hankó Gergely reassigned HIVE-25839: Assignee: Hankó Gergely > Upgrade master to log4j 2.17.1 > -- > > Key: HIVE-25839 > URL: https://issues.apache.org/jira/browse/HIVE-25839 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: weidong >Assignee: Hankó Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25839) Upgrade master to log4j 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25839?focusedWorklogId=705274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705274 ] ASF GitHub Bot logged work on HIVE-25839: - Author: ASF GitHub Bot Created on: 07/Jan/22 21:40 Start Date: 07/Jan/22 21:40 Worklog Time Spent: 10m Work Description: ghanko opened a new pull request #2930: URL: https://github.com/apache/hive/pull/2930 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705274) Remaining Estimate: 0h Time Spent: 10m > Upgrade master to log4j 2.17.1 > -- > > Key: HIVE-25839 > URL: https://issues.apache.org/jira/browse/HIVE-25839 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: weidong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25839) Upgrade master to log4j 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25839: -- Labels: pull-request-available (was: ) > Upgrade master to log4j 2.17.1 > -- > > Key: HIVE-25839 > URL: https://issues.apache.org/jira/browse/HIVE-25839 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: weidong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-21100) Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause
[ https://issues.apache.org/jira/browse/HIVE-21100?focusedWorklogId=705102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705102 ] ASF GitHub Bot logged work on HIVE-21100: - Author: ASF GitHub Bot Created on: 07/Jan/22 17:22 Start Date: 07/Jan/22 17:22 Worklog Time Spent: 10m Work Description: hsnusonic commented on pull request #2921: URL: https://github.com/apache/hive/pull/2921#issuecomment-1007588914 @zabetak @pvary Could you help to review this patch? Or recommend someone? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705102) Time Spent: 20m (was: 10m) > Allow flattening of table subdirectories resulted when using TEZ engine and > UNION clause > > > Key: HIVE-21100 > URL: https://issues.apache.org/jira/browse/HIVE-21100 > Project: Hive > Issue Type: Improvement >Reporter: George Pachitariu >Assignee: George Pachitariu >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-21100.1.patch, HIVE-21100.2.patch, > HIVE-21100.3.patch, HIVE-21100.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Right now, when writing data into a table with Tez engine and the clause > UNION ALL is the last step of the query, Hive on Tez will create a > subdirectory for each branch of the UNION ALL. > With this patch the subdirectories are removed, and the files are renamed and > moved to the parent directory. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25824) Upgrade branch-2.3 to log4j 2.17.0
[ https://issues.apache.org/jira/browse/HIVE-25824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470742#comment-17470742 ] Luca Toscano commented on HIVE-25824: - Found the issue, it was liblog4j-extras1.2-java (on Debian) added to the classpath by mistake. > Upgrade branch-2.3 to log4j 2.17.0 > -- > > Key: HIVE-25824 > URL: https://issues.apache.org/jira/browse/HIVE-25824 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: Luca Toscano >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Hi everybody, > I am wondering if there are any plans to upgrade branch-2.3 to log4j 2.17.0 > as it was done in HIVE-25795 (and related). > In Apache Bigtop we created https://github.com/apache/bigtop/pull/844, since > the one before the last release (Bigtop 1.5.0) shipped Hive 2.3.6. > I can try to file a pull request for branch-2.3 adapting what was done for > Bigtop (if the branch is still maintained), but I am currently experiencing > some mvn package failures (that seem unrelated to log4j) so I'd need some > help for you in case :) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25832) Exclude Category-X JDBC drivers from binary distribution
[
https://issues.apache.org/jira/browse/HIVE-25832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-25832.
Fix Version/s: 4.0.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/588e634308afbeb02b85e094b3d63529dacb7f6f.
Thanks for the reviews [~asolimando], [~kgyrtkirk]!
> Exclude Category-X JDBC drivers from binary distribution
>
>
> Key: HIVE-25832
> URL: https://issues.apache.org/jira/browse/HIVE-25832
> Project: Hive
> Issue Type: Task
> Components: distribution
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The binary distribution contains all the required elements to be able to run
> Hive in a cluster. It can be obtained by building from source using the
> following command:
> {code:java}
> mvn clean package -DskipTests -Pdist{code}
> The binary distribution is also published during a release along with the
> source code.
>
> In current master, commit 8572c1201e1d483eb03c7e413f4ff7f9b6f4a3d2, the
> binary distribution includes the following JDBC drivers:
> * derby-10.14.1.0.jar
> * postgresql-42.2.14.jar
> * ojdbc8-21.3.0.0.jar
> * mssql-jdbc-6.2.1.jre8.jar
> * mysql-connector-java-8.0.27.jar
> JDBC drivers are needed:
> * by schemaTool to initialize the database backend for the Metastore
> * by metastore to communicate with underlying database
> so if we want Hive to work out of the box we have to provide at least one.
> The Oracle (ojdbc8) and MySQL (mysql-connector-java) drivers must be removed
> cause their license is not compatible with Apache License 2 (see [category
> x|https://www.apache.org/legal/resolved.html#category-x]).
> Previous Hive releases (e.g., 3.1.2) are not affected since they only contain:
> * derby-10.14.1.0.jar
> * postgresql-9.4.1208.jre7.jar
> The additional drivers that appear in the binary distribution are a side
> effect of HIVE-25701.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25832) Exclude Category-X JDBC drivers from binary distribution
[
https://issues.apache.org/jira/browse/HIVE-25832?focusedWorklogId=705077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705077
]
ASF GitHub Bot logged work on HIVE-25832:
-
Author: ASF GitHub Bot
Created on: 07/Jan/22 15:56
Start Date: 07/Jan/22 15:56
Worklog Time Spent: 10m
Work Description: zabetak closed pull request #2905:
URL: https://github.com/apache/hive/pull/2905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 705077)
Time Spent: 20m (was: 10m)
> Exclude Category-X JDBC drivers from binary distribution
>
>
> Key: HIVE-25832
> URL: https://issues.apache.org/jira/browse/HIVE-25832
> Project: Hive
> Issue Type: Task
> Components: distribution
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The binary distribution contains all the required elements to be able to run
> Hive in a cluster. It can be obtained by building from source using the
> following command:
> {code:java}
> mvn clean package -DskipTests -Pdist{code}
> The binary distribution is also published during a release along with the
> source code.
>
> In current master, commit 8572c1201e1d483eb03c7e413f4ff7f9b6f4a3d2, the
> binary distribution includes the following JDBC drivers:
> * derby-10.14.1.0.jar
> * postgresql-42.2.14.jar
> * ojdbc8-21.3.0.0.jar
> * mssql-jdbc-6.2.1.jre8.jar
> * mysql-connector-java-8.0.27.jar
> JDBC drivers are needed:
> * by schemaTool to initialize the database backend for the Metastore
> * by metastore to communicate with underlying database
> so if we want Hive to work out of the box we have to provide at least one.
> The Oracle (ojdbc8) and MySQL (mysql-connector-java) drivers must be removed
> cause their license is not compatible with Apache License 2 (see [category
> x|https://www.apache.org/legal/resolved.html#category-x]).
> Previous Hive releases (e.g., 3.1.2) are not affected since they only contain:
> * derby-10.14.1.0.jar
> * postgresql-9.4.1208.jre7.jar
> The additional drivers that appear in the binary distribution are a side
> effect of HIVE-25701.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25849) Disable insert overwrite for bucket partitioned Iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-25849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod resolved HIVE-25849. --- Resolution: Fixed > Disable insert overwrite for bucket partitioned Iceberg tables > -- > > Key: HIVE-25849 > URL: https://issues.apache.org/jira/browse/HIVE-25849 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Insert overwrite should be disabled where the target Iceberg table is a > bucket partitioned table, since which existing partitions will be overwritten > is very hard to predict from a user's POV, as it depends on the bucket hash > values calculated for the new dataset's rows. It's better to be on the safe > side and disable this operation to avoid unwanted data loss. > Note: this the same approach followed by Impala too. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25849) Disable insert overwrite for bucket partitioned Iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-25849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470700#comment-17470700 ] Marton Bod commented on HIVE-25849: --- Pushed to master. Thanks [~szita] and [~pvary] for checking it. > Disable insert overwrite for bucket partitioned Iceberg tables > -- > > Key: HIVE-25849 > URL: https://issues.apache.org/jira/browse/HIVE-25849 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Insert overwrite should be disabled where the target Iceberg table is a > bucket partitioned table, since which existing partitions will be overwritten > is very hard to predict from a user's POV, as it depends on the bucket hash > values calculated for the new dataset's rows. It's better to be on the safe > side and disable this operation to avoid unwanted data loss. > Note: this the same approach followed by Impala too. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25849) Disable insert overwrite for bucket partitioned Iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-25849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25849: -- Labels: pull-request-available (was: ) > Disable insert overwrite for bucket partitioned Iceberg tables > -- > > Key: HIVE-25849 > URL: https://issues.apache.org/jira/browse/HIVE-25849 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Insert overwrite should be disabled where the target Iceberg table is a > bucket partitioned table, since which existing partitions will be overwritten > is very hard to predict from a user's POV, as it depends on the bucket hash > values calculated for the new dataset's rows. It's better to be on the safe > side and disable this operation to avoid unwanted data loss. > Note: this the same approach followed by Impala too. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25849) Disable insert overwrite for bucket partitioned Iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-25849?focusedWorklogId=705070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705070 ] ASF GitHub Bot logged work on HIVE-25849: - Author: ASF GitHub Bot Created on: 07/Jan/22 15:34 Start Date: 07/Jan/22 15:34 Worklog Time Spent: 10m Work Description: marton-bod merged pull request #2856: URL: https://github.com/apache/hive/pull/2856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705070) Remaining Estimate: 0h Time Spent: 10m > Disable insert overwrite for bucket partitioned Iceberg tables > -- > > Key: HIVE-25849 > URL: https://issues.apache.org/jira/browse/HIVE-25849 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Insert overwrite should be disabled where the target Iceberg table is a > bucket partitioned table, since which existing partitions will be overwritten > is very hard to predict from a user's POV, as it depends on the bucket hash > values calculated for the new dataset's rows. It's better to be on the safe > side and disable this operation to avoid unwanted data loss. > Note: this the same approach followed by Impala too. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25852) Introduce IN clauses at the very end of query planning
[ https://issues.apache.org/jira/browse/HIVE-25852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25852: -- Labels: pull-request-available (was: ) > Introduce IN clauses at the very end of query planning > -- > > Key: HIVE-25852 > URL: https://issues.apache.org/jira/browse/HIVE-25852 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 4.0.0 >Reporter: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Calcite "explodes" IN clauses into the equivalent OR form, and therefore it > does not handle such clauses in most of the codebase (notably in > _RexSimplify_). > In Hive, the same happens, but _HivePointLookupOptimizerRule_ re-introduces > IN clauses, and it happens in _applyPreJoinOrderingTransforms_ phase, which > is pretty early and which mixes several other rules which might not fully > support IN (notably, _HiveReduceExpressionsRule_ which is based on > _RexSimplify_). > The problem will become even harder in later versions of Calcite (current is > 1.25) based on SARG, which does not support IN clauses. > IN clauses can be converted into efficient runtime operators, we therefore > want to keep them in the final plan, intuitively we just want this > translation to happen in a later step, in order to leave the rest of the > codebase (Hive and Calcite) unaware of IN clauses. > The goal of the ticket is as follows: > # re-convert the output expression of _HivePointLookupOptimizerRule_ into the > OR form (keep the logic as-is to benefit from the rule) > # add a rule, in the last step of the planning process, that only converts > eligible OR expressions into IN clauses -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25852) Introduce IN clauses at the very end of query planning
[ https://issues.apache.org/jira/browse/HIVE-25852?focusedWorklogId=705062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705062 ] ASF GitHub Bot logged work on HIVE-25852: - Author: ASF GitHub Bot Created on: 07/Jan/22 14:56 Start Date: 07/Jan/22 14:56 Worklog Time Spent: 10m Work Description: asolimando opened a new pull request #2928: URL: https://github.com/apache/hive/pull/2928 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 705062) Remaining Estimate: 0h Time Spent: 10m > Introduce IN clauses at the very end of query planning > -- > > Key: HIVE-25852 > URL: https://issues.apache.org/jira/browse/HIVE-25852 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 4.0.0 >Reporter: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Calcite "explodes" IN clauses into the equivalent OR form, and therefore it > does not handle such clauses in most of the codebase (notably in > _RexSimplify_). > In Hive, the same happens, but _HivePointLookupOptimizerRule_ re-introduces > IN clauses, and it happens in _applyPreJoinOrderingTransforms_ phase, which > is pretty early and which mixes several other rules which might not fully > support IN (notably, _HiveReduceExpressionsRule_ which is based on > _RexSimplify_). > The problem will become even harder in later versions of Calcite (current is > 1.25) based on SARG, which does not support IN clauses. > IN clauses can be converted into efficient runtime operators, we therefore > want to keep them in the final plan, intuitively we just want this > translation to happen in a later step, in order to leave the rest of the > codebase (Hive and Calcite) unaware of IN clauses. > The goal of the ticket is as follows: > # re-convert the output expression of _HivePointLookupOptimizerRule_ into the > OR form (keep the logic as-is to benefit from the rule) > # add a rule, in the last step of the planning process, that only converts > eligible OR expressions into IN clauses -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25854) Port Iceberg Hive fix - ORC vectorization fails after split offsets are considered during split generation
[
https://issues.apache.org/jira/browse/HIVE-25854?focusedWorklogId=705019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705019
]
ASF GitHub Bot logged work on HIVE-25854:
-
Author: ASF GitHub Bot
Created on: 07/Jan/22 13:05
Start Date: 07/Jan/22 13:05
Worklog Time Spent: 10m
Work Description: szlta opened a new pull request #2927:
URL: https://github.com/apache/hive/pull/2927
Porting Iceberg commit:
https://github.com/apache/iceberg/commit/339866d72784a3e90625037d3a9de4fdafc5fcef
Hive: ORC vectorization fails when split offsets are considered during split
generation (#3748
.. to fix issue with reading non-first splits of ORC files in vectorized mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 705019)
Remaining Estimate: 0h
Time Spent: 10m
> Port Iceberg Hive fix - ORC vectorization fails after split offsets are
> considered during split generation
> --
>
> Key: HIVE-25854
> URL: https://issues.apache.org/jira/browse/HIVE-25854
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Porting Iceberg commit:
> [https://github.com/apache/iceberg/commit/339866d72784a3e90625037d3a9de4fdafc5fcef]
> {code:java}
> Hive: ORC vectorization fails when split offsets are considered during split
> generation (#3748 {code}
> .. to fix issue with reading non-first splits of ORC files in vectorized mode
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25854) Port Iceberg Hive fix - ORC vectorization fails after split offsets are considered during split generation
[
https://issues.apache.org/jira/browse/HIVE-25854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25854:
--
Labels: pull-request-available (was: )
> Port Iceberg Hive fix - ORC vectorization fails after split offsets are
> considered during split generation
> --
>
> Key: HIVE-25854
> URL: https://issues.apache.org/jira/browse/HIVE-25854
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Porting Iceberg commit:
> [https://github.com/apache/iceberg/commit/339866d72784a3e90625037d3a9de4fdafc5fcef]
> {code:java}
> Hive: ORC vectorization fails when split offsets are considered during split
> generation (#3748 {code}
> .. to fix issue with reading non-first splits of ORC files in vectorized mode
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25854) Port Iceberg Hive fix - ORC vectorization fails after split offsets are considered during split generation
[
https://issues.apache.org/jira/browse/HIVE-25854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-25854:
--
Description:
Porting Iceberg commit:
[https://github.com/apache/iceberg/commit/339866d72784a3e90625037d3a9de4fdafc5fcef]
{code:java}
Hive: ORC vectorization fails when split offsets are considered during split
generation (#3748 {code}
.. to fix issue with reading non-first splits of ORC files in vectorized mode
was:
Porting Iceberg commit:
{code:java}
339866d72784a3e90625037d3a9de4fdafc5fcef
Hive: ORC vectorization fails when split offsets are considered during split
generation (#3748 {code}
.. to fix issue with reading non-first splits of ORC files in vectorized mode
> Port Iceberg Hive fix - ORC vectorization fails after split offsets are
> considered during split generation
> --
>
> Key: HIVE-25854
> URL: https://issues.apache.org/jira/browse/HIVE-25854
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> Porting Iceberg commit:
> [https://github.com/apache/iceberg/commit/339866d72784a3e90625037d3a9de4fdafc5fcef]
> {code:java}
> Hive: ORC vectorization fails when split offsets are considered during split
> generation (#3748 {code}
> .. to fix issue with reading non-first splits of ORC files in vectorized mode
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25854) Port Iceberg Hive fix - ORC vectorization fails after split offsets are considered during split generation
[
https://issues.apache.org/jira/browse/HIVE-25854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita reassigned HIVE-25854:
-
> Port Iceberg Hive fix - ORC vectorization fails after split offsets are
> considered during split generation
> --
>
> Key: HIVE-25854
> URL: https://issues.apache.org/jira/browse/HIVE-25854
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> Porting Iceberg commit:
> {code:java}
> 339866d72784a3e90625037d3a9de4fdafc5fcef
> Hive: ORC vectorization fails when split offsets are considered during split
> generation (#3748 {code}
> .. to fix issue with reading non-first splits of ORC files in vectorized mode
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25853) Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GuangMing Lu updated HIVE-25853: Summary: Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1 (was: Security Vulnerability CVE-2021-44832 log4j2 upgrade to 2.17.1) > Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1 > --- > > Key: HIVE-25853 > URL: https://issues.apache.org/jira/browse/HIVE-25853 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: GuangMing Lu >Priority: Major > > Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25853) Security Vulnerability CVE-2021-44832 log4j2 upgrade to 2.17.1
[ https://issues.apache.org/jira/browse/HIVE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GuangMing Lu updated HIVE-25853: Description: Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1 (was: Security Vulnerability CVE-2021-44832 log4j2 upgrade to 2.17.1) > Security Vulnerability CVE-2021-44832 log4j2 upgrade to 2.17.1 > -- > > Key: HIVE-25853 > URL: https://issues.apache.org/jira/browse/HIVE-25853 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: GuangMing Lu >Priority: Major > > Security Vulnerability CVE-2021-44832 log4j2 need upgrade to 2.17.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25275) OOM during query planning due to HiveJoinPushTransitivePredicatesRule matching infinitely
[
https://issues.apache.org/jira/browse/HIVE-25275?focusedWorklogId=704991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704991
]
ASF GitHub Bot logged work on HIVE-25275:
-
Author: ASF GitHub Bot
Created on: 07/Jan/22 10:55
Start Date: 07/Jan/22 10:55
Worklog Time Spent: 10m
Work Description: asolimando closed pull request #2925:
URL: https://github.com/apache/hive/pull/2925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704991)
Time Spent: 1h (was: 50m)
> OOM during query planning due to HiveJoinPushTransitivePredicatesRule
> matching infinitely
> -
>
> Key: HIVE-25275
> URL: https://issues.apache.org/jira/browse/HIVE-25275
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While running the following query OOM is raised during the planning phase
> {code:sql}
> CREATE TABLE A (`value_date` date) STORED AS ORC;
> CREATE TABLE B (`business_date` date) STORED AS ORC;
> SELECT A.VALUE_DATE
> FROM A, B
> WHERE A.VALUE_DATE = BUSINESS_DATE
> AND A.VALUE_DATE = TRUNC(BUSINESS_DATE, 'MONTH');
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25275) OOM during query planning due to HiveJoinPushTransitivePredicatesRule matching infinitely
[
https://issues.apache.org/jira/browse/HIVE-25275?focusedWorklogId=704990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704990
]
ASF GitHub Bot logged work on HIVE-25275:
-
Author: ASF GitHub Bot
Created on: 07/Jan/22 10:54
Start Date: 07/Jan/22 10:54
Worklog Time Spent: 10m
Work Description: asolimando closed pull request #2923:
URL: https://github.com/apache/hive/pull/2923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704990)
Time Spent: 50m (was: 40m)
> OOM during query planning due to HiveJoinPushTransitivePredicatesRule
> matching infinitely
> -
>
> Key: HIVE-25275
> URL: https://issues.apache.org/jira/browse/HIVE-25275
> Project: Hive
> Issue Type: Bug
>Reporter: László Pintér
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> While running the following query OOM is raised during the planning phase
> {code:sql}
> CREATE TABLE A (`value_date` date) STORED AS ORC;
> CREATE TABLE B (`business_date` date) STORED AS ORC;
> SELECT A.VALUE_DATE
> FROM A, B
> WHERE A.VALUE_DATE = BUSINESS_DATE
> AND A.VALUE_DATE = TRUNC(BUSINESS_DATE, 'MONTH');
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24367) Explore whether HiveAlterHandler::alterTable can be optimised for non-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-24367?focusedWorklogId=704987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704987
]
ASF GitHub Bot logged work on HIVE-24367:
-
Author: ASF GitHub Bot
Created on: 07/Jan/22 10:38
Start Date: 07/Jan/22 10:38
Worklog Time Spent: 10m
Work Description: abstractdog commented on pull request #2771:
URL: https://github.com/apache/hive/pull/2771#issuecomment-1007307171
@HarshitGupta11: please provide a description about what's actually
implemented here, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704987)
Time Spent: 20m (was: 10m)
> Explore whether HiveAlterHandler::alterTable can be optimised for
> non-partitioned tables
>
>
> Key: HIVE-24367
> URL: https://issues.apache.org/jira/browse/HIVE-24367
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {color:#22}Writing lots of delta in non-partitioned table creates runtime
> issues, when lot of delta folders are present.{color}
> {color:#22} {color}
> {color:#22}Following code in HiveAlterHandler is invoked for every insert
> operation. It computes {{{color}
> {color:#22}updateTableStatsSlow}} for every insert causing runtime
> delays.{color}
> {color:#22} {color}
> {noformat}
> if (MetaStoreUtils.requireCalStats(null, null, newt, environmentContext) &&
> !isPartitionedTable) {
> Database db = msdb.getDatabase(catName, newDbName);
> assert(isReplicated == HiveMetaStore.HMSHandler.isDbReplicationTarget(db));
> // Update table stats. For partitioned table, we update stats in
> alterPartition()
> MetaStoreUtils.updateTableStatsSlow(db, newt, wh, false, true,
> environmentContext);
> }
> {noformat}
> {color:#22}It would be good to explore whether only the newly added delta
> can be listed for computing stats. This would avoid huge listing call during
> stats collection.{color}
> {color:#22}e.g queries to repro{color}
> {noformat}
> CREATE TABLE IF NOT EXISTS test (name String, value int);
> INSERT INTO test VALUES('K1',1);
> INSERT INTO test VALUES('K2',2);
> ..
> ..
> ..
> INSERT INTO test VALUES('K2',2)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25824) Upgrade branch-2.3 to log4j 2.17.0
[ https://issues.apache.org/jira/browse/HIVE-25824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470505#comment-17470505 ] Luca Toscano commented on HIVE-25824: - [~csun] Hi! If you have time to help: for some reason I end up in a a no-class-def issue with the new patch (while testing 2.3.6 in some environments) - https://issues.apache.org/jira/browse/BIGTOP-3619 I am still not able to reproduce with the Apache Bigtop's testing Docker images, any suggestion? I checked the datanucleus jars, they are there and they contain the .class files needed (I also checked that they are included in the Hive metastore's classpath inspecting the STARTUP_MSG log). > Upgrade branch-2.3 to log4j 2.17.0 > -- > > Key: HIVE-25824 > URL: https://issues.apache.org/jira/browse/HIVE-25824 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.8 >Reporter: Luca Toscano >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Hi everybody, > I am wondering if there are any plans to upgrade branch-2.3 to log4j 2.17.0 > as it was done in HIVE-25795 (and related). > In Apache Bigtop we created https://github.com/apache/bigtop/pull/844, since > the one before the last release (Bigtop 1.5.0) shipped Hive 2.3.6. > I can try to file a pull request for branch-2.3 adapting what was done for > Bigtop (if the branch is still maintained), but I am currently experiencing > some mvn package failures (that seem unrelated to log4j) so I'd need some > help for you in case :) -- This message was sent by Atlassian Jira (v8.20.1#820001)
