[jira] [Reopened] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PJ Fanning reopened DRILL-8460: --- Assignee: (was: PJ Fanning) This is not fixed. The CI build had some test failures that indicate that we may nor be able to upgrade. > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8457) Allow configuring csv parser in http storage plugin configuration
[ https://issues.apache.org/jira/browse/DRILL-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zbigniew Tomanek closed DRILL-8457. --- Fix Version/s: 1.22.0 (was: Future) Reviewer: Charles Givre Resolution: Implemented > Allow configuring csv parser in http storage plugin configuration > - > > Key: DRILL-8457 > URL: https://issues.apache.org/jira/browse/DRILL-8457 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - HTTP >Affects Versions: Future >Reporter: Zbigniew Tomanek >Priority: Minor > Fix For: 1.22.0 > > > Currently there is no way to configure csv parser when http plugin is used. > Because of that some kind of files cannot be parsed (e.g. when any column has > more than 4096 chars or file has a delimiter different from `,`). > Since in DataWalk we utilize http plugin quite often we've changed our > internal fork of Drill so following parser/format properties can be > configured using additional `csvOptions` field: > > {code:json} > { > "csvOptions": { > "delimiter": "\t", > "quote": "\"", > "quote_escape": "\"", > "line_separator": "\n", > "header_extraction_enabled": null, > "number_of_rows_to_skip": 0, > "number_of_records_to_read": -1, > "line_separator_detection_enabled": true, > "max_columns": 512, > "max_chars_per_column": 4096, > "skip_empty_lines": true, > "ignore_leading_whitespaces": true, > "ignore_trailing_whitespaces": true, > "null_value": null > } > }{code} > I'd be glad to get feedback whether creating PR with these changes would > bring any value to the Drill -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8457) Allow configuring csv parser in http storage plugin configuration
[ https://issues.apache.org/jira/browse/DRILL-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781429#comment-17781429 ] ASF GitHub Bot commented on DRILL-8457: --- cgivre merged PR #2840: URL: https://github.com/apache/drill/pull/2840 > Allow configuring csv parser in http storage plugin configuration > - > > Key: DRILL-8457 > URL: https://issues.apache.org/jira/browse/DRILL-8457 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - HTTP >Affects Versions: Future >Reporter: Zbigniew Tomanek >Priority: Minor > Fix For: Future > > > Currently there is no way to configure csv parser when http plugin is used. > Because of that some kind of files cannot be parsed (e.g. when any column has > more than 4096 chars or file has a delimiter different from `,`). > Since in DataWalk we utilize http plugin quite often we've changed our > internal fork of Drill so following parser/format properties can be > configured using additional `csvOptions` field: > > {code:json} > { > "csvOptions": { > "delimiter": "\t", > "quote": "\"", > "quote_escape": "\"", > "line_separator": "\n", > "header_extraction_enabled": null, > "number_of_rows_to_skip": 0, > "number_of_records_to_read": -1, > "line_separator_detection_enabled": true, > "max_columns": 512, > "max_chars_per_column": 4096, > "skip_empty_lines": true, > "ignore_leading_whitespaces": true, > "ignore_trailing_whitespaces": true, > "null_value": null > } > }{code} > I'd be glad to get feedback whether creating PR with these changes would > bring any value to the Drill -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8446) Incorrect use of OperatingSystemMXBean
[ https://issues.apache.org/jira/browse/DRILL-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8446: --- Assignee: James Turton > Incorrect use of OperatingSystemMXBean > -- > > Key: DRILL-8446 > URL: https://issues.apache.org/jira/browse/DRILL-8446 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Mahmoud Ouali Alami >Assignee: James Turton >Priority: Major > Attachments: image-2023-07-04-15-36-42-905.png, > image-2023-07-04-16-24-59-662.png > > > *Context :* > In Drill "CpuGaugeSet" class, we use an internal class instead of a public > class : com.sun.management.OperatingSystemMXBean; > !image-2023-07-04-15-36-42-905.png|width=387,height=257! > This can result to a NoClassDefFoundError: > !image-2023-07-04-16-24-59-662.png|width=845,height=108! > *To do :* > Replace the private class "com.sun.managemenet.OperatingSystemMXBean" with > "java.lang.management.OperatingSystemMXBean", > > Kind regards, > Mahmoud > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8447) Jetty TLS support broken for server certificate with multiple SANs
[ https://issues.apache.org/jira/browse/DRILL-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781421#comment-17781421 ] James Turton commented on DRILL-8447: - Hi, in DRILL-8452 we did a lot of library uprgades including, by chance, the changes you mention here. Can you run a test using a snapshot of the Drill master branch? I can also upload a snapshot build for you if that would be helpful. > Jetty TLS support broken for server certificate with multiple SANs > -- > > Key: DRILL-8447 > URL: https://issues.apache.org/jira/browse/DRILL-8447 > Project: Apache Drill > Issue Type: Bug > Components: Server, Web Server >Affects Versions: 1.21.1 >Reporter: Lev C >Assignee: James Turton >Priority: Minor > > The update of Jetty dependency to version 9.4 (DRILL-7135) has led to Drill > no longer supporting a TLS server certificate with multiple Subject Alternate > Name (SAN) values. > If you try to use a keystore containing a single cert with multiple SANs, you > get the following error on startup of the Drillbit: > > {code:java} > [main] ERROR o.apache.drill.exec.server.Drillbit - Failure during initial > startup of Drillbit. > java.lang.illegalStateException: KeyStores with multiple certificates are not > supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. > (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or > org.eclipse.jetty.util.SslContextFactory$Client instead){code} > It appears that Jetty version 9.4 has deprecated the > org.eclipse.jetty.util.ssl.SslContextFactory class, and Drill should now use > the org.eclipse.jetty.util.ssl.SslContextFactory.Server class instead. > > I was unable to find a Drill configuration file to change which class is used > (eg an instance of jetty-ssl.xml), and it looks like the specific > SslContextFactory class is hardcoded in > org.apache.drill.exec.server.rest.ssl.SslContextFactoryConfigurator: > {code:java} > public SslContextFactory configureNewSslContextFactory() throws Exception { > > SSLConfig sslConf = new SSLConfigBuilder() > .config(config) > .mode(SSLConfig.Mode.SERVER) > .initializeSSLContext(false) > .validateKeyStore(true) > .build(); > final SslContextFactory sslContextFactory = new SslContextFactory(); > if (sslConf.isSslValid()) { > useOptionsConfiguredByUser(sslContextFactory, sslConf); > } else { > useAutoGeneratedSelfSignedCertificate(sslContextFactory); > } > return sslContextFactory; > } {code} > Relevant links for other products affected by the same issue: > https://issues.apache.org/jira/browse/NIFI-7730 > [https://opennms.discourse.group/t/jetty-with-ssl-throws-error-with-keystores-with-multiple-certificates-are-not-supported/1489] > [https://community.microfocus.com/adtd/sws-qc/f/itrc-895/513861/java-lang-illegalstateexception-keystores-with-multiple-certificates-are-not-supported-on-the-base-class-org-eclipse-jetty-util-ssl-sslcontextfactory] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8447) Jetty TLS support broken for server certificate with multiple SANs
[ https://issues.apache.org/jira/browse/DRILL-8447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8447: --- Assignee: James Turton > Jetty TLS support broken for server certificate with multiple SANs > -- > > Key: DRILL-8447 > URL: https://issues.apache.org/jira/browse/DRILL-8447 > Project: Apache Drill > Issue Type: Bug > Components: Server, Web Server >Affects Versions: 1.21.1 >Reporter: Lev C >Assignee: James Turton >Priority: Minor > > The update of Jetty dependency to version 9.4 (DRILL-7135) has led to Drill > no longer supporting a TLS server certificate with multiple Subject Alternate > Name (SAN) values. > If you try to use a keystore containing a single cert with multiple SANs, you > get the following error on startup of the Drillbit: > > {code:java} > [main] ERROR o.apache.drill.exec.server.Drillbit - Failure during initial > startup of Drillbit. > java.lang.illegalStateException: KeyStores with multiple certificates are not > supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. > (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or > org.eclipse.jetty.util.SslContextFactory$Client instead){code} > It appears that Jetty version 9.4 has deprecated the > org.eclipse.jetty.util.ssl.SslContextFactory class, and Drill should now use > the org.eclipse.jetty.util.ssl.SslContextFactory.Server class instead. > > I was unable to find a Drill configuration file to change which class is used > (eg an instance of jetty-ssl.xml), and it looks like the specific > SslContextFactory class is hardcoded in > org.apache.drill.exec.server.rest.ssl.SslContextFactoryConfigurator: > {code:java} > public SslContextFactory configureNewSslContextFactory() throws Exception { > > SSLConfig sslConf = new SSLConfigBuilder() > .config(config) > .mode(SSLConfig.Mode.SERVER) > .initializeSSLContext(false) > .validateKeyStore(true) > .build(); > final SslContextFactory sslContextFactory = new SslContextFactory(); > if (sslConf.isSslValid()) { > useOptionsConfiguredByUser(sslContextFactory, sslConf); > } else { > useAutoGeneratedSelfSignedCertificate(sslContextFactory); > } > return sslContextFactory; > } {code} > Relevant links for other products affected by the same issue: > https://issues.apache.org/jira/browse/NIFI-7730 > [https://opennms.discourse.group/t/jetty-with-ssl-throws-error-with-keystores-with-multiple-certificates-are-not-supported/1489] > [https://community.microfocus.com/adtd/sws-qc/f/itrc-895/513861/java-lang-illegalstateexception-keystores-with-multiple-certificates-are-not-supported-on-the-base-class-org-eclipse-jetty-util-ssl-sslcontextfactory] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8450. --- Resolution: Fixed > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8452: Fix Version/s: 1.22.0 (was: 1.21.2) > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8453. --- Resolution: Fixed > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8452. --- Resolution: Fixed > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8453: Fix Version/s: 1.22.0 (was: 1.21.2) > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8456. --- Resolution: Fixed > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8456: - > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8454. --- Resolution: Fixed > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8456: Affects Version/s: 1.21.1 > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8456: Fix Version/s: 1.22.0 > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8459: Affects Version/s: 1.21.1 > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Affects Version/s: 1.21.1 > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8459: Fix Version/s: 1.22.0 > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8460. --- Resolution: Fixed > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8460: - > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Fix Version/s: 1.22.0 > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8459. --- Resolution: Fixed > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8459: - > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8459: Summary: Bump avro to 1.11.3 due to CVE (was: Bump avro to 1.11.3 due to cve) > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8459. --- Assignee: PJ Fanning Resolution: Fixed > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8460. --- Assignee: PJ Fanning Resolution: Fixed > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Bump zookeeper jar to 3.7.2 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Bump zookeeper jar to 3.7.2 due to CVE (was: bump zookeeper jar to 3.7.2 due to cve) > Bump zookeeper jar to 3.7.2 due to CVE > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8459) Bump avro to 1.11.3 due to cve
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8459: Summary: Bump avro to 1.11.3 due to cve (was: bump avro to 1.11.3 due to cve) > Bump avro to 1.11.3 due to cve > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8456: --- Assignee: PJ Fanning > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8456: Summary: Upgrade POI to 5.2.4 (was: uptake POI 5.2.4) > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8456. --- Resolution: Fixed > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8436. --- > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton resolved DRILL-8436. - Resolution: Fixed > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[ https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8458. --- > Reading Parquet v2 data page with repetition levels larger than column data > throws IllegalArgumentException > --- > > Key: DRILL-8458 > URL: https://issues.apache.org/jira/browse/DRILL-8458 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.21.1 >Reporter: Peter Franzen >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > > When the size of the repetition level bytes in a Parquet v2 data page is > larger than the size of the column data bytes, > {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}} > throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set > the limit of a ByteBuffer to a value large than its capacity. > > The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}: > > {code:java} > 217 int pageBufOffset = 0; > 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 219 BytesInput repLevelBytes = BytesInput.from( > 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize) > 221 ); > 222 pageBufOffset += repLevelSize; > 223 > 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 225 final BytesInput defLevelBytes = BytesInput.from( > 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize) > 227 ); > 228 pageBufOffset += defLevelSize; {code} > > The buffer {{pageBuf}} contains the repetition level bytes followed by the > definition level bytes followed by the column data bytes. > > The code at lines 217-221 reads the repetition level bytes, and then updates > the position of the {{pageBuf}} buffer to the start of the definition level > bytes (lines 222 and 224). > > The code at lines 225-227 reads the definition level bytes, and when creating > a slice of the \{{pageBuf }}buffer containing the definition level bytes, the > slice's limit is set as if the position was at the beginning of the > repetition level bytes (line 226), i.e as if it not had been updated. > > This means that if the capacity of the pageBuf buffer (which is the size of > the repetition level bytes + the size of the definition level bytes + the > size of the column data bytes) is less than (repLevelSize + repLevelSize + > defLevelSize), the call to limit() will throw. > > The fix is to change line 226 to > {code:java} > (ByteBuffer) bb.slice().limit(defLevelSize){code} > > For symmetry, line 220 could also be changed to > {code:java} > (ByteBuffer) bb.slice().limit(repLevelSize){code} > > although {{pageBufOffset}} is always 0 there and will not cause the limit to > exceed the capacity. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[ https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton resolved DRILL-8458. - Resolution: Fixed > Reading Parquet v2 data page with repetition levels larger than column data > throws IllegalArgumentException > --- > > Key: DRILL-8458 > URL: https://issues.apache.org/jira/browse/DRILL-8458 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.21.1 >Reporter: Peter Franzen >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > > When the size of the repetition level bytes in a Parquet v2 data page is > larger than the size of the column data bytes, > {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}} > throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set > the limit of a ByteBuffer to a value large than its capacity. > > The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}: > > {code:java} > 217 int pageBufOffset = 0; > 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 219 BytesInput repLevelBytes = BytesInput.from( > 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize) > 221 ); > 222 pageBufOffset += repLevelSize; > 223 > 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 225 final BytesInput defLevelBytes = BytesInput.from( > 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize) > 227 ); > 228 pageBufOffset += defLevelSize; {code} > > The buffer {{pageBuf}} contains the repetition level bytes followed by the > definition level bytes followed by the column data bytes. > > The code at lines 217-221 reads the repetition level bytes, and then updates > the position of the {{pageBuf}} buffer to the start of the definition level > bytes (lines 222 and 224). > > The code at lines 225-227 reads the definition level bytes, and when creating > a slice of the \{{pageBuf }}buffer containing the definition level bytes, the > slice's limit is set as if the position was at the beginning of the > repetition level bytes (line 226), i.e as if it not had been updated. > > This means that if the capacity of the pageBuf buffer (which is the size of > the repetition level bytes + the size of the definition level bytes + the > size of the column data bytes) is less than (repLevelSize + repLevelSize + > defLevelSize), the call to limit() will throw. > > The fix is to change line 226 to > {code:java} > (ByteBuffer) bb.slice().limit(defLevelSize){code} > > For symmetry, line 220 could also be changed to > {code:java} > (ByteBuffer) bb.slice().limit(repLevelSize){code} > > although {{pageBufOffset}} is always 0 there and will not cause the limit to > exceed the capacity. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[ https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781412#comment-17781412 ] ASF GitHub Bot commented on DRILL-8458: --- jnturton merged PR #2838: URL: https://github.com/apache/drill/pull/2838 > Reading Parquet v2 data page with repetition levels larger than column data > throws IllegalArgumentException > --- > > Key: DRILL-8458 > URL: https://issues.apache.org/jira/browse/DRILL-8458 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.21.1 >Reporter: Peter Franzen >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > > When the size of the repetition level bytes in a Parquet v2 data page is > larger than the size of the column data bytes, > {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}} > throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set > the limit of a ByteBuffer to a value large than its capacity. > > The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}: > > {code:java} > 217 int pageBufOffset = 0; > 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 219 BytesInput repLevelBytes = BytesInput.from( > 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize) > 221 ); > 222 pageBufOffset += repLevelSize; > 223 > 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 225 final BytesInput defLevelBytes = BytesInput.from( > 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize) > 227 ); > 228 pageBufOffset += defLevelSize; {code} > > The buffer {{pageBuf}} contains the repetition level bytes followed by the > definition level bytes followed by the column data bytes. > > The code at lines 217-221 reads the repetition level bytes, and then updates > the position of the {{pageBuf}} buffer to the start of the definition level > bytes (lines 222 and 224). > > The code at lines 225-227 reads the definition level bytes, and when creating > a slice of the \{{pageBuf }}buffer containing the definition level bytes, the > slice's limit is set as if the position was at the beginning of the > repetition level bytes (line 226), i.e as if it not had been updated. > > This means that if the capacity of the pageBuf buffer (which is the size of > the repetition level bytes + the size of the definition level bytes + the > size of the column data bytes) is less than (repLevelSize + repLevelSize + > defLevelSize), the call to limit() will throw. > > The fix is to change line 226 to > {code:java} > (ByteBuffer) bb.slice().limit(defLevelSize){code} > > For symmetry, line 220 could also be changed to > {code:java} > (ByteBuffer) bb.slice().limit(repLevelSize){code} > > although {{pageBufOffset}} is always 0 there and will not cause the limit to > exceed the capacity. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8460) bump zookeeper jar to 3.7.2 due to cve
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781264#comment-17781264 ] ASF GitHub Bot commented on DRILL-8460: --- pjfanning commented on PR #2842: URL: https://github.com/apache/drill/pull/2842#issuecomment-1786721132 looks like the zookeeper lib upgrade breaks tests in the Phoenix module when default Hadoop is used. Hadoop 2 tests pass. > bump zookeeper jar to 3.7.2 due to cve > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[ https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781231#comment-17781231 ] ASF GitHub Bot commented on DRILL-8458: --- handmadecode commented on PR #2838: URL: https://github.com/apache/drill/pull/2838#issuecomment-1786560246 PR updated with refactored test code > Reading Parquet v2 data page with repetition levels larger than column data > throws IllegalArgumentException > --- > > Key: DRILL-8458 > URL: https://issues.apache.org/jira/browse/DRILL-8458 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.21.1 >Reporter: Peter Franzen >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > > When the size of the repetition level bytes in a Parquet v2 data page is > larger than the size of the column data bytes, > {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}} > throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set > the limit of a ByteBuffer to a value large than its capacity. > > The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}: > > {code:java} > 217 int pageBufOffset = 0; > 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 219 BytesInput repLevelBytes = BytesInput.from( > 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize) > 221 ); > 222 pageBufOffset += repLevelSize; > 223 > 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset); > 225 final BytesInput defLevelBytes = BytesInput.from( > 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize) > 227 ); > 228 pageBufOffset += defLevelSize; {code} > > The buffer {{pageBuf}} contains the repetition level bytes followed by the > definition level bytes followed by the column data bytes. > > The code at lines 217-221 reads the repetition level bytes, and then updates > the position of the {{pageBuf}} buffer to the start of the definition level > bytes (lines 222 and 224). > > The code at lines 225-227 reads the definition level bytes, and when creating > a slice of the \{{pageBuf }}buffer containing the definition level bytes, the > slice's limit is set as if the position was at the beginning of the > repetition level bytes (line 226), i.e as if it not had been updated. > > This means that if the capacity of the pageBuf buffer (which is the size of > the repetition level bytes + the size of the definition level bytes + the > size of the column data bytes) is less than (repLevelSize + repLevelSize + > defLevelSize), the call to limit() will throw. > > The fix is to change line 226 to > {code:java} > (ByteBuffer) bb.slice().limit(defLevelSize){code} > > For symmetry, line 220 could also be changed to > {code:java} > (ByteBuffer) bb.slice().limit(repLevelSize){code} > > although {{pageBufOffset}} is always 0 there and will not cause the limit to > exceed the capacity. > -- This message was sent by Atlassian Jira (v8.20.10#820010)