[jira] [Commented] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780610#comment-17780610
]
ASF GitHub Bot commented on DRILL-8458:
---
jnturton commented on PR #2838:
URL: https://github.com/apache/drill/pull/2838#issuecomment-1783778303
A test data file name like that would be great thank you. Yes, please may
you move the test itself inside TestParquetComplex and see if you can integrate
your test data file generation code into ParquetSimpleTestFileGenerator? It
already takes a Parquet version number parameter IIRC (up until now, only ever
set to v1). I think we may need to come back and organise our tests better for
Parquet v2, but I don't want to saddle this PR with that reorganisation.
> Reading Parquet v2 data page with repetition levels larger than column data
> throws IllegalArgumentException
> ---
>
> Key: DRILL-8458
> URL: https://issues.apache.org/jira/browse/DRILL-8458
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Peter Franzen
>Assignee: James Turton
>Priority: Major
> Fix For: 1.22.0
>
>
> When the size of the repetition level bytes in a Parquet v2 data page is
> larger than the size of the column data bytes,
> {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}}
> throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set
> the limit of a ByteBuffer to a value large than its capacity.
>
> The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}:
>
> {code:java}
> 217 int pageBufOffset = 0;
> 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 219 BytesInput repLevelBytes = BytesInput.from(
> 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize)
> 221 );
> 222 pageBufOffset += repLevelSize;
> 223
> 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 225 final BytesInput defLevelBytes = BytesInput.from(
> 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize)
> 227 );
> 228 pageBufOffset += defLevelSize; {code}
>
> The buffer {{pageBuf}} contains the repetition level bytes followed by the
> definition level bytes followed by the column data bytes.
>
> The code at lines 217-221 reads the repetition level bytes, and then updates
> the position of the {{pageBuf}} buffer to the start of the definition level
> bytes (lines 222 and 224).
>
> The code at lines 225-227 reads the definition level bytes, and when creating
> a slice of the \{{pageBuf }}buffer containing the definition level bytes, the
> slice's limit is set as if the position was at the beginning of the
> repetition level bytes (line 226), i.e as if it not had been updated.
>
> This means that if the capacity of the pageBuf buffer (which is the size of
> the repetition level bytes + the size of the definition level bytes + the
> size of the column data bytes) is less than (repLevelSize + repLevelSize +
> defLevelSize), the call to limit() will throw.
>
> The fix is to change line 226 to
> {code:java}
> (ByteBuffer) bb.slice().limit(defLevelSize){code}
>
> For symmetry, line 220 could also be changed to
> {code:java}
> (ByteBuffer) bb.slice().limit(repLevelSize){code}
>
> although {{pageBufOffset}} is always 0 there and will not cause the limit to
> exceed the capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779873#comment-17779873
]
ASF GitHub Bot commented on DRILL-8458:
---
handmadecode commented on PR #2838:
URL: https://github.com/apache/drill/pull/2838#issuecomment-1781047548
> Argh, a basic buffer arithmetic bug by _yours truly_. I guess it's
remained undetected so far because Parquet v2 is still uncommon the wild. And
because of insufficient Parquet v2 test coverage.
I guess we've all caused our fair share of those bugs ;-)
> Thank you very much for this fix which looks great. Would you mind seeing
if you can relocate the test and its data? We've got TestParquetComplex already
and also some Parquet v2 test files for which a naming pattern has been
started, e.g.
>
> ```
>
exec/java-exec/src/test/resources/parquet/parquet_v2_logical_types_simple.parquet
Sure, how about renaming the test file to
`exec/java-exec/src/test/resources/parquet/parquet_v2_large_repetition_levels.parquet`?
Would you prefer if the test was moved to an existing test class, e.g.
`TestParquetComplex`?
> Reading Parquet v2 data page with repetition levels larger than column data
> throws IllegalArgumentException
> ---
>
> Key: DRILL-8458
> URL: https://issues.apache.org/jira/browse/DRILL-8458
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Peter Franzen
>Assignee: James Turton
>Priority: Major
> Fix For: 1.22.0
>
>
> When the size of the repetition level bytes in a Parquet v2 data page is
> larger than the size of the column data bytes,
> {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}}
> throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set
> the limit of a ByteBuffer to a value large than its capacity.
>
> The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}:
>
> {code:java}
> 217 int pageBufOffset = 0;
> 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 219 BytesInput repLevelBytes = BytesInput.from(
> 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize)
> 221 );
> 222 pageBufOffset += repLevelSize;
> 223
> 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 225 final BytesInput defLevelBytes = BytesInput.from(
> 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize)
> 227 );
> 228 pageBufOffset += defLevelSize; {code}
>
> The buffer {{pageBuf}} contains the repetition level bytes followed by the
> definition level bytes followed by the column data bytes.
>
> The code at lines 217-221 reads the repetition level bytes, and then updates
> the position of the {{pageBuf}} buffer to the start of the definition level
> bytes (lines 222 and 224).
>
> The code at lines 225-227 reads the definition level bytes, and when creating
> a slice of the \{{pageBuf }}buffer containing the definition level bytes, the
> slice's limit is set as if the position was at the beginning of the
> repetition level bytes (line 226), i.e as if it not had been updated.
>
> This means that if the capacity of the pageBuf buffer (which is the size of
> the repetition level bytes + the size of the definition level bytes + the
> size of the column data bytes) is less than (repLevelSize + repLevelSize +
> defLevelSize), the call to limit() will throw.
>
> The fix is to change line 226 to
> {code:java}
> (ByteBuffer) bb.slice().limit(defLevelSize){code}
>
> For symmetry, line 220 could also be changed to
> {code:java}
> (ByteBuffer) bb.slice().limit(repLevelSize){code}
>
> although {{pageBufOffset}} is always 0 there and will not cause the limit to
> exceed the capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8457) Allow configuring csv parser in http storage plugin configuration
[
https://issues.apache.org/jira/browse/DRILL-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779443#comment-17779443
]
ASF GitHub Bot commented on DRILL-8457:
---
ztomanek-dw opened a new pull request, #2840:
URL: https://github.com/apache/drill/pull/2840
# [DRILL-8457](https://issues.apache.org/jira/browse/DRILL-8457): Allow
configuring csv parser in http storage plugin configuration
## Description
HttpApiConfiguration was extended with `csvOptions` field which allows
setting a following properties:
```json
{
"csvOptions": {
"delimiter": ",",
"quote": "\"",
"quoteEscape": "\"",
"lineSeparator": "\n",
"headerExtractionEnabled": null,
"numberOfRowsToSkip": 0,
"numberOfRecordsToRead": -1,
"lineSeparatorDetectionEnabled": true,
"maxColumns": 512,
"maxCharsPerColumn": 4096,
"skipEmptyLines": true,
"ignoreLeadingWhitespaces": true,
"ignoreTrailingWhitespaces": true,
"nullValue": null
}
}
```
this provides greater csv parsing flexibility since user can set different
delimiters, number of columns or max column size.
Also backward compatibility is ensured and parser works same as before if
`csvOptions` is null.
## Documentation
Add a following paragraph into
https://drill.apache.org/docs/http-storage-plugin/#configuring-the-api-connections
```
# CSV parser options
CSV parser of HTTP Storage plugin can be configured using `csvOptions`.
```json
{
"csvOptions": {
"delimiter": ",",
"quote": "\"",
"quoteEscape": "\"",
"lineSeparator": "\n",
"headerExtractionEnabled": null,
"numberOfRowsToSkip": 0,
"numberOfRecordsToRead": -1,
"lineSeparatorDetectionEnabled": true,
"maxColumns": 512,
"maxCharsPerColumn": 4096,
"skipEmptyLines": true,
"ignoreLeadingWhitespaces": true,
"ignoreTrailingWhitespaces": true,
"nullValue": null
}
}
```
E.g. to parse `.tsv` files you can use a following config:
```json
{
"csvOptions": {
"delimiter": "\t"
}
}
```
```
## Testing
Create a following storage plugin with name `github`
```json
{
"type": "http",
"connections": {
"test-data": {
"url":
"https://raw.githubusercontent.com/semantic-web-company/wic-tsv/master/data/de/Test/test_examples.txt;,
"requireTail": false,
"method": "GET",
"authType": "none",
"inputType": "csv",
"xmlDataLevel": 1,
"postParameterLocation": "QUERY_STRING",
"csvOptions": {
"delimiter": "\t",
"quote": "\"",
"quoteEscape": "\"",
"lineSeparator": "\n",
"numberOfRecordsToRead": -1,
"lineSeparatorDetectionEnabled": true,
"maxColumns": 512,
"maxCharsPerColumn": 4096,
"skipEmptyLines": true,
"ignoreLeadingWhitespaces": true,
"ignoreTrailingWhitespaces": true
},
"verifySSLCert": true
}
},
"timeout": 5,
"retryDelay": 1000,
"proxyType": "direct",
"authMode": "SHARED_USER",
"enabled": true
}
```
Then query tsv file with
```sql
SELECT * from github.`test-data`
```.
You should see a result set containing three columns
> Allow configuring csv parser in http storage plugin configuration
> -
>
> Key: DRILL-8457
> URL: https://issues.apache.org/jira/browse/DRILL-8457
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: Future
>Reporter: Zbigniew Tomanek
>Priority: Minor
> Fix For: Future
>
>
> Currently there is no way to configure csv parser when http plugin is used.
> Because of that some kind of files cannot be parsed (e.g. when any column has
> more than 4096 chars or file has a delimiter different from `,`).
> Since in DataWalk we utilize http plugin quite often we've changed our
> internal fork of Drill so following parser/format properties can be
> configured using additional `csvOptions` field:
>
> {code:json}
> {
> "csvOptions": {
> "delimiter": "\t",
> "quote": "\"",
> "quote_escape": "\"",
> "line_separator": "\n",
> "header_extraction_enabled": null,
> "number_of_rows_to_skip": 0,
> "number_of_records_to_read": -1,
> "line_separator_detection_enabled": true,
> "max_columns": 512,
> "max_chars_per_column": 4096,
> "skip_empty_lines": true,
> "ignore_leading_whitespaces": true,
> "ignore_trailing_whitespaces": true,
> "null_value": null
> }
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778027#comment-17778027 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton merged PR #2825: URL: https://github.com/apache/drill/pull/2825 > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1698#comment-1698 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton commented on PR #2825: URL: https://github.com/apache/drill/pull/2825#issuecomment-1772514807 > @cgivre let me convert the comment blocks to deletions as suggested by you and @rymarm and then let's merge it. Done... > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774782#comment-17774782 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton commented on PR #2825: URL: https://github.com/apache/drill/pull/2825#issuecomment-1760767117 @cgivre let me convert the comment blocks to deletions as suggested by you and @rymarm and then let's merge it. > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774526#comment-17774526 ] ASF GitHub Bot commented on DRILL-8454: --- cgivre commented on PR #2825: URL: https://github.com/apache/drill/pull/2825#issuecomment-1759632305 @jnturton Can we merge this? > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8456) uptake POI 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774524#comment-17774524 ] ASF GitHub Bot commented on DRILL-8456: --- cgivre merged PR #2833: URL: https://github.com/apache/drill/pull/2833 > uptake POI 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8456) uptake POI 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770276#comment-17770276 ] ASF GitHub Bot commented on DRILL-8456: --- pjfanning opened a new pull request, #2833: URL: https://github.com/apache/drill/pull/2833 # [DRILL-8456](https://issues.apache.org/jira/browse/DRILL-8456): uptake POI 5.2.4 ## Documentation (Please describe user-visible changes similar to what should appear in the Drill documentation.) ## Testing (Please describe how this PR has been tested.) > uptake POI 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760013#comment-17760013 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre merged PR #2824: URL: https://github.com/apache/drill/pull/2824 > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760012#comment-17760012 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1697705099 @jnturton I fixed imports. @mbeckerle I added one exception which removed a TODO. > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759606#comment-17759606 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton commented on PR #2825: URL: https://github.com/apache/drill/pull/2825#issuecomment-1695823466 A note to the reviewers. Git's default diff algorithm makes it very hard to see that the changes this PR makes to pom.xml are simply whole sections moving into comment blocks. GitHub does not allow selection diff algorithm at this time so, to ease review, I have therefore [uploaded the diff of pom.xml generated using `git diff --diff-algorithm=patience` separately](https://gist.github.com/jnturton/f7a623046da2861bad93419529ff78b3). It is of course also possible to run this locally after pulling down my branch. > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759554#comment-17759554 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre commented on code in PR #2824: URL: https://github.com/apache/drill/pull/2824#discussion_r1307318742 ## pom.xml: ## @@ -689,6 +690,34 @@ **/*.accdb **/*.access_log **/.asf.yaml + **/*.woff2 Review Comment: Done! > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759432#comment-17759432 ] ASF GitHub Bot commented on DRILL-8453: --- jnturton commented on code in PR #2824: URL: https://github.com/apache/drill/pull/2824#discussion_r1306886825 ## pom.xml: ## @@ -689,6 +690,34 @@ **/*.accdb **/*.access_log **/.asf.yaml + **/*.woff2 Review Comment: This list was recently sorted alphabetically in master, can we retain that? > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759418#comment-17759418 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1694918366 > @mbeckerle We always squash commits for Drill PRs :-) I think the TODOs are ok here since this is part 1. > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759417#comment-17759417 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1694917115 @jnturton Are we good to go? > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8420) Remove shaded Guava and upgrade stock Guava 30.1.1-jre → 32.1.2-jre
[ https://issues.apache.org/jira/browse/DRILL-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759056#comment-17759056 ] ASF GitHub Bot commented on DRILL-8420: --- jnturton merged PR #2827: URL: https://github.com/apache/drill/pull/2827 > Remove shaded Guava and upgrade stock Guava 30.1.1-jre → 32.1.2-jre > --- > > Key: DRILL-8420 > URL: https://issues.apache.org/jira/browse/DRILL-8420 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > > * Remove shaded Guava. > * Upgrade stock Guava 30.1.1-jre → 32.1.2-jre -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759030#comment-17759030 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton merged PR #2821: URL: https://github.com/apache/drill/pull/2821 > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8420) Remove Guava shading and patching
[ https://issues.apache.org/jira/browse/DRILL-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758981#comment-17758981 ] ASF GitHub Bot commented on DRILL-8420: --- jnturton opened a new pull request, #2827: URL: https://github.com/apache/drill/pull/2827 # [DRILL-8420](https://issues.apache.org/jira/browse/DRILL-8420): Remove shaded Guava upgrade stock Guava 32.1.2-jre ## Description - Remove shaded Guava. - Upgrade stock Guava 30.1.1-jre → 32.1.2-jre ## Documentation N/A ## Testing All unit tests pass. Manual testing of Drill JDBC driver. > Remove Guava shading and patching > - > > Key: DRILL-8420 > URL: https://issues.apache.org/jira/browse/DRILL-8420 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.0 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8420) Remove Guava shading and patching
[ https://issues.apache.org/jira/browse/DRILL-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758979#comment-17758979 ] ASF GitHub Bot commented on DRILL-8420: --- jnturton closed pull request #2786: DRILL-8420: Remove Guava shading and patching URL: https://github.com/apache/drill/pull/2786 > Remove Guava shading and patching > - > > Key: DRILL-8420 > URL: https://issues.apache.org/jira/browse/DRILL-8420 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.0 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758972#comment-17758972 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton commented on PR #2825: URL: https://github.com/apache/drill/pull/2825#issuecomment-1693009858 This one's ready for review. > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758816#comment-17758816 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on code in PR #2821: URL: https://github.com/apache/drill/pull/2821#discussion_r1305095341 ## exec/jdbc-all/pom.xml: ## @@ -679,86 +688,85 @@ *:* -**/logback.xml Review Comment: Another list that's been sorted alphabetically in this PR. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758815#comment-17758815 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on code in PR #2821: URL: https://github.com/apache/drill/pull/2821#discussion_r1305093910 ## exec/jdbc-all/pom.xml: ## @@ -388,51 +396,52 @@ *:* - io.protostuff:* Review Comment: @cgivre All the diff noise here comes about because I decided to sort this list alphabetically in the hopes that contributors will keep it sorted from here on, making checking what's in it that much easier. I can revert the sorting if preferred though. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758812#comment-17758812 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on code in PR #2821: URL: https://github.com/apache/drill/pull/2821#discussion_r1305092866 ## contrib/storage-phoenix/pom.xml: ## @@ -29,9 +29,9 @@ Drill : Contrib : Storage : Phoenix -5.1.2 +5.1.3 Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758621#comment-17758621 ] ASF GitHub Bot commented on DRILL-8453: --- mbeckerle commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1691830388 I'm ok with merging this. It's still a bit of a work-in-progress (hence the Part 1) Some TODOs in here are mine. I do intend to get to them, but no reason to hold up this change set for that. I highly recommend that you squash these 15 commits together into one coherent commit rather than commit all 15 as is. > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8420) Remove Guava shading and patching
[ https://issues.apache.org/jira/browse/DRILL-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758577#comment-17758577 ] ASF GitHub Bot commented on DRILL-8420: --- jnturton commented on PR #2786: URL: https://github.com/apache/drill/pull/2786#issuecomment-1691627763 > @jnturton Is this PR still relevant? Yes, I'm going to resurrect it after all the other clean up and upgrade PRs are in. > Remove Guava shading and patching > - > > Key: DRILL-8420 > URL: https://issues.apache.org/jira/browse/DRILL-8420 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.0 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8454) Disable unsupported MapR profile and plugin
[ https://issues.apache.org/jira/browse/DRILL-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758572#comment-17758572 ] ASF GitHub Bot commented on DRILL-8454: --- jnturton opened a new pull request, #2825: URL: https://github.com/apache/drill/pull/2825 # [DRILL-8454](https://issues.apache.org/jira/browse/DRILL-8454): Disable unsupported MapR profile and plugin ## Description The MapR build profile and format plugin, which fell out of support in the open source Drill codebase years ago, are disabled. Drill users needing support for these components should contact HPE. ## Documentation See the HPE Ezmeral docs. ## Testing Build and unit tests of remaining modules pass. > Disable unsupported MapR profile and plugin > --- > > Key: DRILL-8454 > URL: https://issues.apache.org/jira/browse/DRILL-8454 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > The MapR build profile and format plugin, which fell out of support in the > open source Drill codebase years ago, are disabled. Drill users needing > support for these components should contact HPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[
https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758547#comment-17758547
]
ASF GitHub Bot commented on DRILL-8453:
---
cgivre commented on code in PR #2824:
URL: https://github.com/apache/drill/pull/2824#discussion_r1304230698
##
contrib/format-xml/src/test/resources/logback-test.xml:
##
@@ -0,0 +1,69 @@
+
+
+
+
+
+
+true
+1
+true
+${LILITH_HOSTNAME:-localhost}
+
Review Comment:
This is a file which should have not been included in the PR. ;-).
> Add XSD Support to XML Reader (Part 1)
> --
>
> Key: DRILL-8453
> URL: https://issues.apache.org/jira/browse/DRILL-8453
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.21.2
>
>
> This PR is a part of a series to add better support for reading XML data to
> Drill. One of the main challenges is that XML data does not have a way of
> inferring data types, nor does it have a way of detecting arrays.
> The only way to do this really well is to have a schema. Some XML files link
> a schema definition file to the data. This PR adds the capability for Drill
> to map XSD schema files into Drill schemas.
> The current plan is as follows: Part 1 of this PR simply adds the reader but
> adds no new user detectable functionality. Part 2 will include the actual
> integration with the XML reader. Part 3 will include the ability to read
> arrays.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8420) Remove Guava shading and patching
[ https://issues.apache.org/jira/browse/DRILL-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758546#comment-17758546 ] ASF GitHub Bot commented on DRILL-8420: --- cgivre commented on PR #2786: URL: https://github.com/apache/drill/pull/2786#issuecomment-1691559682 @jnturton Is this PR still relevant? > Remove Guava shading and patching > - > > Key: DRILL-8420 > URL: https://issues.apache.org/jira/browse/DRILL-8420 > Project: Apache Drill > Issue Type: Improvement > Components: Tools, Build Test >Affects Versions: 1.21.0 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758341#comment-17758341 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-1690970313 > I thought I was reviewing the other PR for the library updates. Could we rebase this on master once that has been merged? Done. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758331#comment-17758331 ] ASF GitHub Bot commented on DRILL-8452: --- jnturton merged PR #2823: URL: https://github.com/apache/drill/pull/2823 > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758284#comment-17758284 ] ASF GitHub Bot commented on DRILL-8453: --- mbeckerle commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1690742314 Sorry bogged down. Will review soon. > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758230#comment-17758230 ] ASF GitHub Bot commented on DRILL-8452: --- cgivre commented on PR #2823: URL: https://github.com/apache/drill/pull/2823#issuecomment-1690587902 I wonder if we could get rid of `httpclient` and replace it with `okhttp3` which is also in use elsewhere. I don't think that is for this PR however. > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758231#comment-17758231 ] ASF GitHub Bot commented on DRILL-8436: --- cgivre commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-169053 I thought I was reviewing the other PR for the library updates. Could we rebase this on master once that has been merged? > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758024#comment-17758024 ] ASF GitHub Bot commented on DRILL-8452: --- jnturton commented on PR #2823: URL: https://github.com/apache/drill/pull/2823#issuecomment-1689949398 @vvysotskyi @luocooong @cgivre I believe this PR is ready for a review. > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757618#comment-17757618 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre commented on PR #2824: URL: https://github.com/apache/drill/pull/2824#issuecomment-1688723270 @mbeckerle > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757485#comment-17757485 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-1688246753 I've got the JDBC driver working by bundling a core-site.xml file in it that handles the relocation of org.apache.hadoop to oadd.org.apache.hadoop. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[
https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757474#comment-17757474
]
ASF GitHub Bot commented on DRILL-8453:
---
cgivre commented on code in PR #2824:
URL: https://github.com/apache/drill/pull/2824#discussion_r1301658422
##
contrib/format-xml/src/test/java/org/apache/drill/exec/store/xml/xsd/TestXSDSchema.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.xml.xsd;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.util.DrillFileUtils;
+import org.apache.drill.exec.record.metadata.MapBuilder;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.junit.Test;
+
+import java.io.File;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class TestXSDSchema {
+
+ @Test
+ public void testSimpleXSD() throws Exception {
+File simple_xsd = DrillFileUtils.getResourceAsFile("/xsd/simple.xsd");
+TupleMetadata schema = DrillXSDSchemaUtils.getSchema(simple_xsd.getPath());
+
+TupleMetadata expectedSchema = new SchemaBuilder()
+.addMap("shiporder")
+ .addMap("attributes")
+.addNullable("orderid", MinorType.VARCHAR)
+ .resumeMap()
+ .addNullable("orderperson", MinorType.VARCHAR)
+ .addMap("shipto")
+.addNullable("name", MinorType.VARCHAR)
+.addNullable("address", MinorType.VARCHAR)
+.addNullable("city", MinorType.VARCHAR)
+.addNullable("country", MinorType.VARCHAR)
+.resumeMap()
+ .addMapArray("item")
+.addNullable("title", MinorType.VARCHAR)
+.addNullable("note", MinorType.VARCHAR)
+.addNullable("quantity", MinorType.VARDECIMAL)
+.addNullable("price", MinorType.VARDECIMAL)
+ .resumeMap()
+.resumeSchema()
+ .buildSchema();
+assertTrue(expectedSchema.isEquivalent(schema));
+ }
+
+
+ @Test
+ public void testComplexXSD() throws Exception {
+File complex_xsd = DrillFileUtils.getResourceAsFile("/xsd/complex.xsd");
+TupleMetadata schema =
DrillXSDSchemaUtils.getSchema(complex_xsd.getPath());
+
+SchemaBuilder sb1 = new SchemaBuilder();
+MapBuilder sb2 = sb1
+.addNullable("comment", MinorType.VARCHAR) // global comment element
+.addMap("infoType")
+ .addMap("attributes")
+.addNullable("kind", MinorType.VARCHAR)
+ .resumeMap()
+.resumeSchema()
+.addMap("purchaseOrder") // global purchaseOrder element
+ .addMap("attributes")
+.addNullable("orderDate", MinorType.DATE) // an attribute
+.addNullable("confirmDate", MinorType.DATE) // an attribute
+ .resumeMap()
+ .addMap("shipTo")
+.addMap("attributes")
+ .addNullable("country", MinorType.VARCHAR) // an attribute
+.resumeMap()
+.addNullable("name", MinorType.VARCHAR)
+.addNullable("street", MinorType.VARCHAR)
+.addNullable("city", MinorType.VARCHAR)
+.addNullable("state", MinorType.VARCHAR)
+.addNullable("zip", MinorType.VARDECIMAL)
+ .resumeMap(); // end shipTo
+MapBuilder sb3 = sb2
+ .addMap("billTo")
+.addMap("attributes")
+ .addNullable("country", MinorType.VARCHAR) // an attribute
+.resumeMap()
+.addNullable("name", MinorType.VARCHAR)
+.addNullable("street", MinorType.VARCHAR)
+ .addNullable("city", MinorType.VARCHAR)
+.addNullable("state", MinorType.VARCHAR)
+.addNullable("zip", MinorType.VARDECIMAL)
+ .resumeMap();
+MapBuilder sb4 = sb3
+ .addNullable("comment", MinorType.VARCHAR)
+ .addMap("items")
+.addMapArray("item")
+ .addMap("attributes")
+.addNullable("partNum", MinorType.VARCHAR) // an attribute
+ .resumeMap()
+
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[
https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757382#comment-17757382
]
ASF GitHub Bot commented on DRILL-8453:
---
jnturton commented on code in PR #2824:
URL: https://github.com/apache/drill/pull/2824#discussion_r1301478103
##
contrib/format-xml/src/main/java/org/apache/drill/exec/store/xml/xsd/DrillXSDSchemaUtils.java:
##
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.xml.xsd;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.apache.ws.commons.schema.XmlSchema;
+import org.apache.ws.commons.schema.XmlSchemaCollection;
+import org.apache.ws.commons.schema.XmlSchemaElement;
+
+import org.apache.ws.commons.schema.XmlSchemaObject;
+import org.apache.ws.commons.schema.walker.XmlSchemaWalker;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.xml.transform.stream.StreamSource;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class DrillXSDSchemaUtils {
+ private static final MinorType DEFAULT_TYPE = MinorType.VARCHAR;
+ private static final Logger logger =
LoggerFactory.getLogger(DrillXSDSchemaUtils.class);
+
+ /**
+ * This map maps the data types defined by the XSD definition to Drill data
types.
+ */
+ public static final ImmutableMap XML_TYPE_MAPPINGS =
ImmutableMap.builder()
+.put("BASE64BINARY", MinorType.VARBINARY)
+.put("BOOLEAN", MinorType.BIT)
+.put("DATE", MinorType.DATE)
+.put("DATETIME", MinorType.TIMESTAMP)
+.put("DECIMAL", MinorType.VARDECIMAL)
+.put("DOUBLE", MinorType.FLOAT8)
+.put("DURATION", MinorType.INTERVAL)
+.put("FLOAT", MinorType.FLOAT4)
+.put("HEXBINARY", MinorType.VARBINARY)
+.put("STRING", MinorType.VARCHAR)
+.put("TIME", MinorType.TIME)
+.build();
+
+ /**
+ * This function is only used for testing, but accepts a XSD file as input
rather than a {@link InputStream}
+ * @param filename A {@link String} containing an XSD file.
+ * @return A {@link TupleMetadata} containing a Drill representation of the
XSD schema.
+ * @throws IOException If anything goes wrong or the file is not found.
+ */
+ public static TupleMetadata getSchema(String filename) throws IOException {
Review Comment:
```suggestion
@VisibleForTesting
public static TupleMetadata getSchema(String filename) throws IOException {
```
##
contrib/format-xml/src/test/java/org/apache/drill/exec/store/xml/xsd/TestXSDSchema.java:
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.xml.xsd;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.util.DrillFileUtils;
+import org.apache.drill.exec.record.metadata.MapBuilder;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.junit.Test;
+
+import java.io.File;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class TestXSDSchema {
+
+ @Test
+ public
[jira] [Commented] (DRILL-8453) Add XSD Support to XML Reader (Part 1)
[ https://issues.apache.org/jira/browse/DRILL-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757205#comment-17757205 ] ASF GitHub Bot commented on DRILL-8453: --- cgivre opened a new pull request, #2824: URL: https://github.com/apache/drill/pull/2824 # [DRILL-8453](https://issues.apache.org/jira/browse/DRILL-8453): Add XSD Support to XML Reader (Part 1) ## Description This PR is a part of a series to add better support for reading XML data to Drill. One of the main challenges is that XML data does not have a way of inferring data types, nor does it have a way of detecting arrays. The only way to do this really well is to have a schema. Some XML files link a schema definition file to the data. This PR adds the capability for Drill to map XSD schema files into Drill schemas. The current plan is as follows: Part 1 of this PR simply adds the reader but adds no new user detectable functionality. Part 2 will include the actual integration with the XML reader. Part 3 will include the ability to read arrays. ## Documentation No user facing changes. ## Testing Added new unit tests. > Add XSD Support to XML Reader (Part 1) > -- > > Key: DRILL-8453 > URL: https://issues.apache.org/jira/browse/DRILL-8453 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This PR is a part of a series to add better support for reading XML data to > Drill. One of the main challenges is that XML data does not have a way of > inferring data types, nor does it have a way of detecting arrays. > The only way to do this really well is to have a schema. Some XML files link > a schema definition file to the data. This PR adds the capability for Drill > to map XSD schema files into Drill schemas. > The current plan is as follows: Part 1 of this PR simply adds the reader but > adds no new user detectable functionality. Part 2 will include the actual > integration with the XML reader. Part 3 will include the ability to read > arrays. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757165#comment-17757165 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre merged PR #2819: URL: https://github.com/apache/drill/pull/2819 > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756975#comment-17756975 ] ASF GitHub Bot commented on DRILL-8450: --- jnturton commented on PR #2819: URL: https://github.com/apache/drill/pull/2819#issuecomment-1686562600 LGTM > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756962#comment-17756962 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre commented on PR #2819: URL: https://github.com/apache/drill/pull/2819#issuecomment-1686494732 @mbeckerle @jnturton Are we ok to merge this? I'll add support for arrays in a separate PR. > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[
https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756425#comment-17756425
]
ASF GitHub Bot commented on DRILL-8450:
---
cgivre commented on code in PR #2819:
URL: https://github.com/apache/drill/pull/2819#discussion_r1299285670
##
contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXmlOptions.java:
##
@@ -111,7 +111,7 @@ public String toString() {
public static class HttpXmlOptionsBuilder {
private int dataLevel;
-private boolean allTextMode;
+private Boolean allTextMode;
Review Comment:
@mbeckerle
In the JSON reader there are two parameters: `allTextMode` and
`readAllNumbersAsDouble`. Both are boolean.For the XML reader, I chose not
to implement the `readAllNumbersAsDouble` parameter because in practice, it
requires very clean data. From using Drill with clients, I can tell you from
a lot of personal experience that this was one of the biggest data challenges.
For instance, you'd get data where there was an DOUBLE field and then there
would be a row with zero denoted as `0`. This would then cause schema change
exceptions.
We have actually made significant improvements in Drill's implicit casting
rules which do prevent a lot of schema change exceptions and as a result, IMHO,
it makes distinguishing between INTs and DOUBLES a lot less important. So..
out of laziness I decided it wasn't worth it. I can be convinced otherwise.
> Add Data Type Inference to XML Format Plugin
>
>
> Key: DRILL-8450
> URL: https://issues.apache.org/jira/browse/DRILL-8450
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> This PR adds data type inference to the XML format plugin. In similar
> fashion to other plugins, it adds a new configuration parameter: allTextMode,
> which when set to true, reads all data as strings. The default is true.
> Note that the inference is limited to doubles, date, timestamps, boolean and
> strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[
https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756109#comment-17756109
]
ASF GitHub Bot commented on DRILL-8450:
---
mbeckerle commented on code in PR #2819:
URL: https://github.com/apache/drill/pull/2819#discussion_r1298815764
##
contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpXmlOptions.java:
##
@@ -111,7 +111,7 @@ public String toString() {
public static class HttpXmlOptionsBuilder {
private int dataLevel;
-private boolean allTextMode;
+private Boolean allTextMode;
Review Comment:
I thought there were 3 modes: allTextMode, allNumbersAreDouble mode, and
infer-types mode.
So why is this a boolean vs am enum?
> Add Data Type Inference to XML Format Plugin
>
>
> Key: DRILL-8450
> URL: https://issues.apache.org/jira/browse/DRILL-8450
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> This PR adds data type inference to the XML format plugin. In similar
> fashion to other plugins, it adds a new configuration parameter: allTextMode,
> which when set to true, reads all data as strings. The default is true.
> Note that the inference is limited to doubles, date, timestamps, boolean and
> strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756001#comment-17756001 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre commented on PR #2819: URL: https://github.com/apache/drill/pull/2819#issuecomment-1684011222 @mbeckerle Could you please take another look. I had to fix a few things for a unit test. Thx! > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755564#comment-17755564 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-1682344122 I've just set this PR to Draft because I rediscovered a problem in the Drill JDBC driver. I'll paste a chat message I sent to @vvysotskyi a few months back below, to reveal the nature of the problem. I'm sure it's ultimately fixable, but I don't know of an elegant fix yet. > Hi Vova! I decided to try upgrading Drill's Hadoop libs to 3.3.5. Things are working but there is a problem in the Drill JDBC fat jar. There, the shade plugin relocates Hadoop to underneath oadd as usual but now there are class names present in the core-default.xml file in hadoop-common.jar which are not updated by the shade plugin. The result is that the JDBC driver is broken. While the shade plugin can update some kinds of text config files, it doesn't appear that it can update arbitrary XML config like core-default.xml. I thought of including our own manually updated copy of core-default.xml in exec/jdbc-all/src/resources and trying to make sure the shade plugin picks that one instead of the one in hadoop-common.jar. My only reservation is that introducing this copy creates a maintenance burden for the future so I thought to ask you if you have any ideas... > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755348#comment-17755348 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on code in PR #2821: URL: https://github.com/apache/drill/pull/2821#discussion_r1296649086 ## exec/jdbc-all/pom.xml: ## @@ -102,6 +102,18 @@ commons-codec commons-codec + + com.sun.jersey + jersey-core + + + com.sun.jersey + jersey-server + + + com.sun.jersey Review Comment: Ah! Thanks. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755161#comment-17755161 ] ASF GitHub Bot commented on DRILL-8436: --- pjfanning commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-1680885511 > So, this seems to work but not in JDK 8 In JDK, it can't find io/netty/handler/codec/http/HttpRequest. Maybe, we need to add an explicit dependency on the io.netty:netty-codec-http jar > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755157#comment-17755157 ] ASF GitHub Bot commented on DRILL-8436: --- pjfanning commented on code in PR #2821: URL: https://github.com/apache/drill/pull/2821#discussion_r1296126455 ## exec/jdbc-all/pom.xml: ## @@ -102,6 +102,18 @@ commons-codec commons-codec + + com.sun.jersey + jersey-core + + + com.sun.jersey + jersey-server + + + com.sun.jersey Review Comment: hadoop 3.3.6 uses my fork of jersey-json > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755063#comment-17755063 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton commented on PR #2821: URL: https://github.com/apache/drill/pull/2821#issuecomment-1680623578 So, this seems to work but not in JDK 8 > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755061#comment-17755061 ] ASF GitHub Bot commented on DRILL-8452: --- jnturton opened a new pull request, #2823: URL: https://github.com/apache/drill/pull/2823 # [DRILL-8452](https://issues.apache.org/jira/browse/DRILL-8452): Library upgrades - hbase.version -> 2.5.5-hadoop3 - avro.version -> 1.11.2 - metrics.version -> 4.2.19 - jersey.version -> 2.40 - asm.version -> 9.5 - antlr.version -> 4.13.0 - maven.version -> 3.9.4 - commons.validator.version -> 1.7 - protostuff.version -> 1.8.0 - joda.version -> 2.12.5 - surefire.version -> 3.1.2 - jna.version -> 5.13.0 - commons.compress.version -> 1.23.0 - hikari.version -> 5.0.1 - httpclient.version -> 4.5.14 - libthrift.version -> 0.18.1 - snakeyaml.version -> 2.1 - testcontainers.version -> 1.18.3 - httpdlog-parser.version -> 5.10.0 - log4j.version -> 2.20.0 - aircompressor.version -> 0.25 - hbase.version -> 2.5.5 ## Documentation N/A ## Testing Existing unit tests. > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > - hbase.version -> 2.5.5-hadoop3 > - avro.version -> 1.11.2 > - metrics.version -> 4.2.19 > - jersey.version -> 2.40 > - asm.version -> 9.5 > - antlr.version -> 4.13.0 > - maven.version -> 3.9.4 > - commons.validator.version -> 1.7 > - protostuff.version -> 1.8.0 > - joda.version -> 2.12.5 > - surefire.version -> 3.1.2 > - jna.version -> 5.13.0 > - commons.compress.version -> 1.23.0 > - hikari.version -> 5.0.1 > - httpclient.version -> 4.5.14 > - libthrift.version -> 0.18.1 > - snakeyaml.version -> 2.1 > - testcontainers.version -> 1.18.3 > - httpdlog-parser.version -> 5.10.0 > - log4j.version -> 2.20.0 > - aircompressor.version -> 0.25 > - hbase.version -> 2.5.5 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753868#comment-17753868 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre commented on PR #2819: URL: https://github.com/apache/drill/pull/2819#issuecomment-1676572517 @mbeckerle Unit tests fixed. I also added the data type inference for APIs that generate XML. @jnturton, The CI is still failing with that Kerberos issue. > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8451) options and profile pages have bad order symbols style
[ https://issues.apache.org/jira/browse/DRILL-8451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753274#comment-17753274 ] ASF GitHub Bot commented on DRILL-8451: --- cgivre merged PR #2820: URL: https://github.com/apache/drill/pull/2820 > options and profile pages have bad order symbols style > -- > > Key: DRILL-8451 > URL: https://issues.apache.org/jira/browse/DRILL-8451 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Maksym Rymar >Assignee: Maksym Rymar >Priority: Minor > Fix For: 1.22.0 > > Attachments: image-2023-08-09-13-16-56-687.png, > image-2023-08-09-13-18-28-574.png, image-2023-08-09-13-19-41-196.png > > > "Options" and "Profile" pages have bad order symbols style: > Current: > !image-2023-08-09-13-16-56-687.png! > !image-2023-08-09-13-18-28-574.png! > Expected: > !image-2023-08-09-13-19-41-196.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8436) Upgrade Hadoop 3.2.4 -> 3.3.6
[ https://issues.apache.org/jira/browse/DRILL-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753225#comment-17753225 ] ASF GitHub Bot commented on DRILL-8436: --- jnturton opened a new pull request, #2821: URL: https://github.com/apache/drill/pull/2821 # [DRILL-8436](https://issues.apache.org/jira/browse/DRILL-8436): Upgrade Hadoop 3.2.4 -> 3.3.6 ## Description Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. ## Documentation N/A ## Testing Existing unit tests, manual testing of Drill HTTP services. > Upgrade Hadoop 3.2.4 -> 3.3.6 > - > > Key: DRILL-8436 > URL: https://issues.apache.org/jira/browse/DRILL-8436 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8451) options and profile pages have bad order symbols style
[ https://issues.apache.org/jira/browse/DRILL-8451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753220#comment-17753220 ] ASF GitHub Bot commented on DRILL-8451: --- rymarm commented on PR #2820: URL: https://github.com/apache/drill/pull/2820#issuecomment-1674773018 @cgivre Sure! Done. > options and profile pages have bad order symbols style > -- > > Key: DRILL-8451 > URL: https://issues.apache.org/jira/browse/DRILL-8451 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Maksym Rymar >Assignee: Maksym Rymar >Priority: Minor > Fix For: 1.22.0 > > Attachments: image-2023-08-09-13-16-56-687.png, > image-2023-08-09-13-18-28-574.png, image-2023-08-09-13-19-41-196.png > > > "Options" and "Profile" pages have bad order symbols style: > Current: > !image-2023-08-09-13-16-56-687.png! > !image-2023-08-09-13-18-28-574.png! > Expected: > !image-2023-08-09-13-19-41-196.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752841#comment-17752841 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre commented on PR #2819: URL: https://github.com/apache/drill/pull/2819#issuecomment-1673416654 Converting to draft. There's a unit test failing in the HTTP plugin. > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8451) options and profile pages have bad order symbols style
[ https://issues.apache.org/jira/browse/DRILL-8451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752840#comment-17752840 ] ASF GitHub Bot commented on DRILL-8451: --- cgivre commented on PR #2820: URL: https://github.com/apache/drill/pull/2820#issuecomment-1673415150 @rymarm Would you mind please rebasing on current master? There's now a merge conflict as a result of your other PR. > options and profile pages have bad order symbols style > -- > > Key: DRILL-8451 > URL: https://issues.apache.org/jira/browse/DRILL-8451 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Maksym Rymar >Assignee: Maksym Rymar >Priority: Minor > Fix For: 1.22.0 > > Attachments: image-2023-08-09-13-16-56-687.png, > image-2023-08-09-13-18-28-574.png, image-2023-08-09-13-19-41-196.png > > > "Options" and "Profile" pages have bad order symbols style: > Current: > !image-2023-08-09-13-16-56-687.png! > !image-2023-08-09-13-18-28-574.png! > Expected: > !image-2023-08-09-13-19-41-196.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8449) Typo in FreeMarker templates
[
https://issues.apache.org/jira/browse/DRILL-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752839#comment-17752839
]
ASF GitHub Bot commented on DRILL-8449:
---
cgivre merged PR #2818:
URL: https://github.com/apache/drill/pull/2818
> Typo in FreeMarker templates
>
>
> Key: DRILL-8449
> URL: https://issues.apache.org/jira/browse/DRILL-8449
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Minor
> Fix For: 1.22.0
>
> Attachments: image-2023-08-07-21-15-56-847.png
>
>
> Css based properties use colon({{{}:{}}}) to assign values. The
> {{result.ftl}} and {{options.ftl have property }}width{{{} with equal
> sign({}}}={{{}) instead of colon({}}}:{{{}).{}}}
>
> This typo makes query result table has incorrect display.
> Steps to reproduce:
> # Execute example query: {{select full_name from cp.`employee.json` limit 1}}
> # Push on "Column visibility" button and togle "full_name" column visibility
> off and turn back on.
> !image-2023-08-07-21-15-56-847.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8451) options and profile pages have bad order symbols style
[ https://issues.apache.org/jira/browse/DRILL-8451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752401#comment-17752401 ] ASF GitHub Bot commented on DRILL-8451: --- rymarm opened a new pull request, #2820: URL: https://github.com/apache/drill/pull/2820 # [DRILL-8451](https://issues.apache.org/jira/browse/DRILL-8451): options and profile pages have bad order symbols style ## Description "Options" and "Profile" pages have bad order symbols style: Current:   Expected:  "Options" and "Profile" pages were using Drill own sorting symbols styling instead of dataTables style that is used on all other pages. I've removed outdated `black-asc.gif`, `black-dsc.gif`, `black-unsorted.gif` sorting symbols, and replaced them with dataTable styled symbols. I also disabled sorting for "value" and "description" columns on "Options" page. Sorting on "value" doesn't work properly due to different types of values, and sorting on the "description" column is just redundant. ### Options page before:  after:  ### Profile page before:  after:  ## Documentation None requerid ## Testing Visual check of UI in Firefox and Edge browsers. > options and profile pages have bad order symbols style > -- > > Key: DRILL-8451 > URL: https://issues.apache.org/jira/browse/DRILL-8451 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Maksym Rymar >Assignee: Maksym Rymar >Priority: Minor > Fix For: 1.22.0 > > Attachments: image-2023-08-09-13-16-56-687.png, > image-2023-08-09-13-18-28-574.png, image-2023-08-09-13-19-41-196.png > > > "Options" and "Profile" pages have bad order symbols style: > Current: > !image-2023-08-09-13-16-56-687.png! > !image-2023-08-09-13-18-28-574.png! > Expected: > !image-2023-08-09-13-19-41-196.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[
https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752112#comment-17752112
]
ASF GitHub Bot commented on DRILL-8450:
---
mbeckerle commented on code in PR #2819:
URL: https://github.com/apache/drill/pull/2819#discussion_r1287322034
##
common/src/main/java/org/apache/drill/common/Typifier.java:
##
@@ -88,6 +96,40 @@ public class Typifier {
// If a String contains any of these, try to evaluate it as an equation
private static final char[] MathCharacters = new char[]{'+', '-', '/', '*',
'='};
+ /**
+ * This function infers the Drill data type of unknown data.
+ * @param data The input text of unknown data type.
+ * @return A {@link MinorType} of the Drill data type.
+ */
+ public static MinorType typifyToDrill (String data) {
+Entry result = Typifier.typify(data);
+String dataType = result.getKey().getSimpleName();
+
+// If the string is empty, return UNKNOWN
Review Comment:
Makes perfect sense.
For XML you need XSD to know what's potentially repeating.
Sometimes that is easy because of minOccurs/maxOccurs.
But there's also these "implied arrays".
```
Add Data Type Inference to XML Format Plugin
>
>
> Key: DRILL-8450
> URL: https://issues.apache.org/jira/browse/DRILL-8450
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> This PR adds data type inference to the XML format plugin. In similar
> fashion to other plugins, it adds a new configuration parameter: allTextMode,
> which when set to true, reads all data as strings. The default is true.
> Note that the inference is limited to doubles, date, timestamps, boolean and
> strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[
https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752099#comment-17752099
]
ASF GitHub Bot commented on DRILL-8450:
---
cgivre commented on code in PR #2819:
URL: https://github.com/apache/drill/pull/2819#discussion_r1287295957
##
common/src/main/java/org/apache/drill/common/Typifier.java:
##
@@ -88,6 +96,40 @@ public class Typifier {
// If a String contains any of these, try to evaluate it as an equation
private static final char[] MathCharacters = new char[]{'+', '-', '/', '*',
'='};
+ /**
+ * This function infers the Drill data type of unknown data.
+ * @param data The input text of unknown data type.
+ * @return A {@link MinorType} of the Drill data type.
+ */
+ public static MinorType typifyToDrill (String data) {
+Entry result = Typifier.typify(data);
+String dataType = result.getKey().getSimpleName();
+
+// If the string is empty, return UNKNOWN
Review Comment:
@mbeckerle Drill doesn't really have an `UNKNOWN` data type. The way the
typifier works is that if it can't determine the datatype, it falls back to
string which can basically accept anything.
Regarding the lists... The issue is that to create a list, you have to set
the data mode to `REPEATED`. The problem with XML is that there's no real way
to know if a field is repeated or not. Consider this:
```xml
a
a1
a2
```
Since Drill uses the streaming reader, when it first encounters the `author`
field, it would add an entry for a VARCHAR field. However, when it gets to the
next author record, it should be list, but there's no way to really know that
w/o a schema.
With JSON we don't have this problem because it uses `[` to denote lists.
Does that make sense?
> Add Data Type Inference to XML Format Plugin
>
>
> Key: DRILL-8450
> URL: https://issues.apache.org/jira/browse/DRILL-8450
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> This PR adds data type inference to the XML format plugin. In similar
> fashion to other plugins, it adds a new configuration parameter: allTextMode,
> which when set to true, reads all data as strings. The default is true.
> Note that the inference is limited to doubles, date, timestamps, boolean and
> strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[
https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752087#comment-17752087
]
ASF GitHub Bot commented on DRILL-8450:
---
mbeckerle commented on code in PR #2819:
URL: https://github.com/apache/drill/pull/2819#discussion_r1287251884
##
contrib/format-xml/README.md:
##
@@ -15,12 +15,15 @@ The default configuration is shown below:
"extensions": [
"xml"
],
+ "allTextMode": true,
"dataLevel": 2
}
```
## Data Types
-All fields are read as strings. Nested fields are read as maps. Future
functionality could include support for lists.
+The XML reader has an `allTextMode` which, when set to `true` reads all data
fields as strings.
+When set to `false`, Drill will attempt to infer data types.
+Nested fields are read as maps. Future functionality could include support
for lists.
Review Comment:
Not really part of this change set, but I don't know what you are suggesting
by "future functionality could include support for lists." I'd like to
understand that plan/idea just as part of grokking all of this XML mapping.
##
common/src/main/java/org/apache/drill/common/Typifier.java:
##
@@ -88,6 +96,40 @@ public class Typifier {
// If a String contains any of these, try to evaluate it as an equation
private static final char[] MathCharacters = new char[]{'+', '-', '/', '*',
'='};
+ /**
+ * This function infers the Drill data type of unknown data.
+ * @param data The input text of unknown data type.
+ * @return A {@link MinorType} of the Drill data type.
+ */
+ public static MinorType typifyToDrill (String data) {
+Entry result = Typifier.typify(data);
+String dataType = result.getKey().getSimpleName();
+
+// If the string is empty, return UNKNOWN
Review Comment:
The next line of code contradicts this comment by returning VARCHAR.
(Unless VARCHAR == UNKNOWN, which is news to me.)
> Add Data Type Inference to XML Format Plugin
>
>
> Key: DRILL-8450
> URL: https://issues.apache.org/jira/browse/DRILL-8450
> Project: Apache Drill
> Issue Type: Improvement
> Components: Format - XML
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> This PR adds data type inference to the XML format plugin. In similar
> fashion to other plugins, it adds a new configuration parameter: allTextMode,
> which when set to true, reads all data as strings. The default is true.
> Note that the inference is limited to doubles, date, timestamps, boolean and
> strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8450) Add Data Type Inference to XML Format Plugin
[ https://issues.apache.org/jira/browse/DRILL-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752065#comment-17752065 ] ASF GitHub Bot commented on DRILL-8450: --- cgivre opened a new pull request, #2819: URL: https://github.com/apache/drill/pull/2819 # [DRILL-8450](https://issues.apache.org/jira/browse/DRILL-8450): Add Data Type Inference to XML Format Plugin ## Description This PR adds data type inference to the XML format plugin. In similar fashion to other plugins, it adds a new configuration parameter: `allTextMode`, which when set to `true`, reads all data as strings. The default is `true`. Note that the inference is limited to doubles, date, timestamps, boolean and strings. ## Documentation Updated README ## Testing Added unit test. > Add Data Type Inference to XML Format Plugin > > > Key: DRILL-8450 > URL: https://issues.apache.org/jira/browse/DRILL-8450 > Project: Apache Drill > Issue Type: Improvement > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > This PR adds data type inference to the XML format plugin. In similar > fashion to other plugins, it adds a new configuration parameter: allTextMode, > which when set to true, reads all data as strings. The default is true. > Note that the inference is limited to doubles, date, timestamps, boolean and > strings. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8449) Typo in FreeMarker templates
[
https://issues.apache.org/jira/browse/DRILL-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751778#comment-17751778
]
ASF GitHub Bot commented on DRILL-8449:
---
cgivre commented on PR #2818:
URL: https://github.com/apache/drill/pull/2818#issuecomment-1668420106
Thanks @rymarm
I'll merge once the CI passes.
> Typo in FreeMarker templates
>
>
> Key: DRILL-8449
> URL: https://issues.apache.org/jira/browse/DRILL-8449
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Minor
> Fix For: 1.22.0
>
> Attachments: image-2023-08-07-21-15-56-847.png
>
>
> Css based properties use colon({{{}:{}}}) to assign values. The
> {{result.ftl}} and {{options.ftl have property }}width{{{} with equal
> sign({}}}={{{}) instead of colon({}}}:{{{}).{}}}
>
> This typo makes query result table has incorrect display.
> Steps to reproduce:
> # Execute example query: {{select full_name from cp.`employee.json` limit 1}}
> # Push on "Column visibility" button and togle "full_name" column visibility
> off and turn back on.
> !image-2023-08-07-21-15-56-847.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8449) Typo in FreeMarker templates
[
https://issues.apache.org/jira/browse/DRILL-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751772#comment-17751772
]
ASF GitHub Bot commented on DRILL-8449:
---
rymarm opened a new pull request, #2818:
URL: https://github.com/apache/drill/pull/2818
# [DRILL-8449](https://issues.apache.org/jira/browse/DRILL-8449): Typo in
FreeMarker templates
## Description
CSS-based properties use a colon(`:`) to assign values. The `result.ftl` and
`options.ftl` have the property `width` with an equal sign(`=`) instead of a
colon(`:`).
This typo makes a query result table has an incorrect display.
Steps to reproduce:
1. Execute example query: `select full_name from cp.`employee.json` limit 1`;
2. Push the "Column visibility" button, toggle "full_name" column visibility
off, and turn it back on.
## Documentation
None requerid
## Testing
Visual test of UI
> Typo in FreeMarker templates
>
>
> Key: DRILL-8449
> URL: https://issues.apache.org/jira/browse/DRILL-8449
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Minor
> Fix For: 1.22.0
>
> Attachments: image-2023-08-07-21-15-56-847.png
>
>
> Css based properties use colon({{{}:{}}}) to assign values. The
> {{result.ftl}} and {{options.ftl have property }}width{{{} with equal
> sign({}}}={{{}) instead of colon({}}}:{{{}).{}}}
>
> This typo makes query result table has incorrect display.
> Steps to reproduce:
> # Execute example query: {{select full_name from cp.`employee.json` limit 1}}
> # Push on "Column visibility" button and togle "full_name" column visibility
> off and turn back on.
> !image-2023-08-07-21-15-56-847.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750559#comment-17750559
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre merged PR #2747:
URL: https://github.com/apache/drill/pull/2747
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750049#comment-17750049
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1661290803
> @LYCJeff Could you please rebase on current master. Other than that, LGTM
+1.
>
> Sorry this took so long. Thank you very much for this contribution!
@cgivre I have updated, thanks for your review.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749982#comment-17749982
]
ASF GitHub Bot commented on DRILL-8393:
---
cgivre commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1660866785
@LYCJeff Could you please rebase on current master. Other than that, LGTM
+1. Sorry this took so long.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749520#comment-17749520
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1659744329
@cgivre Is there anything else I should do for this PR?
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json;, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8443) upgrade netty to 4.1.94 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748572#comment-17748572 ] ASF GitHub Bot commented on DRILL-8443: --- vvysotskyi commented on PR #2813: URL: https://github.com/apache/drill/pull/2813#issuecomment-1655539951 Now tests for this PR fail for other reason (likely due to changes in the PR): ``` Error: Failures: Error: TestResultSetLoaderOmittedValues.testOmittedValuesAtEndWithOverflow:264 Row 0 col d should be null Error:TestResultSetLoaderOverflow.testBatchSizeLimit:164 expected:<16385> but was:<8193> Error:TestOutputBatchSize.testSizerRepeatedRepeatedList:2922 expected:<1048576> but was:<1048560> Error: Errors: Error:TestResultSetLoaderOmittedValues>SubOperatorTest.classTeardown:39 » IllegalState Allocator[ROOT] closed with outstanding buffers allocated (9). ``` > upgrade netty to 4.1.94 due to CVE > -- > > Key: DRILL-8443 > URL: https://issues.apache.org/jira/browse/DRILL-8443 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/45 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8442) NPE on DeltaRowGroupScan
[
https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743437#comment-17743437
]
ASF GitHub Bot commented on DRILL-8442:
---
vvysotskyi merged PR #2811:
URL: https://github.com/apache/drill/pull/2811
> NPE on DeltaRowGroupScan
>
>
> Key: DRILL-8442
> URL: https://issues.apache.org/jira/browse/DRILL-8442
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.21.1
> Environment: pyspark 3.4.0
> delta-spark 2.4.0
> Ubuntu 22.04.2 LTS
>
>Reporter: Matt Keranen
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> SELECT * on Delta table (Parquet) throws null pointer exception:
>
> {noformat}
> 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET
> `exec.query.max_rows`=1000
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING -->
> FINISHED
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED
> 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *
> from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))
> limit 5
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException:
> Cannot construct instance of
> `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem:
> `java.lang.NullPointerException`
> at [Source: (String)"{
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
>
[jira] [Commented] (DRILL-8443) upgrade netty to 4.1.94 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736813#comment-17736813 ] ASF GitHub Bot commented on DRILL-8443: --- cgivre commented on PR #2813: URL: https://github.com/apache/drill/pull/2813#issuecomment-1605819647 @pjfanning I think we may have a CI issue. @vvysotskyi @jnturton Any ideas here? Could this be related to that issue we encountered before with the CI and connections? > upgrade netty to 4.1.94 due to CVE > -- > > Key: DRILL-8443 > URL: https://issues.apache.org/jira/browse/DRILL-8443 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/45 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8443) upgrade netty to 4.1.94 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736764#comment-17736764 ] ASF GitHub Bot commented on DRILL-8443: --- pjfanning commented on PR #2813: URL: https://github.com/apache/drill/pull/2813#issuecomment-1605613862 Some tests failing with this error - `IO Running in secure mode, but config doesn't have a keytab` Seems like a Hadoop issue. We may need to upgrade Hadoop or at least review the Hadoop config used in the tests. > upgrade netty to 4.1.94 due to CVE > -- > > Key: DRILL-8443 > URL: https://issues.apache.org/jira/browse/DRILL-8443 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/45 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8443) upgrade netty to 4.1.94 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736688#comment-17736688 ] ASF GitHub Bot commented on DRILL-8443: --- pjfanning opened a new pull request, #2813: URL: https://github.com/apache/drill/pull/2813 ## Description https://github.com/apache/drill/security/dependabot/45 ## Testing CI build > upgrade netty to 4.1.94 due to CVE > -- > > Key: DRILL-8443 > URL: https://issues.apache.org/jira/browse/DRILL-8443 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > https://github.com/apache/drill/security/dependabot/45 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8442) NPE on DeltaRowGroupScan
[
https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735831#comment-17735831
]
ASF GitHub Bot commented on DRILL-8442:
---
vvysotskyi opened a new pull request, #2811:
URL: https://github.com/apache/drill/pull/2811
# [DRILL-8442](https://issues.apache.org/jira/browse/DRILL-8442): Fix
DeltaRowGroupScan deserialization
## Description
(Please describe the change. If more than one ticket is fixed, include a
reference to those tickets.)
## Documentation
NA
## Testing
Updated unit test to cover the failing case.
Closes #2810
> NPE on DeltaRowGroupScan
>
>
> Key: DRILL-8442
> URL: https://issues.apache.org/jira/browse/DRILL-8442
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.21.1
> Environment: pyspark 3.4.0
> delta-spark 2.4.0
> Ubuntu 22.04.2 LTS
>
>Reporter: Matt Keranen
>Priority: Minor
>
> SELECT * on Delta table (Parquet) throws null pointer exception:
>
> {noformat}
> 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET
> `exec.query.max_rows`=1000
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING -->
> FINISHED
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED
> 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *
> from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))
> limit 5
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by:
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735764#comment-17735764 ] ASF GitHub Bot commented on DRILL-8353: --- kmatt commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1600977689 #2810, #2809 > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.21.0 > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735428#comment-17735428 ] ASF GitHub Bot commented on DRILL-8353: --- cgivre commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1599387786 @kmatt A github issue is good! Please be sure to tag @vvysotskyi in it as he was the original developer of this plugin. > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.21.0 > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735427#comment-17735427 ] ASF GitHub Bot commented on DRILL-8353: --- kmatt commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1599386040 @vvysotskyi https://issues.apache.org/jira/browse/DRILL-8442 Should this be a GitHub issue, or is Jira the correct place for it? > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.21.0 > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725560#comment-17725560 ] ASF GitHub Bot commented on DRILL-8438: --- cgivre merged PR #2808: URL: https://github.com/apache/drill/pull/2808 > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725426#comment-17725426 ] ASF GitHub Bot commented on DRILL-8438: --- nielsbasjes commented on PR #2808: URL: https://github.com/apache/drill/pull/2808#issuecomment-1559461058 My fixes seem to have had the desired effect. > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725380#comment-17725380 ] ASF GitHub Bot commented on DRILL-8438: --- cgivre commented on PR #2808: URL: https://github.com/apache/drill/pull/2808#issuecomment-1559145929 @nielsbasjes Thanks for this. I'll rerun the failed test. Not sure why we're getting network issues but that seems unrelated to this PR. > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8158) Remove non-reproducible build outputs
[ https://issues.apache.org/jira/browse/DRILL-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724778#comment-17724778 ] ASF GitHub Bot commented on DRILL-8158: --- jnturton merged PR #2805: URL: https://github.com/apache/drill/pull/2805 > Remove non-reproducible build outputs > - > > Key: DRILL-8158 > URL: https://issues.apache.org/jira/browse/DRILL-8158 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Herve Boutemy >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > For context see [1] and [2]. The git-commit-id plugin includes information > like build host, email and time which is not compatible with a reproducible > build. Drill's built in sys.version table will return the build email and > time if they are present in the build's git.properties file so these columns > must be deprecated. Other useful Git-related information is retained. > In accompanying commits, some Kerberos unit test fixes are applied, and the > tests reenabled, and some updates to Release.md are included. > [1] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html] > [2] > [https://github.com/jvm-repo-rebuild/reproducible-central#org.apache.drill:drill-root] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8437) Add Header Index Pagination
[ https://issues.apache.org/jira/browse/DRILL-8437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724713#comment-17724713 ] ASF GitHub Bot commented on DRILL-8437: --- cgivre merged PR #2806: URL: https://github.com/apache/drill/pull/2806 > Add Header Index Pagination > --- > > Key: DRILL-8437 > URL: https://issues.apache.org/jira/browse/DRILL-8437 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - HTTP >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > Some APIs include pagination fields in the HTTP response headers. This PR > adds a new pagination method called Header Index which supports that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8158) Remove non-reproducible build outputs
[ https://issues.apache.org/jira/browse/DRILL-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724672#comment-17724672 ] ASF GitHub Bot commented on DRILL-8158: --- hboutemy commented on code in PR #2805: URL: https://github.com/apache/drill/pull/2805#discussion_r1199775459 ## pom.xml: ## @@ -562,7 +562,6 @@ ^git\.commit\..*$ ^git\.dirty$ ^git\.tags$ - ^git\.total\.commit\.count$ Review Comment: in fact, if you do a shallow git clone (= what Maven Release Plugin does to reduce download), you get 1 instead of the value of the full content: yes, not so easy to guess :) > Remove non-reproducible build outputs > - > > Key: DRILL-8158 > URL: https://issues.apache.org/jira/browse/DRILL-8158 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Herve Boutemy >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > For context see [1] and [2]. The git-commit-id plugin includes information > like build host, email and time which is not compatible with a reproducible > build. Drill's built in sys.version table will return the build email and > time if they are present in the build's git.properties file so these columns > must be deprecated. Other useful Git-related information is retained. > In accompanying commits, some Kerberos unit test fixes are applied, and the > tests reenabled, and some updates to Release.md are included. > [1] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html] > [2] > [https://github.com/jvm-repo-rebuild/reproducible-central#org.apache.drill:drill-root] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8158) Remove non-reproducible build outputs
[ https://issues.apache.org/jira/browse/DRILL-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724670#comment-17724670 ] ASF GitHub Bot commented on DRILL-8158: --- hboutemy commented on code in PR #2805: URL: https://github.com/apache/drill/pull/2805#discussion_r1199775459 ## pom.xml: ## @@ -562,7 +562,6 @@ ^git\.commit\..*$ ^git\.dirty$ ^git\.tags$ - ^git\.total\.commit\.count$ Review Comment: in fact, if you do a shallow git clone, you get 1 instead of the value of the full content: yes, not so easy to guess :) > Remove non-reproducible build outputs > - > > Key: DRILL-8158 > URL: https://issues.apache.org/jira/browse/DRILL-8158 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Herve Boutemy >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > For context see [1] and [2]. The git-commit-id plugin includes information > like build host, email and time which is not compatible with a reproducible > build. Drill's built in sys.version table will return the build email and > time if they are present in the build's git.properties file so these columns > must be deprecated. Other useful Git-related information is retained. > In accompanying commits, some Kerberos unit test fixes are applied, and the > tests reenabled, and some updates to Release.md are included. > [1] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html] > [2] > [https://github.com/jvm-repo-rebuild/reproducible-central#org.apache.drill:drill-root] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8437) Add Header Index Pagination
[
https://issues.apache.org/jira/browse/DRILL-8437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724647#comment-17724647
]
ASF GitHub Bot commented on DRILL-8437:
---
cgivre opened a new pull request, #2806:
URL: https://github.com/apache/drill/pull/2806
# [DRILL-8437](https://issues.apache.org/jira/browse/DRILL-8437): Add Header
Index Pagination
(Please replace `PR Title` with actual PR Title)
## Description
See below
## Documentation
Updated README.
## Header Index Pagination
Header index pagination is used when the API in question returns a link to
the next page in the response header. Shopify is one such example of an API
that does this.
The only configuration option is the `nextPageParam` which is the parameter
that Drill should look for in the response header.
```json
"paginator": {
"nextPageParam": "page",
"method": "HEADER_INDEX"
}
```
## Testing
Added UT and manually tested with Shopify API.
> Add Header Index Pagination
> ---
>
> Key: DRILL-8437
> URL: https://issues.apache.org/jira/browse/DRILL-8437
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.21.1
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.22.0
>
>
> Some APIs include pagination fields in the HTTP response headers. This PR
> adds a new pagination method called Header Index which supports that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8158) Remove non-reproducible build outputs
[ https://issues.apache.org/jira/browse/DRILL-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724621#comment-17724621 ] ASF GitHub Bot commented on DRILL-8158: --- jnturton commented on code in PR #2805: URL: https://github.com/apache/drill/pull/2805#discussion_r1199746266 ## pom.xml: ## @@ -562,7 +562,6 @@ ^git\.commit\..*$ ^git\.dirty$ ^git\.tags$ - ^git\.total\.commit\.count$ Review Comment: Thank you for this. I'd have thought that the count of the commits leading to up to some tagged commit _is_ reproducible? Or are we catering for the case where a build system makes a shallow clone for efficiency and that truncates the commit history? > Remove non-reproducible build outputs > - > > Key: DRILL-8158 > URL: https://issues.apache.org/jira/browse/DRILL-8158 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Herve Boutemy >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > For context see [1] and [2]. The git-commit-id plugin includes information > like build host, email and time which is not compatible with a reproducible > build. Drill's built in sys.version table will return the build email and > time if they are present in the build's git.properties file so these columns > must be deprecated. Other useful Git-related information is retained. > In accompanying commits, some Kerberos unit test fixes are applied, and the > tests reenabled, and some updates to Release.md are included. > [1] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html] > [2] > [https://github.com/jvm-repo-rebuild/reproducible-central#org.apache.drill:drill-root] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8158) Remove non-reproducible build outputs
[ https://issues.apache.org/jira/browse/DRILL-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724594#comment-17724594 ] ASF GitHub Bot commented on DRILL-8158: --- hboutemy opened a new pull request, #2805: URL: https://github.com/apache/drill/pull/2805 [DRILL-8158](https://issues.apache.org/jira/browse/DRILL-DRILL-8158 see rebuild of release 1.21.1 https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/org/apache/drill/README.md > Remove non-reproducible build outputs > - > > Key: DRILL-8158 > URL: https://issues.apache.org/jira/browse/DRILL-8158 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Herve Boutemy >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > For context see [1] and [2]. The git-commit-id plugin includes information > like build host, email and time which is not compatible with a reproducible > build. Drill's built in sys.version table will return the build email and > time if they are present in the build's git.properties file so these columns > must be deprecated. Other useful Git-related information is retained. > In accompanying commits, some Kerberos unit test fixes are applied, and the > tests reenabled, and some updates to Release.md are included. > [1] [https://maven.apache.org/guides/mini/guide-reproducible-builds.html] > [2] > [https://github.com/jvm-repo-rebuild/reproducible-central#org.apache.drill:drill-root] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8433) Add Percent Change UDF to Drill
[ https://issues.apache.org/jira/browse/DRILL-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721372#comment-17721372 ] ASF GitHub Bot commented on DRILL-8433: --- cgivre merged PR #2801: URL: https://github.com/apache/drill/pull/2801 > Add Percent Change UDF to Drill > --- > > Key: DRILL-8433 > URL: https://issues.apache.org/jira/browse/DRILL-8433 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > Adds a function to calculate the percent change between two columns. Doing > this without a custom function is cumbersome because you have to include a > check for division by zero. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[ https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721368#comment-17721368 ] ASF GitHub Bot commented on DRILL-8430: --- pjfanning commented on PR #2800: URL: https://github.com/apache/drill/pull/2800#issuecomment-1542255539 @cgivre no worries - I had timeout and other issues with GitHub around that time too. Some sort of general GitHub health issues, I guess. > add factory method for creating Jackson ObjectMappers > - > > Key: DRILL-8430 > URL: https://issues.apache.org/jira/browse/DRILL-8430 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > See https://issues.apache.org/jira/browse/DRILL-8415 > It's useful to keep any customisation of the ObjectMapper creation in 1 place -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[ https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721309#comment-17721309 ] ASF GitHub Bot commented on DRILL-8430: --- cgivre commented on PR #2800: URL: https://github.com/apache/drill/pull/2800#issuecomment-1542048115 @pjfanning Sorry for the repeated comments yesterday. Github was giving me an error message and it didn't look like the comment had actually posted. > add factory method for creating Jackson ObjectMappers > - > > Key: DRILL-8430 > URL: https://issues.apache.org/jira/browse/DRILL-8430 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > See https://issues.apache.org/jira/browse/DRILL-8415 > It's useful to keep any customisation of the ObjectMapper creation in 1 place -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[
https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721262#comment-17721262
]
ASF GitHub Bot commented on DRILL-8430:
---
jnturton commented on code in PR #2800:
URL: https://github.com/apache/drill/pull/2800#discussion_r1189578197
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/http/oauth/OAuthUtils.java:
##
@@ -36,6 +37,7 @@
public class OAuthUtils {
private static final Logger logger =
LoggerFactory.getLogger(OAuthUtils.class);
+ private static final ObjectMapper MAPPER = JacksonUtils.createObjectMapper();
Review Comment:
Hmm, reading [on little](https://stackoverflow.com/a/36162525/1153953)
reveals that lock contention in a shared ObjectMapper degrades performance for
multithreaded applications . Now I'm in two minds.
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/http/oauth/OAuthUtils.java:
##
@@ -36,6 +37,7 @@
public class OAuthUtils {
private static final Logger logger =
LoggerFactory.getLogger(OAuthUtils.class);
+ private static final ObjectMapper MAPPER = JacksonUtils.createObjectMapper();
Review Comment:
Hmm, reading [on a little](https://stackoverflow.com/a/36162525/1153953)
reveals that lock contention in a shared ObjectMapper degrades performance for
multithreaded applications . Now I'm in two minds.
> add factory method for creating Jackson ObjectMappers
> -
>
> Key: DRILL-8430
> URL: https://issues.apache.org/jira/browse/DRILL-8430
> Project: Apache Drill
> Issue Type: Task
> Components: Server
>Reporter: PJ Fanning
>Priority: Major
>
> See https://issues.apache.org/jira/browse/DRILL-8415
> It's useful to keep any customisation of the ObjectMapper creation in 1 place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[
https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721260#comment-17721260
]
ASF GitHub Bot commented on DRILL-8430:
---
pjfanning commented on code in PR #2800:
URL: https://github.com/apache/drill/pull/2800#discussion_r1189568415
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/http/oauth/OAuthUtils.java:
##
@@ -36,6 +37,7 @@
public class OAuthUtils {
private static final Logger logger =
LoggerFactory.getLogger(OAuthUtils.class);
+ private static final ObjectMapper MAPPER = JacksonUtils.createObjectMapper();
Review Comment:
the problem with global object mappers, writers and readers is that if they
are public, then someone can modify their config - exactly the issue I found in
our test code
I opened https://issues.apache.org/jira/browse/DRILL-8431 to look at
wrapping the Jackson classes to create immutable instances that can be more
safely shared. So far, that looks like a lot of work and the benefits may not
be worth it.
> add factory method for creating Jackson ObjectMappers
> -
>
> Key: DRILL-8430
> URL: https://issues.apache.org/jira/browse/DRILL-8430
> Project: Apache Drill
> Issue Type: Task
> Components: Server
>Reporter: PJ Fanning
>Priority: Major
>
> See https://issues.apache.org/jira/browse/DRILL-8415
> It's useful to keep any customisation of the ObjectMapper creation in 1 place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[
https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721257#comment-17721257
]
ASF GitHub Bot commented on DRILL-8430:
---
jnturton commented on code in PR #2800:
URL: https://github.com/apache/drill/pull/2800#discussion_r1189562746
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/http/oauth/OAuthUtils.java:
##
@@ -36,6 +37,7 @@
public class OAuthUtils {
private static final Logger logger =
LoggerFactory.getLogger(OAuthUtils.class);
+ private static final ObjectMapper MAPPER = JacksonUtils.createObjectMapper();
Review Comment:
Follow up: [this post](https://stackoverflow.com/a/3909846/1153953) suggests
that global ObjectReader and ObjectWriter objects might be a better choice than
a global ObjectMapper.
> add factory method for creating Jackson ObjectMappers
> -
>
> Key: DRILL-8430
> URL: https://issues.apache.org/jira/browse/DRILL-8430
> Project: Apache Drill
> Issue Type: Task
> Components: Server
>Reporter: PJ Fanning
>Priority: Major
>
> See https://issues.apache.org/jira/browse/DRILL-8415
> It's useful to keep any customisation of the ObjectMapper creation in 1 place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[
https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721253#comment-17721253
]
ASF GitHub Bot commented on DRILL-8430:
---
jnturton commented on code in PR #2800:
URL: https://github.com/apache/drill/pull/2800#discussion_r1189557009
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/http/oauth/OAuthUtils.java:
##
@@ -36,6 +37,7 @@
public class OAuthUtils {
private static final Logger logger =
LoggerFactory.getLogger(OAuthUtils.class);
+ private static final ObjectMapper MAPPER = JacksonUtils.createObjectMapper();
Review Comment:
We're converting method scope ObjectMappers to static class members which is
efficient in terms of rework but will add a bit to Drill's fixed heap
requirement since they can never be collected. It looks like ObjectMappers are
thread safe so, for the cases where the caller does not need to do mapper
customisation, could we get even better reuse from a new singleton
`JacksonUtils.DEFAULT_MAPPER`?
##
common/src/test/java/org/apache/drill/test/DrillTest.java:
##
@@ -37,11 +38,10 @@
public class DrillTest extends BaseTest {
- protected static final ObjectMapper objectMapper;
+ private static final ObjectMapper objectMapper =
JacksonUtils.createObjectMapper();
Review Comment:
Nice catch, thank you.
> add factory method for creating Jackson ObjectMappers
> -
>
> Key: DRILL-8430
> URL: https://issues.apache.org/jira/browse/DRILL-8430
> Project: Apache Drill
> Issue Type: Task
> Components: Server
>Reporter: PJ Fanning
>Priority: Major
>
> See https://issues.apache.org/jira/browse/DRILL-8415
> It's useful to keep any customisation of the ObjectMapper creation in 1 place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8433) Add Percent Change UDF to Drill
[ https://issues.apache.org/jira/browse/DRILL-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721172#comment-17721172 ] ASF GitHub Bot commented on DRILL-8433: --- cgivre opened a new pull request, #2801: URL: https://github.com/apache/drill/pull/2801 # [DRILL-8433](https://issues.apache.org/jira/browse/DRILL-8433): Add Percent Change UDF to Drill ## Description Adds a new UDF, `percent_change(x,y)` which calculates the percent change between the two variables. While this is relatively easy to do in SQL, it also requires null and zero checks to prevent division by zero errors. This makes this calculation simpler. ## Documentation See README. ## Testing Added unit tests. > Add Percent Change UDF to Drill > --- > > Key: DRILL-8433 > URL: https://issues.apache.org/jira/browse/DRILL-8433 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > Adds a function to calculate the percent change between two columns. Doing > this without a custom function is cumbersome because you have to include a > check for division by zero. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8430) add factory method for creating Jackson ObjectMappers
[
https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721065#comment-17721065
]
ASF GitHub Bot commented on DRILL-8430:
---
pjfanning commented on code in PR #2800:
URL: https://github.com/apache/drill/pull/2800#discussion_r1189058399
##
common/src/test/java/org/apache/drill/test/DrillTest.java:
##
@@ -37,11 +38,10 @@
public class DrillTest extends BaseTest {
- protected static final ObjectMapper objectMapper;
+ private static final ObjectMapper objectMapper =
JacksonUtils.createObjectMapper();
Review Comment:
Some tests were using this mapper and modifying it - which seems like a
dangerous thing to do because other tests will be affected by the modified
mapper. I've changed the tests that used this mapper to create their own
separate mappers.
> add factory method for creating Jackson ObjectMappers
> -
>
> Key: DRILL-8430
> URL: https://issues.apache.org/jira/browse/DRILL-8430
> Project: Apache Drill
> Issue Type: Task
> Components: Server
>Reporter: PJ Fanning
>Priority: Major
>
> See https://issues.apache.org/jira/browse/DRILL-8415
> It's useful to keep any customisation of the ObjectMapper creation in 1 place
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8429) jackson 2.14.3
[ https://issues.apache.org/jira/browse/DRILL-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721021#comment-17721021 ] ASF GitHub Bot commented on DRILL-8429: --- cgivre merged PR #2798: URL: https://github.com/apache/drill/pull/2798 > jackson 2.14.3 > -- > > Key: DRILL-8429 > URL: https://issues.apache.org/jira/browse/DRILL-8429 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > Jackson 2.14.3 has perf and security hardening improvements > https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.14.3 > prelude to DRILL-8415 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8429) jackson 2.14.3
[ https://issues.apache.org/jira/browse/DRILL-8429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721019#comment-17721019 ] ASF GitHub Bot commented on DRILL-8429: --- pjfanning commented on PR #2798: URL: https://github.com/apache/drill/pull/2798#issuecomment-1540568759 Disabling one of the broken tests was suggested by @cgivre - see https://github.com/apache/drill/pull/2800#issuecomment-1540012488 > jackson 2.14.3 > -- > > Key: DRILL-8429 > URL: https://issues.apache.org/jira/browse/DRILL-8429 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > Jackson 2.14.3 has perf and security hardening improvements > https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.14.3 > prelude to DRILL-8415 -- This message was sent by Atlassian Jira (v8.20.10#820010)
