RE: Re: [VOTE] Release Apache Drill 1.21.1 - RC0

[Vova Vysotskyi]

Hi all,

Verified signatures and checksum, and ran several queries using prebuilt 
binaries, checked query profiles, no issues found.

+1 (binding)

Kind regards,
Volodymyr Vysotskyi

On 2023/04/28 13:38:19 "Rumar, Maksym" wrote:
> Hello all,
> 
> Started Drill from the tarball (RC0) and ran different queries. LGTM +1
> 
> By the way, the link in the James mail leads to an incorrect RC version - 
> RC1, but we are voting for RC0 now.
> 
> Regards,
> Maksym
> 
> Від: Charles Givre 
> Надіслано: 25 квітня 2023 р. 15:09
> Кому: dev@drill.apache.org 
> Копія: u...@drill.apache.org 
> Тема: Re: [VOTE] Release Apache Drill 1.21.1 - RC0
> 
> Hello all,
> Built from source, ran various queries. LGTM +1 (Binding)
> -- C
> 
> 
> 
> > On Apr 19, 2023, at 5:26 AM, James Turton  wrote:
> >
> > Hi all
> >
> > I'd like to propose the first release candidate (RC0) of Apache Drill, 
> > version 1.21.1.
> >
> > The release candidate covers a total of 12 resolved Jira issues [1]. Thanks 
> > to everyone who contributed to this release.
> >
> > The tarball artefacts are hosted at [2] and the Maven artefacts are hosted 
> > at [3].
> >
> > This release candidate is based on commit 
> > 6da4b77ceb1510df4de4a2cfcc43a536b99f37ab located at [4].
> >
> > [ ] +1
> > [ ] +0
> > [ ] -1
> >
> > [1] 
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12352949
> > [2] https://dist.apache.org/repos/dist/dev/drill/1.21.1-rc1/
> > [3] https://repository.apache.org/content/repositories/orgapachedrill-1105/
> > [4] https://github.com/jnturton/drill/commits/drill-1.21.1
> 
> 

Re: [VOTE] Release Apache Drill 1.21.0 - RC0

[Vova Vysotskyi]

Hi all!

I have built Drill from the sources archive on JDK 8, run it, checked the 
distributions, submitted several queries, checked the Drill JDBC driver with 
SQuirrelSQL, and verified signatures. No issues were found.

+1 (binding)

Kind regards,
Volodymyr Vysotskyi

On 2023/02/17 14:30:09 James Turton wrote:
> Apologies for incorrect GitHub tarball artifacts link, please see below.
> 
> [2] https://dist.apache.org/repos/dist/dev/drill/1.21.0-rc0/
> 
> On 2023/02/15 17:00, James Turton wrote:
> > P.S.
> >
> > - Please do not merge anything into the master branch during the 
> > release freeze.
> > - Here are the CI runs for the final commit included into 1.21.0 RC0 [1]
> >
> > [1] https://github.com/apache/drill/actions/runs/4180497758
> >
> > On 2023/02/15 11:05, James Turton wrote:
> >> Hi all,
> >>
> >> I'd like to propose the first release candidate (RC0) of Apache 
> >> Drill, version 1.21.0.
> >>
> >> The release candidate covers a total of 106 resolved JIRAs [1]. 
> >> Thanks to everyone who contributed to this release.
> >>
> >> The tarball artifacts are hosted at [2] and the maven artifacts are 
> >> hosted at [3].
> >>
> >> This release candidate is based on commit 
> >> ff86c29b125776c56f361fc9187fb96458de37a6 located at [4].
> >>
> >> Please download and try out the release, especially using JDK 8 
> >> because this release was built with JDK 17.
> >>
> >> The vote ends at Sat 18 Feb 2023 12:00:00 UTC.
> >>
> >> [ ] +1
> >> [ ] +0
> >> [ ] -1
> >>
> >> [1] 
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12335462
> >> [2] https://github.com/jnturton/drill/releases/tag/drill-1.21.0
> >> [3] 
> >> https://repository.apache.org/content/repositories/orgapachedrill-1104/
> >> [4] https://github.com/jnturton/drill/commits/drill-1.21.0
> >
> 
> 

[jira] [Created] (DRILL-8400) Fix pruning partitions with pushed transitive predicates

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8400:
-

 Summary: Fix pruning partitions with pushed transitive predicates
 Key: DRILL-8400
 URL: https://issues.apache.org/jira/browse/DRILL-8400
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


See {{TestHivePartitionPruning.prunePartitionsBasedOnTransitivePredicates()}} 
test for details.

The issue occurs for queries like these:
{code:sql}
SELECT * FROM hive.partition_pruning_test t1 
JOIN hive.partition_with_few_schemas t2 ON t1.`d` = t2.`d` AND t1.`e` = t2.`e` 
WHERE t2.`e` IS NOT NULL AND t1.`d` = 1
{code}

The expected behavior is to create additional filters based on the existing 
filters and join conditions. We have a {{TRANSITIVE_CLOSURE}} planning phase, 
which is responsible for such query transformations, but Drill pushes down 
filters from the WHERE condition before that phase, so the optimization is not 
performed.

Ideally, we should move rules from the {{TRANSITIVE_CLOSURE}} phase to the 
{{LOGICAL}} phase so that the planner will choose the most optimal plan, but it 
wouldn't help until CALCITE-1048 is fixed (it is required to pull predicates 
when three has {{RelSubset}} nodes).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8398) Fix GitHub Actions to use proper JDK version

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8398:
-

 Summary: Fix GitHub Actions to use proper JDK version
 Key: DRILL-8398
 URL: https://issues.apache.org/jira/browse/DRILL-8398
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


In one of the changes GitHub Actions `actions/setup-java@v2` action was 
removed, and after that provided java version (11) was used instead of the 
specified one in the 
matrix.java.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8397) Drill prints warnings to console when starting it

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8397:
-

 Summary: Drill prints warnings to console when starting it
 Key: DRILL-8397
 URL: https://issues.apache.org/jira/browse/DRILL-8397
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


When starting the drill in embedded mode, it prints the following warnings:
{noformat}
11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is 
logback-classic version 1.4.5
11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could 
NOT find resource [logback-test.xml]
11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found 
resource [logback.xml] at 
[file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - 
 element is deprecated. Near [level] on line 73
11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please 
use "level" attribute within  or  elements instead.
11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - 
 element is deprecated. Near [level] on line 78
11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please 
use "level" attribute within  or  elements instead.
11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - 
 element is deprecated. Near [level] on line 91
11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please 
use "level" attribute within  or  elements instead.
11:19:55,652 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- Processing appender named [STDOUT]
11:19:55,652 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
11:19:55,658 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler 
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] 
for [encoder] property
11:19:55,689 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- Processing appender named [QUERY]
11:19:55,689 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- About to instantiate appender of type 
[ch.qos.logback.core.rolling.RollingFileAppender]
11:19:55,697 |-INFO in 
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No compression 
will be used
11:19:55,699 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler 
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] 
for [encoder] property
11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY] - 
Active log file name: 
/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY] - 
File property is set to 
[/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
11:19:55,700 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- Processing appender named [FILE]
11:19:55,700 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler 
- About to instantiate appender of type 
[ch.qos.logback.core.rolling.RollingFileAppender]
11:19:55,700 |-INFO in 
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No compression 
will be used
11:19:55,701 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler 
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] 
for [encoder] property
11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - 
Active log file name: 
/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log
11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - 
File property is set to 
[/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log]
11:19:55,702 |-INFO in 
ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity 
of logger [org.apache.drill] to false
11:19:55,702 |-INFO in ch.qos.logback.classic.model.processor.LevelModelHandler 
- org.apache.drill level set to INFO
11:19:55,702 |-INFO in 
ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching 
appender named [FILE] to Logger[org.apache.drill]
11:19:55,703 |-INFO in 
ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity 
of logger [query.logger] to false
11:19:55,703 |-INFO in ch.qos.logback.classic.model.processor.LevelModelHandler 
- query.logger level set to INFO
11:19:55,703 |-INFO in 
ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching 
appender named [QUERY] to Log

[jira] [Created] (DRILL-8396) Update checkstyle version

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8396:
-

 Summary: Update checkstyle version
 Key: DRILL-8396
 URL: https://issues.apache.org/jira/browse/DRILL-8396
 Project: Apache Drill
  Issue Type: Task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Update com.puppycrawl.tools:checkstyle version to the latest one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8381) Add support for filtered aggregate calls

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8381:
-

 Summary: Add support for filtered aggregate calls
 Key: DRILL-8381
 URL: https://issues.apache.org/jira/browse/DRILL-8381
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Currently, Drill ignores filters for filtered aggregate calls and returns 
incorrect results.
Here is the example query for which Drill will return incorrect results:
{code:sql}
SELECT count(n_name) FILTER(WHERE n_regionkey = 1) AS nations_count_in_1_region,
count(n_name) FILTER(WHERE n_regionkey = 2) AS nations_count_in_2_region,
count(n_name) FILTER(WHERE n_regionkey = 3) AS nations_count_in_3_region,
count(n_name) FILTER(WHERE n_regionkey = 4) AS nations_count_in_4_region,
count(n_name) FILTER(WHERE n_regionkey = 0) AS nations_count_in_0_region
FROM cp.`tpch/nation.parquet`
{code}

{noformat}
+---+---+---+---+---+
| nations_count_in_1_region | nations_count_in_2_region | 
nations_count_in_3_region | nations_count_in_4_region | 
nations_count_in_0_region |
+---+---+---+---+---+
| 25| 25| 25
| 25| 25|
+---+---+---+---+---+
{noformat}

But correct result is
{noformat}
+---+---+---+---+---+
| nations_count_in_1_region | nations_count_in_2_region | 
nations_count_in_3_region | nations_count_in_4_region | 
nations_count_in_0_region |
+---+---+---+---+---+
| 5 | 5 | 5 
| 5 | 5 |
+---+---+---+---+---+
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8380) Remove customised SqlValidatorImpl.deriveAlias

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8380:
-

 Summary: Remove customised SqlValidatorImpl.deriveAlias
 Key: DRILL-8380
 URL: https://issues.apache.org/jira/browse/DRILL-8380
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8379) Update Calcite to 1.33.0

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8379:
-

 Summary: Update Calcite to 1.33.0
 Key: DRILL-8379
 URL: https://issues.apache.org/jira/browse/DRILL-8379
 Project: Apache Drill
  Issue Type: Task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release Apache Drill 1.20.3 - RC0

[Vova Vysotskyi]

Hi all,

Verified the release, and no issues were found.

+1 (binding)

Kind regards,
Volodymyr Vysotskyi

On 2023/01/04 15:16:21 Charles Givre wrote:
> Hello all,
> Just a reminder, to please try this out and record your vote here.  I think 
> we need one more vote to publish the RC0.
> Best,
> -- C
> 
> > On Dec 28, 2022, at 1:21 AM, James Turton  wrote:
> > 
> > Herewith an attempt to unmangle the formatting of my original email :(.
> > 
> > Hi all,
> > 
> > I'd like to propose the first release candidate (RC0) of Apache Drill, 
> > version 1.20.3. This is very likely to
> > be the final update to the 1.20.x series given that 1.21 is expected in the 
> > near future.
> > 
> > The release candidate covers a total of 30 resolved JIRAs [1]. Thanks to 
> > everyone who contributed to this release
> > and to Jingchuan Hu for some nontrivial backporting of fixes.
> > 
> > The tarball artifacts are hosted at [2] and the maven artifacts are hosted 
> > at [3]. The unit test run based on the
> > final merge into the 1.20 branch may be viewed here [4].
> > 
> > This release candidate is based on commit 
> > f49d553d71db553fda2fc330f9a3f05d7dcdd17f located at [5].
> > 
> > Please download and try out the release.
> > 
> > [ ] +1
> > [ ] +0
> > [ ] -1
> > 
> > Here's my vote: +1
> > 
> > [1] 
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12352165
> > [2] https://dist.apache.org/repos/dist/dev/drill/1.20.3-rc0
> > [3] https://repository.apache.org/content/repositories/orgapachedrill-1103/
> > [4] https://github.com/apache/drill/actions/runs/3780949700
> > [5] https://github.com/jnturton/drill/releases/tag/drill-1.20.3
> > 
> > Season's greetings
> > James
> 
> 

[jira] [Created] (DRILL-8369) Add support for querying DeltaLake snapshots by version

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8369:
-

 Summary: Add support for querying DeltaLake snapshots by version
 Key: DRILL-8369
 URL: https://issues.apache.org/jira/browse/DRILL-8369
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8358) Storage plugin for querying other Apache Drill clusters

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8358:
-

 Summary: Storage plugin for querying other Apache Drill clusters
 Key: DRILL-8358
 URL: https://issues.apache.org/jira/browse/DRILL-8358
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8353) Format plugin for Delta Lake

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8353:
-

 Summary: Format plugin for Delta Lake
 Key: DRILL-8353
 URL: https://issues.apache.org/jira/browse/DRILL-8353
 Project: Apache Drill
  Issue Type: New Feature
Affects Versions: 1.20.2
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: Future


Implement format plugin for Delta Lake.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8304) Update Calcite to 1.32

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8304:
-

 Summary: Update Calcite to 1.32
 Key: DRILL-8304
 URL: https://issues.apache.org/jira/browse/DRILL-8304
 Project: Apache Drill
  Issue Type: Task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8303) Add support for inserts into JDBC storage

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8303:
-

 Summary: Add support for inserts into JDBC storage
 Key: DRILL-8303
 URL: https://issues.apache.org/jira/browse/DRILL-8303
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Allow inserting into JDBC tables and pushing down complete insert statement 
where possible.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8279) Use thick Phoenix driver

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8279:
-

 Summary: Use thick Phoenix driver
 Key: DRILL-8279
 URL: https://issues.apache.org/jira/browse/DRILL-8279
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


phoenix-queryserver-client shades Avatica classes, so it causes issues when 
starting Drill and shaded class from phoenix jars is loaded before, so Drill 
wouldn't be able to start correctly.

To avoid that, phoenix thick client can be used, it also will improve query 
performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-6371) Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical stage

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-6371.
---
Resolution: Fixed

Fixed in DRILL-7523

> Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical 
> stage
> --
>
> Key: DRILL-6371
> URL: https://issues.apache.org/jira/browse/DRILL-6371
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.13.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Minor
> Fix For: Future
>
>
> FilterSetOpTransposeRule, ProjectSetOpTransposeRule are leveraged in 
> DRILL-3855.
>  They are used in HepPlanner, but if they additionally will be enabled in 
> main logical planning stage for Volcano planner, more cases will be covered 
> with these rules.
>  For example:
> {code:java}
> WITH year_total_1
>  AS (SELECT c.r_regionkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/region.parquet` c
>  UNION ALL
>  SELECT c.n_nationkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/nation.parquet` c),
>  year_total_2
>  AS (SELECT c.r_regionkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/region.parquet` c
>  UNION ALL
>  SELECT c.n_nationkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/nation.parquet` c)
> SELECT count(t_w_firstyear.customer_id) as ct
> FROM   year_total_1 t_w_firstyear,
>year_total_2 t_w_secyear
> WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
>  AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
> {code}
> Current plan after performing rules:
> {code:java}
> LogicalAggregate(group=[{}], ct=[COUNT($0)])
>   LogicalProject(customer_id=[$0])
> LogicalFilter(condition=[AND(=($1, $3), >($1, 0), >($3, 0))])
>   LogicalJoin(condition=[true], joinType=[inner])
> LogicalUnion(all=[true])
>   LogicalProject(customer_id=[$1], year_total=[1])
> EnumerableTableScan(table=[[cp, tpch/region.parquet]])
>   LogicalProject(customer_id=[$1], year_total=[1])
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
> LogicalUnion(all=[true])
>   LogicalProject(customer_id=[$1], year_total=[1])
> EnumerableTableScan(table=[[cp, tpch/region.parquet]])
>   LogicalProject(customer_id=[$1], year_total=[1])
> EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
> {code}
> Since LogicalFilter isn't under LogicalUnion the FilterSetOpTransposeRule is 
> not performed. 
>  FilterJoinRule from main Drill logical stage pushes LogicalFilter below, but 
> the stage with FilterSetOpTransposeRule is already finished. 
>  That's why FilterSetOpTransposeRule and ProjectSetOpTransposeRule should be 
> used in Drill main logical stage with Volcano planner.
> Currently using them in Volcano Planner can cause infinite loops - 
> CALCITE-1271 (can be resolved after solving CALCITE-2223)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-4086) Query hangs in planing

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-4086.
---
Resolution: Fixed

Fixed in DRILL-7523

> Query hangs in planing
> --
>
> Key: DRILL-4086
> URL: https://issues.apache.org/jira/browse/DRILL-4086
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning  Optimization
>Affects Versions: 1.2.0
>Reporter: boris chmiel
>    Assignee: Vova Vysotskyi
>Priority: Major
>
> The query is stuck seems blocked on planning (pending)
> View : 
> {noformat}
> create or replace view View1 AS (
> SELECT 
> B1.columns[0] c0,
> B1.columns[1] c1
> FROM dfs.tmp.`TEST\B1.csv` B1
> LEFT OUTER JOIN dfs.tmp.`TEST\BK.csv` BK
> ON B1.columns[1] = BK.columns[0]
> WHERE BK.columns[0] is null AND trim(B1.columns[1]) <> ''
> );
> {noformat}
> {noformat}
> create or replace view View2 AS (
> SELECT 
> View1.c0,
> View1.c1
> FROM View1
> LEFT OUTER JOIN dfs.tmp.`TEST\BK.csv` BK
> ON View1.c1 = BK.columns[0]
> WHERE BK.columns[0] is null AND trim(View1.c1) <> ''
> );
> {noformat}
> Query :
> {noformat}
> select * FROM dfs.tmp.View2 
> {noformat}
> => Infinite Pending
> data set : 
> {panel:title=B1}
> A;
> B;F
> C;A
> D;E
> E;
> F;C
> {panel}
> {panel:title=BK}
> A;1
> B;2
> F;4
> {panel}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-8063) OOM planning a certain query

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-8063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-8063.
---
Resolution: Fixed

Fixed in DRILL-7523

> OOM planning a certain query
> 
>
> Key: DRILL-8063
> URL: https://issues.apache.org/jira/browse/DRILL-8063
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning  Optimization
>Affects Versions: 1.19.0
>Reporter: James Turton
>    Assignee: Vova Vysotskyi
>Priority: Critical
>
> This looks like an infinite planning bug in Calcite.  To reproduce, copy the 
> two referenced TPCH Parquet files from contrib/data/tpch-sample-data/parquet/ 
> to dfs.tmp then run the following. Uncommenting the `magic_fix` column is 
> just one of the changes that can be made to make the query planning succeed.
>  
> {code:java}
> select
> p_brand,
> -- 'foobar' as magic_fix,
> case
> when f1 then v1
> else null
> end as `m_1`,
> case
> when f1 then v2
> else null
> end as `m_2`
> from
> (select
>  part.`p_brand`,
>sum(t.l_extendedprice) as v1,
>avg(t.l_extendedprice) as v2,
>  true as f1
>  from
>   dfs.tmp.`lineitem.parquet` `t`
>  inner join dfs.tmp.`part.parquet` part on `t`.`l_partkey` = 
> part.`p_partkey`
>  group by part.`p_brand`) as `t2`; {code}
>  
>  
> Stack trace snippet:
>  
> {code:java}
> 2021-12-01 13:12:15,172 [1e58a77f-0a5d-22b5-47f6-4c51bc31dbe6:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Cat
> astrophic Failure Occurred, exiting. Information message: Unable to handle 
> out of memory condition in Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> at 
> java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
> at 
> java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:168)
> at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
> at java.base/java.lang.String.valueOf(String.java:2951)
> at java.base/java.lang.StringBuilder.append(StringBuilder.java:168)
> at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
> ...{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-7722) CREATE VIEW with LATERAL UNNEST creates an invalid view

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7722.
---
Resolution: Fixed

Fixed in DRILL-7523

> CREATE VIEW with LATERAL UNNEST creates an invalid view
> ---
>
> Key: DRILL-7722
> URL: https://issues.apache.org/jira/browse/DRILL-7722
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser
>Affects Versions: 1.17.0
>Reporter: Matevž Bradač
>    Assignee: Vova Vysotskyi
>Priority: Blocker
>
> Creating a view from a query containing LATERAL UNNEST results in a view that 
> cannot be parsed by the engine. The generated view contains superfluous 
> parentheses, thus the failed parsing.
> {code:bash|title=a simple JSON database}
> $ cat /tmp/t.json
> [{"name": "item_1", "related": ["id1"]}, {"name": "item_2", "related": 
> ["id1", "id2"]}, {"name": "item_3", "related": ["id2"]}]
> {code}
> {code:SQL|title=drill query, working}
> SELECT
>   item.name,
>   relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
>   SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
>  name rels
> 0  item_1  id1
> 1  item_2  id1
> 2  item_2  id2
> 3  item_3  id2
> {code}
> {code:SQL|title=create a drill view from the above query}
> CREATE VIEW dfs.tmp.unnested_view AS
> SELECT
>   item.name,
>   relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
>   SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
> {code}
> {code:bash|title=contents of view file}
> # note the extra parentheses near LATERAL and FROM
> $ cat /tmp/unnested_view.view.drill
> {
>   "name" : "unnested_view",
>   "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS 
> `item`\nINNER JOIN LATERAL((SELECT *\nFROM (UNNEST(`item`.`related`)) AS `i` 
> (`rels`))) AS `relations` ON TRUE",
>   "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
>   }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
>   } ],
>   "workspaceSchemaPath" : [ ]
> }
> {code}
> {code:SQL|title=query the view}
> SELECT * FROM dfs.tmp.unnested_view
> PARSE ERROR: Failure parsing a view your query is dependent upon.
> SQL Query: SELECT `item`.`name`, `relations`.*
> FROM `dfs`.`tmp`.`t.json` AS `item`
> INNER JOIN LATERAL((SELECT *
> FROM (UNNEST(`item`.`related`)) AS `i` (`rels`))) AS `relations` ON TRUE
>  ^
> [Error Id: fd816a27-c2c5-4c2a-b6bf-173ab37eb693 ]
> {code}
> If the view is "fixed" by editing the generated JSON and removing the extra 
> parentheses, e.g.
> {code:bash|title=fixed view}
> $ cat /tmp/fixed_unnested_view.view.drill
> {
>   "name" : "fixed_unnested_view",
>   "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS 
> `item`\nINNER JOIN LATERAL(SELECT *\nFROM UNNEST(`item`.`related`) AS `i` 
> (`rels`)) AS `relations` ON TRUE",
>   "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
>   }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
>   } ],
>   "workspaceSchemaPath" : [ ]
> }
> {code}
> then querying works as expected:
> {code:sql|title=fixed view query}
> SELECT * FROM dfs.tmp.fixed_unnested_view
>  name rels
> 0  item_1  id1
> 1  item_2  id1
> 2  item_2  id2
> 3  item_3  id2
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-7526) Assertion Error when only type is used with schema in table function

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7526.
---
Resolution: Fixed

> Assertion Error when only type is used with schema in table function
> 
>
> Key: DRILL-7526
> URL: https://issues.apache.org/jira/browse/DRILL-7526
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Arina Ielchiieva
>    Assignee: Vova Vysotskyi
>Priority: Major
>
> {{org.apache.drill.TestSchemaWithTableFunction}}
> {noformat}
>   @Test
>   public void testWithTypeAndSchema() {
> String query = "select Year from 
> table(dfs.`store/text/data/cars.csvh`(type=> 'text', " +
>   "schema=>'inline=(`Year` int)')) where Make = 'Ford'";
> queryBuilder().sql(query).print();
>   }
> {noformat}
> {noformat}
> Caused by: java.lang.AssertionError: BOOLEAN
>   at 
> org.apache.calcite.sql.type.SqlTypeExplicitPrecedenceList.compareTypePrecedence(SqlTypeExplicitPrecedenceList.java:140)
>   at org.apache.calcite.sql.SqlUtil.bestMatch(SqlUtil.java:687)
>   at 
> org.apache.calcite.sql.SqlUtil.filterRoutinesByTypePrecedence(SqlUtil.java:656)
>   at 
> org.apache.calcite.sql.SqlUtil.lookupSubjectRoutines(SqlUtil.java:515)
>   at org.apache.calcite.sql.SqlUtil.lookupRoutine(SqlUtil.java:435)
>   at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:240)
>   at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:218)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5640)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5627)
>   at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1692)
>   at 
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl(ProcedureNamespace.java:53)
>   at 
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3129)
>   at 
> org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3111)
>   at 
> org.apache.drill.exec.planner.sql.conversion.DrillValidator.validateFrom(DrillValidator.java:63)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3383)
>   at 
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
>   at 
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1009)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:969)
>   at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:944)
>   at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:651)
>   at 
> org.apache.drill.exec.planner.sql.conversion.SqlConverter.validate(SqlConverter.java:189)
>   at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:648)
>   at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196)
>   at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:170)
>   at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
>   at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
>   at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
>   at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
>   at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:590)
>   at

Re: [VOTE] Release Apache Drill 1.20.2 - RC1

[Vova Vysotskyi]

Hi all,

Checked signatures and hash, submitted several queries, and no issues were 
found.

+1 (binding)

Kind regards,
Volodymyr Vysotskyi

On 2022/08/01 12:50:42 Charles Givre wrote:
> Downloaded RC1.  Unzipped tarball, ran test queries.  LGTM +1.
> 
> Vote +1 (binding)
> 
> > On Aug 1, 2022, at 6:33 AM, James Turton  wrote:
> > 
> > I'd like to propose the second release candidate (RC1) of Apache Drill, 
> > version 1.20.2. The release candidate covers a total of 23 resolved Jiras 
> > since 1.20.1 [1]. Thanks to everyone who contributed to this release and to 
> > Jingchuan Hu for his help in preparing the release.
> > 
> > The tarball artifacts are hosted at [2] and the maven artifacts are hosted 
> > at [3]. This release candidate is based on commit 
> > 3b924b778990c41bf2c15a917097c038a10faf5d located at [4].
> > 
> > Please download and try out the release.
> > 
> > [ ] +1
> > [ ] +0
> > [ ] -1
> > 
> > ✅ Launch Hadoop 3 build under Java 8 using drill-embedded on Linux, check 
> > sys.version, run a CTAS, check the web UI.
> > ✅ Launch Hadoop 2 build under Java 8 using drill-embedded on Linux, check 
> > sys.version, run a CTAS.
> > ✅ Launch Hadoop 3 build under Java 8 using drill-embedded on Windows 10, 
> > run a CTAS.
> > ✅ Check which Hadoop and Netty jars are present in the Hadoop 3 build.
> > ✅ Check which Hadoop and Netty jars are present in the Hadoop 2 build.
> > 
> > I vote +1 (binding).
> > 
> > [1] 
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12351742
> >  
> > [2] https://dist.apache.org/repos/dist/dev/drill/1.20.2-rc1/
> > [3] https://repository.apache.org/content/repositories/orgapachedrill-1101/
> > [4] https://github.com/jnturton/drill/commits/drill-1.20.2
> 
> 

[jira] [Created] (DRILL-8272) Skip MAP column without children when creating parquet tables

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8272:
-

 Summary: Skip MAP column without children when creating parquet 
tables
 Key: DRILL-8272
 URL: https://issues.apache.org/jira/browse/DRILL-8272
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (DRILL-512) Tpch2 fails to decorrelate

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-512.
--
Fix Version/s: (was: Future)
   Resolution: Fixed

Looks like it works for the current master version

> Tpch2 fails to decorrelate
> --
>
> Key: DRILL-512
> URL: https://issues.apache.org/jira/browse/DRILL-512
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning  Optimization
>Reporter: Jacques Nadeau
>Priority: Minor
>
> On top of Optiq 0.6, TPCH2 fails to remove the CorrelatorRel from the logical 
> plan.  Line 1091 of RelDecorrelator is supposed to be populating the map 
> between correlation variables and output positions.  It doesn't look like 
> that is happening.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8255) Update Drill-Calcite version to include fix for CALCITE-4992

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8255:
-

 Summary: Update Drill-Calcite version to include fix for 
CALCITE-4992
 Key: DRILL-8255
 URL: https://issues.apache.org/jira/browse/DRILL-8255
 Project: Apache Drill
  Issue Type: Task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (DRILL-8245) Project pushdown depends on rules order and might not happen

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8245:
-

 Summary: Project pushdown depends on rules order and might not 
happen
 Key: DRILL-8245
 URL: https://issues.apache.org/jira/browse/DRILL-8245
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


When the ProjectRomoveRule deletes the project before applying the rule that 
pushes the project to scan, Drill converts scan to DrillScanRel and adds 
explicitly star column so all columns will be read.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8237) Limit is not pushed for MSSQL

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8237:
-

 Summary: Limit is not pushed for MSSQL
 Key: DRILL-8237
 URL: https://issues.apache.org/jira/browse/DRILL-8237
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


[~dzamo] has noticed that the following test case will fail
{code:java}
  @Test
  public void testLimitPushDownWithOffset() throws Exception {
String query = "select person_id, first_name from mssql.dbo.person limit 
100 offset 10";
queryBuilder()
  .sql(query)
  .planMatcher()
  .include("Jdbc\\(.*SELECT TOP \\(110\\)")
  .include("Limit\\(")
  .match();
  }
{code}
because the limit wasn't pushed down.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8234) Register rules only from plugins used in the query

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8234:
-

 Summary: Register rules only from plugins used in the query
 Key: DRILL-8234
 URL: https://issues.apache.org/jira/browse/DRILL-8234
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Currently, rules from all enabled plugins are collected for query planning. It 
could cause issues when one of the plugins became invalid.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8216) Use EVF-based JSON reader for Values operator

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8216:
-

 Summary: Use EVF-based JSON reader for Values operator
 Key: DRILL-8216
 URL: https://issues.apache.org/jira/browse/DRILL-8216
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


The newer Calcite version simplifies and removes casts for literals. It causes 
wrong results for drillTypeOf and sqlTypeOf functions, since the Values 
operator uses an old JSON reader which reads integers with bigInt type.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8214) Replace EnumerableTableScan usage with LogicalTableScan

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8214:
-

 Summary: Replace EnumerableTableScan usage with LogicalTableScan
 Key: DRILL-8214
 URL: https://issues.apache.org/jira/browse/DRILL-8214
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Newer Calcite version returns LogicalTableScan instead of EnumerableTableScan 
in RelOptTableImpl, so Drill shouldn't rely on this class also and 
LogicalTableScan where possible to avoid planning issues.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8213) Replace deprecated RelNode.getRows with RelNode.estimateRowCount

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8213:
-

 Summary: Replace deprecated RelNode.getRows with 
RelNode.estimateRowCount
 Key: DRILL-8213
 URL: https://issues.apache.org/jira/browse/DRILL-8213
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


In the newer Calcite version RelNode.getRows was removed, so replacing its 
usage with RelNode.estimateRowCount



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8212) Join queries fail with StackOverflowError

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8212:
-

 Summary: Join queries fail with StackOverflowError
 Key: DRILL-8212
 URL: https://issues.apache.org/jira/browse/DRILL-8212
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


With the newer Calcite version, some join queries fail with StackOverflowError. 
In the new version, Calcite uses selectivity code for computing cost for some 
conditions.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8211) Replace deprecated RelNode.getChildExps with Project.getProjects

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8211:
-

 Summary: Replace deprecated RelNode.getChildExps with 
Project.getProjects
 Key: DRILL-8211
 URL: https://issues.apache.org/jira/browse/DRILL-8211
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


In the newer Calcite version RelNode.getChildExps was removed, so replacing it 
with Project.getProjects



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8210) Add substring convertlet

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8210:
-

 Summary: Add substring convertlet
 Key: DRILL-8210
 URL: https://issues.apache.org/jira/browse/DRILL-8210
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Newer Calcite requires adding convertlet for substring to prevent using 
ReflectiveConvertletTable, so queries that use substring will fail with 
{noformat}
Caused by: java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.calcite.sql2rel.ReflectiveConvertletTable.lambda$registerOpTypeMethod$3(ReflectiveConvertletTable.java:140)
at 
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:62)
at 
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:5352)
at 
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4547)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
at 
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:5180)
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8209) Introduce rule for converting join with distinct input to semi-join

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8209:
-

 Summary: Introduce rule for converting join with distinct input to 
semi-join
 Key: DRILL-8209
 URL: https://issues.apache.org/jira/browse/DRILL-8209
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Newer Calcite changed the order of applying rules. AggregateRemoveRule is 
applied before SemiJoinRule, so SemiJoinRule cannot be applied later, since 
aggregate is pruned from planning.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8208) Create builder for SqlSelect

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8208:
-

 Summary: Create builder for SqlSelect
 Key: DRILL-8208
 URL: https://issues.apache.org/jira/browse/DRILL-8208
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Never Calcite version adds more fields to the constructor of SqlSelect. In most 
cases, nulls are passed for these new arguments. Using builder will reduce the 
number of places where it should be added.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (DRILL-8192) Cassandra queries fail when enabled Mongo plugin

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8192:
-

 Summary: Cassandra queries fail when enabled Mongo plugin
 Key: DRILL-8192
 URL: https://issues.apache.org/jira/browse/DRILL-8192
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8187) Dialect factory returns ANSI SQL dialect for BigQuery

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8187:
-

 Summary: Dialect factory returns ANSI SQL dialect for BigQuery
 Key: DRILL-8187
 URL: https://issues.apache.org/jira/browse/DRILL-8187
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


See CALCITE-5064 for details.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8151) Add support for more ElasticSearch and Cassandra data types

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8151:
-

 Summary: Add support for more ElasticSearch and Cassandra data 
types
 Key: DRILL-8151
 URL: https://issues.apache.org/jira/browse/DRILL-8151
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: thinking of our Ukrainian friends

[Vova Vysotskyi]

Thank you all for the support!

Kind regards,
Volodymyr Vysotskyi

On 2022/02/25 11:52:27 James Turton wrote:
> Great to hear that you're okay!
> 
> On 2022/02/25 13:37, Vitalii Diravka wrote:
> > We are trying to be in a safe place, but the second day from time to time
> > we hear explosions near Kiev.
> > I hope it will be finished soon!
> >
> > Thank you all!
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Thu, Feb 24, 2022 at 4:29 PM Charles Givre  wrote:
> >
> >> I would also like to express my sympathy and support for Arina, Vova,
> >> Vitalii, Igor, Anton and the people of Ukraine.
> >> -- C
> >>
> >>> On Feb 24, 2022, at 5:07 AM, James Turton  wrote:
> >>>
> >>> I too would like to express my sympathy and solidarity.
> >>>
> >>> On 2022/02/24 11:43, Z0ltrix wrote:
>  oh my goodness i hope this will end soon.
>  Stay safe!
> 
>  --- Original Message ---
> 
>  luoc  schrieb am Donnerstag, 24. Februar 2022 um
> >> 10:24:
> > Vitalii and Vova are my Ukrainian friends, hopefully they will stay
> >> safe as well.
> >> On Feb 24, 2022, at 14:39, Ted Dunning ted.dunn...@gmail.com wrote:
> >>
> >> For commercial historical reasons many of the people who have
> >> contributed
> >> to Drill live in Ukraine.
> >>
> >> My heart is with them tonight. I hope they stay safe.
> >>
> 
> 

[jira] [Created] (DRILL-8137) Prevent reading union inputs after query cancellation

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8137:
-

 Summary: Prevent reading union inputs after query cancellation
 Key: DRILL-8137
 URL: https://issues.apache.org/jira/browse/DRILL-8137
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release Apache Drill 1.20.0 - RC1

[Vova Vysotskyi]

Hi James!

Thanks for doing the RC so rapidly!

I was verifying the previous RC, and after announcing this one switched to it. 
The release should be based on the commit generated by Maven 
([maven-release-plugin] prepare release drill-XXX), and for the previous 
release candidate that was so, but for this one, commit id refers to the 
previous commit: (DRILL-8126: Ignore OAuth Parameter in Storage Plugin).
Running select * from sys.version; query from Drill also returns that commit, 
but in this case, it is strange that the version was 1.20.0 since those changes 
weren't committed in the branch which head is 
73a829a5a0eb21fc35d6cfd878310b7069135ecd...

Kind regards,
Volodymyr Vysotskyi

On 2022/02/08 14:54:36 James Turton wrote:
> Hi all
> 
>  Note from the release manager.
> 
> I'll undertake to add an Hadoop 2 release candidate shortly.  I have 
> checked and the issue found in RC0 (DRILL-8126) is fixed.  That is the 
> only change between this RC and the previous one.
> 
> - Thank, James
> 
> I'd like to propose the second release candidate (RC1) of Apache Drill, 
> version 1.20.0.
> 
> The release candidate covers a total of 106 resolved JIRAs [1]. Thanks 
> to everyone who contributed to this release.
> 
> The tarball artifacts are hosted at [2] and the maven artifacts are 
> hosted at [3].
> 
> This release candidate is based on commit 
> 73a829a5a0eb21fc35d6cfd878310b7069135ecd located at [4].
> 
> Please download and try out the release.
> 
> [ ] +1
> [ ] +0
> [ ] -1
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12350301=12313820
> [2] https://dist.apache.org/repos/dist/dev/drill/drill-1.20.0-rc1/
> [3] https://repository.apache.org/content/repositories/orgapachedrill-1088/
> [4] https://github.com/jnturton/drill/commits/drill-1.20.0
> 

Re: [VOTE] Freeze for Drill 1.20

[Vova Vysotskyi]

+1

Kind regards,
Volodymyr Vysotskyi

On 2022/02/02 15:59:55 James Turton wrote:
> PR #2449 was merged and there are now zero Dependabot alerts against 
> master.
> 
> +1 for freezing from me.
> 
> On 2022/02/02 16:36, Charles Givre wrote:
> > Assuming we pass dependabot, big +1 from me!!   Great work everyone!
> > --C
> > 
> >> On Feb 2, 2022, at 9:35 AM, James Turton  wrote:
> >>
> >> Please vote again on the assumption that the very minor Postgresql 42.3.1 
> >> -> 42.3.2 PR will be merged, clearing the last Dependabot alert.  It 
> >> passed local testing so it looks like a safe bet.
> >>
> >>
> >> On 2022/01/30 01:51, Charles Givre wrote:
> >>> Hey James,
> >>> Alas... I'm afraid I'd have to give a -1 on this.  There are some 
> >>> dependabot alerts at the moment, which we really should resolve (or at 
> >>> least look at) before cutting a release.  One of which has is linked to a 
> >>> severe CVE. Also, I just submitted a VERY minor bug fix which I'd love to 
> >>> squeak into this release, but that's not urgent.
> >>> Best,
> >>> --C
> >>>
> >>>
>  On Jan 29, 2022, at 7:36 AM, James Turton  wrote:
> 
>  Hello Dev community
> 
>  Not a moment too soon, we've finally dispatched the last issues holding 
>  back 1.20!  Here's a big thank you from the release manager to everyone 
>  who helped to push us forward to this point.  I'm sure I'm not the only 
>  one receiving the "When it's coming??" questions. As an interesting bit 
>  of trivia, there have been about 9 months separating recent releases and 
>  it has now been about 8 months since 1.19.  Who knew we were so 
>  consistent ;-) ?
> 
>  Please vote for or against a feature freeze on the master branch.  I 
>  assume only critical bug or vulnerability fixes get freeze immunity?
> 
>  Thank you
>  James
> 
> >>
> > 
> 

[jira] [Resolved] (DRILL-8058) NPE: Cannot invoke "org.apache.calcite.rel.core.TableScan.getTable()" because "scan" is null

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-8058.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/493ac43af92f165f31e6f6ca3182bd1f324823e3

> NPE: Cannot invoke "org.apache.calcite.rel.core.TableScan.getTable()" because 
> "scan" is null
> 
>
> Key: DRILL-8058
> URL: https://issues.apache.org/jira/browse/DRILL-8058
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Iceberg
>Affects Versions: 1.19.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
>  Labels: iceberg, storage
> Fix For: Future
>
>
> Checked in Drill embedded the query form 
> _TestE2EUnnestAndLateral#testMultipleBatchesLateral_WithLimitInParent_  test 
> case:
> {code:java}
> SELECT customer.c_name, avg(orders.o_totalprice) AS avgPrice FROM 
> dfs.`/{custom_path}/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.lateraljoin.TestE2EUnnestAndLateral/root/lateraljoin/multipleFiles`
>  
> customer, LATERAL (SELECT t.ord.o_totalprice as o_totalprice FROM 
> UNNEST(customer.c_orders) t(ord) 
> WHERE t.ord.o_totalprice > 10 LIMIT 2) orders GROUP BY customer.c_name; 
> {code}
> But it gives the following error:
> {code:java}
> Caused by: java.lang.NullPointerException: Cannot invoke 
> "org.apache.calcite.rel.core.TableScan.getTable()" because "scan" is null
> at 
> org.apache.drill.exec.planner.common.DrillRelOptUtil.getDrillTable(DrillRelOptUtil.java:691)
> at 
> org.apache.drill.exec.store.iceberg.plan.IcebergPluginImplementor.canImplement(IcebergPluginImplementor.java:101)
> at 
> org.apache.drill.exec.store.plan.rule.PluginConverterRule.matches(PluginConverterRule.java:64)
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.matchRecurse(VolcanoRuleCall.java:263)
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.match(VolcanoRuleCall.java:247)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.fireRules(VolcanoPlanner.java:1566)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1840)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:848)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:864)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:92)
> at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:329) 
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8114) Prevent applying Iceberg project on non-Iceberg tables

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8114:
-

 Summary: Prevent applying Iceberg project on non-Iceberg tables
 Key: DRILL-8114
 URL: https://issues.apache.org/jira/browse/DRILL-8114
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


When running complex queries on non-Iceberg tables, some Iceberg rules are 
still applied, so it might cause whole the query to fail



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8111) Remove lombok usage

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8111:
-

 Summary: Remove lombok usage
 Key: DRILL-8111
 URL: https://issues.apache.org/jira/browse/DRILL-8111
 Project: Apache Drill
  Issue Type: Task
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


See https://lists.apache.org/thread/gdv21h4m9omoqojhbpc4pcxtmyoqwrm6 for 
details.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8103) Unable to use view permissions for non-pam auth or when the view is stored on s3

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8103:
-

 Summary: Unable to use view permissions for non-pam auth or when 
the view is stored on s3
 Key: DRILL-8103
 URL: https://issues.apache.org/jira/browse/DRILL-8103
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: Future






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8073) Add support for persistent table and storage aliases

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8073:
-

 Summary: Add support for persistent table and storage aliases
 Key: DRILL-8073
 URL: https://issues.apache.org/jira/browse/DRILL-8073
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8053) Reduce Docker image size

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8053:
-

 Summary: Reduce Docker image size
 Key: DRILL-8053
 URL: https://issues.apache.org/jira/browse/DRILL-8053
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Exclude unwanted dependencies and optimize docker image



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8050) Fix license-maven-plugin unknown file extension warnings

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8050:
-

 Summary: Fix license-maven-plugin unknown file extension warnings 
 Key: DRILL-8050
 URL: https://issues.apache.org/jira/browse/DRILL-8050
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


When running check for license, `license-maven-plugin` prints the following 
warnings:
{noformat}
[INFO] --- license-maven-plugin:3.0:check (default) @ drill-root ---
[INFO] Checking licenses...
Warning:  Unknown file extension: /home/runner/work/drill/drill/.dockerignore
Warning:  Unknown file extension: 
/home/runner/work/drill/drill/contrib/storage-splunk/src/test/resources/logback-test.xml.bak
Warning:  Unknown file extension: 
/home/runner/work/drill/drill/contrib/format-httpd/src/test/resources/httpd/multiformat.access_log
Warning:  Unknown file extension: 
/home/runner/work/drill/drill/contrib/storage-jdbc/src/test/resources/mysql_config_override/mysql_override.cnf
Warning:  Unknown file extension: 
/home/runner/work/drill/drill/contrib/storage-cassandra/src/test/resources/queries.cql
Warning:  Unknown file extension: 
/home/runner/work/drill/drill/contrib/storage-druid/src/test/resources/druid/environment
Warning:  Unknown file extension: /home/runner/work/drill/drill/lombok.config
Warning:  Unknown file extension: /home/runner/work/drill/drill/hooks/push
Warning:  Unknown file extension: /home/runner/work/drill/drill/hooks/build
Warning:  Unable to find a comment style definition for some files. You may 
want to add a custom mapping for the relevant file extensions.
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8049) Make rule names for Iceberg plugin instances unique

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8049:
-

 Summary: Make rule names for Iceberg plugin instances unique
 Key: DRILL-8049
 URL: https://issues.apache.org/jira/browse/DRILL-8049
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


As [~dzamo] noticed, when defining multiple storage plugins with Iceberg 
format, queries fail with the following error:
{noformat}
Caused by: java.lang.AssertionError: Rule's description should be unique; 
existing 
rule=VertexDrelConverterRuleICEBERG.iceberg(in:ICEBERG.iceberg,out:LOGICAL); 
new rule=VertexDrelConverterRuleICEBERG.iceberg(in:ICEBERG.iceberg,out:LOGICAL)
at 
org.apache.calcite.plan.AbstractRelOptPlanner.mapRuleDescription(AbstractRelOptPlanner.java:152)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.addRule(VolcanoPlanner.java:459)
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:405)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:351)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:245)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:308)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:173)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
... 1 common frames omitted
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (DRILL-8027) Format plugin for Apache Iceberg

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-8027.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/ec76ad05680612b84147993d66312f040430cac0

> Format plugin for Apache Iceberg
> 
>
> Key: DRILL-8027
> URL: https://issues.apache.org/jira/browse/DRILL-8027
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.20.0
>    Reporter: Vova Vysotskyi
>    Assignee: Vova Vysotskyi
>Priority: Major
>  Labels: plugin
> Fix For: 1.20.0
>
>
> Implement a format plugin for Apache Iceberg.
> Plugin should be able to:
> - support reading data from Iceberg tables in Parquet, Avro, and ORC formats
> - push down fields used in the project
> - push down supported filter expressions
> - spit and parallelize reading tasks
> - provide a way for specifying Iceberg-specific configurations
> - read specific snapshot versions if configured
> - read table metadata (entries, files, history, snapshots, manifests, 
> partitions, etc.)
> - support schema provisioning



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8042) Select star from MongoDB with aggregated pipeline fails with empty $project error

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8042:
-

 Summary: Select star from MongoDB with aggregated pipeline fails 
with empty $project error
 Key: DRILL-8042
 URL: https://issues.apache.org/jira/browse/DRILL-8042
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


See https://github.com/apache/drill/issues/2367 for details



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8041) Fix mongo scan spec deserialization

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8041:
-

 Summary: Fix mongo scan spec deserialization
 Key: DRILL-8041
 URL: https://issues.apache.org/jira/browse/DRILL-8041
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


See https://github.com/apache/drill/issues/2355 for details.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (DRILL-8027) Format plugin for Apache Iceberg

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8027:
-

 Summary: Format plugin for Apache Iceberg
 Key: DRILL-8027
 URL: https://issues.apache.org/jira/browse/DRILL-8027
 Project: Apache Drill
  Issue Type: New Feature
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Implement a format plugin for Apache Iceberg.
Plugin should be able to:
- support reading data from Iceberg tables in Parquet, Avro, and ORC formats
- push down fields used in the project
- push down supported filter expressions
- spit and parallelize reading tasks
- provide a way for specifying Iceberg-specific configurations
- read specific snapshot versions if configured
- read table metadata (entries, files, history, snapshots, manifests, 
partitions, etc.)
- support schema provisioning



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-8010) Build fails with unresolved incubator-iceberg dependency

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-8010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-8010.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/0c9451e6720e5028e1187067cc6d1957ff998bef

> Build fails with unresolved incubator-iceberg dependency
> 
>
> Key: DRILL-8010
> URL: https://issues.apache.org/jira/browse/DRILL-8010
> Project: Apache Drill
>  Issue Type: Bug
>    Reporter: Vova Vysotskyi
>    Assignee: Vova Vysotskyi
>Priority: Major
>
> As [~mrymar] noticed, build fails with the following error:
> {noformat}
> [ERROR] Failed to execute goal on project drill-iceberg-metastore: Could not 
> resolve dependencies for project 
> org.apache.drill.metastore:drill-iceberg-metastore:jar:1.20.0-SNAPSHOT: The 
> following artifacts could not be resolved: 
> com.github.apache.incubator-iceberg:iceberg-parquet:jar:93d51b9, 
> com.github.apache.incubator-iceberg:iceberg-data:jar:93d51b9, 
> com.github.apache.incubator-iceberg:iceberg-core:jar:93d51b9, 
> com.github.apache.incubator-iceberg:iceberg-common:jar:93d51b9, 
> com.github.apache.incubator-iceberg:iceberg-api:jar:93d51b9: Failure to find 
> com.github.apache.incubator-iceberg:iceberg-parquet:jar:93d51b9 in 
> https://conjars.org/repo was cached in the local repository, resolution will 
> not be reattempted until the update interval of conjars has elapsed or 
> updates are forced
> {noformat}
> Iceberg was moved out from the incubator, so it is likely that the old 
> JitPack dependency was revoked, but it wasn't rebuilt because the repo URL 
> was changed.
> The solution is to use a newer non-incubator version.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-8010) Build fails with unresolved incubator-iceberg dependency

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8010:
-

 Summary: Build fails with unresolved incubator-iceberg dependency
 Key: DRILL-8010
 URL: https://issues.apache.org/jira/browse/DRILL-8010
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Error:
{noformat}
[ERROR] Failed to execute goal on project drill-iceberg-metastore: Could not 
resolve dependencies for project 
org.apache.drill.metastore:drill-iceberg-metastore:jar:1.20.0-SNAPSHOT: The 
following artifacts could not be resolved: 
com.github.apache.incubator-iceberg:iceberg-parquet:jar:93d51b9, 
com.github.apache.incubator-iceberg:iceberg-data:jar:93d51b9, 
com.github.apache.incubator-iceberg:iceberg-core:jar:93d51b9, 
com.github.apache.incubator-iceberg:iceberg-common:jar:93d51b9, 
com.github.apache.incubator-iceberg:iceberg-api:jar:93d51b9: Failure to find 
com.github.apache.incubator-iceberg:iceberg-parquet:jar:93d51b9 in 
https://conjars.org/repo was cached in the local repository, resolution will 
not be reattempted until the update interval of conjars has elapsed or updates 
are forced
{noformat}

Iceberg was moved out from the incubator, so it is likely that the old JitPack 
dependency was revoked, but it wasn't rebuilt because the repo URL was changed.

The solution is to use a newer non-incubator version.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7843) Fix NPE due running TestZookeeperClient

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7843.
---
Resolution: Duplicate

> Fix NPE due running TestZookeeperClient
> ---
>
> Key: DRILL-7843
> URL: https://issues.apache.org/jira/browse/DRILL-7843
> Project: Apache Drill
>  Issue Type: Test
>  Components: Tools, Build  Test
>Affects Versions: 1.18.0
>Reporter: Vitalii Diravka
>Priority: Minor
> Fix For: Future
>
>
> There is recurring NPE error (13x times) during running _TestZookeeperClient_ 
>  test case:
> {code:java}
>  {code}
>  
> {code:java}
> {code}
> _[INFO] Running org.apache.drill.exec.coord.zk.TestZookeeperClient 
> 2169java.lang.NullPointerException 2170 at 
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:269)
>  2171 at 
> org.apache.zookeeper.server.ZKDatabase.fastForwardDataBase(ZKDatabase.java:251)
>  2172 at 
> org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:583)
>  2173 at 
> org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:546)
>  2174 at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.shutdown(NIOServerCnxnFactory.java:929)
>  2175 at 
> org.apache.curator.test.TestingZooKeeperMain.close(TestingZooKeeperMain.java:178)
>  2176 at 
> org.apache.curator.test.TestingZooKeeperServer.stop(TestingZooKeeperServer.java:118)
>  2177 at 
> org.apache.curator.test.TestingZooKeeperServer.close(TestingZooKeeperServer.java:130)
>  2178 at org.apache.curator.test.TestingServer.close(TestingServer.java:178) 
> 2179 at 
> org.apache.drill.exec.coord.zk.TestZookeeperClient.tearDown(TestZookeeperClient.java:92)
>  2180 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2181 at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> 2182 at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  2183 at java.lang.reflect.Method.invoke(Method.java:498) 2184 at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>  2185 at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  2186 at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>  2187 at 
> mockit.integration.junit4.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:49)
>  2188 at 
> mockit.integration.junit4.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:29)
>  2189 at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
>  2190 at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) 
> 2191 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) 2192 at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>  2193 at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>  2194 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 2195 at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 2196 at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 2197 at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 2198 at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 2199 at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) 2200 at 
> org.junit.runners.Suite.runChild(Suite.java:128) 2201 at 
> org.junit.runners.Suite.runChild(Suite.java:27) 2202 at 
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 2203 at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 2204 at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 2205 at 
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 2206 at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 2207 at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) 2208 at 
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) 2209 at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>  2210 at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
>  2211 at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
>  2212 at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>  2213 at 
> org.apache.maven.surefire.ju

[jira] [Created] (DRILL-8002) Expose clear error message for the case of errors from ElasticSearch

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-8002:
-

 Summary: Expose clear error message for the case of errors  from 
ElasticSearch
 Key: DRILL-8002
 URL: https://issues.apache.org/jira/browse/DRILL-8002
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Errors, thrown from storage plugins that rely on "enumerable" framework have an 
unclear error message
{noformat}
Exception happened when executing generated code
{noformat}

But the actual one is on the bottom of the stack trace.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7990) Unable to disable requireTail in HTTP storage plugin API

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7990.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/c19c4195b5e99478da837611b3c8622dccea0840

> Unable to disable requireTail in HTTP storage plugin API
> 
>
> Key: DRILL-7990
> URL: https://issues.apache.org/jira/browse/DRILL-7990
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.20.0
>    Reporter: Vova Vysotskyi
>    Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.20.0
>
>
> Regression after refactoring for DRILL-7988



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7990) Unable to disable requireTail in HTTP storage plugin API

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7990:
-

 Summary: Unable to disable requireTail in HTTP storage plugin API
 Key: DRILL-7990
 URL: https://issues.apache.org/jira/browse/DRILL-7990
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Regression after refactoring for DRILL-7988



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7988) Add credentials provider support for API connections in HTTP plugin

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7988:
-

 Summary: Add credentials provider support for API connections in 
HTTP plugin
 Key: DRILL-7988
 URL: https://issues.apache.org/jira/browse/DRILL-7988
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Currently, for the HTTP plugin, it is possible to use a credentials provider 
with a proxy username and a proxy password.
But it is possible to specify credentials for specific API connections, so it 
should also be able to use credentials provider.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7986) Drill crashes when using the streaming connection with arrays

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7986:
-

 Summary: Drill crashes when using the streaming connection with 
arrays
 Key: DRILL-7986
 URL: https://issues.apache.org/jira/browse/DRILL-7986
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


When using streaming HTTP connection for queries that return varchar arrays 
within multiple batches, Drillbit may crash with SIGSEGV fatal error:
{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0001166a7593, pid=66979, tid=0x7603
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 
1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode bsd-amd64 
compressed oops)
# Problematic frame:
# J 9781 C2 io.netty.buffer.DrillBuf.getInt(I)I (15 bytes) @ 0x0001166a7593 
[0x0001166a7560+0x33]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x7f95ddbbb800):  JavaThread 
"1eee85ef-2a8d-7c8e-65e4-aaba7bc5c4c6:frag:0:0" daemon [_thread_in_Java, 
id=30211, stack(0x7e21e000,0x7e31e000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 
0x7f95a20bbcc4

Registers:
RAX=0x00015e91e3c0, RBX=0x00015e91e3c0, RCX=0x00077a95df10, 
RDX=0xc0e0ccc4
RSP=0x7e31c4a0, RBP=0x7e31c518, RSI=0x00077a95df10, 
RDI=0x0128
R8 =0x, R9 =0x, R10=0x7f95e12af000, 
R11=0xc0e0ccc4
R12=0x, R13=0x7e31c4c8, R14=0x7e31c568, 
R15=0x7f95ddbbb800
RIP=0x0001166a7593, EFLAGS=0x00010206, ERR=0x0004
  TRAPNO=0x000e

Top of Stack: (sp=0x7e31c4a0)
0x7e31c4a0:    000162da1160
0x7e31c4b0:   7e31c518 000114807da0
0x7e31c4c0:   000114807da0 c0e0ccc4
0x7e31c4d0:   00077a95df10 7e31c4d8
0x7e31c4e0:   000162d9b028 7e31c568
0x7e31c4f0:   000162d9b3d8 
0x7e31c500:   000162d9b0b0 7e31c4c8
0x7e31c510:   7e31c568 7e31c5b0
0x7e31c520:   000114807e70 
0x7e31c530:    
0x7e31c540:    
0x7e31c550:    c0e0ccc4
0x7e31c560:   00077a95df10 0007704ebb20
0x7e31c570:   7e31c570 000162d95e36
0x7e31c580:   7e31c5d8 000162d98e48
0x7e31c590:    000162d95e78
0x7e31c5a0:   7e31c568 7e31c5d8
0x7e31c5b0:   7e31c630 0001148080e0
0x7e31c5c0:    
0x7e31c5d0:   00077a95dfe0 0007704db008
0x7e31c5e0:   00077a98b638 00077a98b638
0x7e31c5f0:   7e31c5f0 000162d95f05
0x7e31c600:   7e31c640 000162d98e48
0x7e31c610:    000162d95f28
0x7e31c620:   7e31c5d8 7e31c640
0x7e31c630:   7e31c690 0001148081b4
0x7e31c640:   0007704db008 0006c1fc6020
0x7e31c650:   7e31c650 00016104cd0c
0x7e31c660:   7e31c6a8 00016104f9f0
0x7e31c670:    00016104cd90
0x7e31c680:   7e31c640 7e31c6a0
0x7e31c690:   7e31c6f0 00011480835d 

Instructions: (pc=0x0001166a7593)
0x0001166a7573:   90 0f 1f 84 00 00 00 00 00 66 66 66 90 48 81 ec
0x0001166a7583:   18 00 00 00 48 89 6c 24 10 4c 8b 56 28 4c 63 da
0x0001166a7593:   43 8b 04 1a 48 83 c4 10 5d 85 05 5e 7a be f0 c3
0x0001166a75a3:   f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 f4 

Register to memory mapping:

RAX={method} {0x00015e91e3c0} 'getInt' '(I)I' in 'io/netty/buffer/DrillBuf'
RBX={method} {0x00015e91e3c0} 'getInt' '(I)I' in 'io/netty/buffer/DrillBuf'
RCX=0x00077a95df10 is an oop
io.netty.buffer.DrillBuf 
 - klass: 'io/netty/buffer/DrillBuf'
RDX=0xc0e0ccc4 is an unknown value
RSP=0x7e31c4a0 is pointing into the stack for thread: 0x7f95ddbbb800
RBP=0x7e31c518 is pointing into the stack for thread: 0x7f95ddbbb800
RSI=0x00077a95df10 is an oop
io.netty.buffer.DrillBuf 
 - klass: 'io/netty/buffer/DrillBuf'
RDI=0x0128 is an unknown value
R8 =0x is an unknown value
R9

[jira] [Created] (DRILL-7985) Support Mongo aggregate, union, project, limit, sort pushdowns

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7985:
-

 Summary: Support Mongo aggregate, union, project, limit, sort 
pushdowns
 Key: DRILL-7985
 URL: https://issues.apache.org/jira/browse/DRILL-7985
 Project: Apache Drill
  Issue Type: New Feature
Affects Versions: 1.20.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Add support for more mongo pushdowns:
- aggregate
- project (including expressions)
- limit (including offset)
- union / union all
- sort



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7972) ClassCastException when joining RDBMS and Elasticsearch tables

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7972:
-

 Summary: ClassCastException when joining RDBMS and Elasticsearch 
tables
 Key: DRILL-7972
 URL: https://issues.apache.org/jira/browse/DRILL-7972
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


See details here: https://github.com/apache/drill/issues/2274



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7971) Support Elasticsearch authentication

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7971:
-

 Summary: Support Elasticsearch authentication
 Key: DRILL-7971
 URL: https://issues.apache.org/jira/browse/DRILL-7971
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.20.0


Currently, it is unable to query secured Elasticsearch instance, it fails with 
the following error:
{noformat}
missing authentication credentials for REST request [/_alias]
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7939) Kafka Key Avro Schema can not cast right

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7939.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/36ac8f32a318c5bcbe27a62d04910ab16389b2cf

> Kafka Key Avro Schema can not cast right 
> -
>
> Key: DRILL-7939
> URL: https://issues.apache.org/jira/browse/DRILL-7939
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Kafka
>Affects Versions: 1.19.0
>Reporter: cdmikechen
>Priority: Major
> Fix For: 1.19.0
>
> Attachments: kafka-avro-key.jpg
>
>
> Using drill 1.19.0-snapshot to query kafka avro message, If kafka key data is 
> avro, drill can not return data right.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Vova Vysotskyi]

Hi Laurent,

* I have provided you with administrator permissions in Apache Drill Jira, so 
you should be able to update the current release date and create a new release 
(and do some batch operations for tickets if required, like updating "Fix 
version", etc.).
* I have access to that Twitter account (but not only I), please send an 
announcement about the release in a private message to @ApacheDrill and I’ll 
post it.
* Done: https://reporter.apache.org/addrelease.html?drill

Kind regards,
Volodymyr Vysotskyi

On 2021/06/10 07:41:46, Laurent Goujon  wrote: 
> Thanks Vova.> 
> 
> I checked that the docker image and the maven artifacts have been published.> 
> 
> I also created a pull request for updating the website:> 
> https://github.com/apache/drill/pull/2257/commits> 
> It contains the javadoc changes, but the commits are separated so it should> 
> be easy for people to review the changes and the blog announcement. Please> 
> let me know if I forgot an important feature.> 
> 
> Still looking at the release documentation (> 
> https://github.com/apache/drill/blob/master/docs/dev/Release.md), there are> 
> a couple of things it seems I won't be able to do but would expect a PMC to> 
> do:> 
> * Updating the current release date in JIRA and creating a new release> 
> * Post the announcement on Twitter> 
> * Post the release date on https://reporter.apache.org/addrelease.html?drill> 
> 
> As for the website update, can someone confirm that those instructions at> 
> https://github.com/apache/drill/blob/gh-pages/README.md#uploading-to-the-apache-website-drill-committers-only>
>  
> are still correct?> 
> 
> Laurent> 
> 
> 
> On Wed, Jun 9, 2021 at 2:44 PM Vova Vysotskyi  wrote:> 
> 
> > Good news, that issue was resolved so I have published artifacts:> 
> > https://dist.apache.org/repos/dist/release/drill/drill-1.19.0/.> 
> >> 
> > We will have to update the release instruction since now it would work> 
> > only with moving artifacts from dist/dev to dist/release> 
> >> 
> > Kind regards,> 
> > Volodymyr Vysotskyi> 
> >> 
> > On 2021/06/09 20:16:12, Ted Dunning  wrote:> 
> > > Thanks.>> 
> > >> 
> > > I figured you would be ahead of me on this.>> 
> > >> 
> > > On Wed, Jun 9, 2021 at 12:27 PM  wrote:>> 
> > >> 
> > > > Hello Ted,>> 
> > > >>> 
> > > > Yes, initially I tried both options.>> 
> > > > I have also left a comment on the ticket, hope it will be resolved> 
> > soon.>> 
> > > >>> 
> > > > Kind regards,>> 
> > > > Volodymyr Vysotskyi>> 
> > > >>> 
> > > > On 2021/06/09 19:04:02, Ted Dunning  wrote:>> 
> > > > > Vova,>>> 
> > > > >>> 
> > > > > Gavin responded on INFRA-21981 to the effect that upload should go> 
> > to>> 
> > > > the>>> 
> > > > > dev side and then svn mv should be used to move to the release> 
> > side.>>> 
> > > > >>> 
> > > > > Is that what you tried to do?>>> 
> > > > >>> 
> > > > >>> 
> > > > >>> 
> > > > > On Wed, Jun 9, 2021 at 10:25 AM  wrote:>>> 
> > > > >>> 
> > > > > > I have some issues, will deploy after>>> 
> > > > > > https://issues.apache.org/jira/browse/INFRA-21981 is fixed.>>> 
> > > > > >>>> 
> > > > > > On 2021/06/09 16:27:12, vo...@apache.org wrote:>>> 
> > > > > > > Hello Laurent,>>>> 
> > > > > > >>>> 
> > > > > > > I’ll publish them later today.>>>> 
> > > > > > >>>> 
> > > > > > > Kind regards,>>>> 
> > > > > > > Volodymyr Vysotskyi>>>> 
> > > > > > >>>> 
> > > > > > >>>> 
> > > > > > > On 2021/06/09 04:39:50, Laurent Goujon > 
> > wrote: >>>> 
> > > > > > > > Hi,> >>>> 
> > > > > > > > >>>> 
> > > > > > > > May I kindly ask for a PMC to push the RC1 artifacts to the> 
> > dist>>> 
> > > > > > repository> >>>> 
> > > > > > > > per instructions at> >>>> 
> > > > > > > >> 
> > https://github.com/apache/drill/blob/master/docs/dev/Release.md?>>> 
> > > > >>>> 
> > > > > > > > >>>> 
> > > > > > > > The artifacts are available at> >>>> 
> > > > > > > > https://home.apache.org/~laurent/drill/releases/1.19.0/rc1/>> 
> > >>>> 
> > > > > > > > >>>> 
> > > > > > > > Laurent> >>>> 
> > > > > > > > >>>> 
> > > > > > > > On Tue, Jun 8, 2021 at 9:36 PM Laurent Goujon <> 
> > la...@dremio.com>>>> 
> > > > > > wrote:> >>>> 
> > > > > > > > >>>> 
> > > > > > > > > Hi all,> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > The vote passes. Thanks to everyone who has tested the> 
> > release>>> 
> > > > >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > >>>> 
> > > > > > > > >>>> 
> > > > > > > > > candidate and given their comments and votes. Final tally:>> 
> > >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > 3x +1 (binding): Laurent, Ted, Vova> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > No 0s or -1s.> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > I'll start the process for pushing the release artifacts> 
> > and>> 
> > > > send>>> 
> > > > > > an> >>>> 
> > > > > > > > > announcement once propagated.> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > Kind regards,> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > > Laurent> >>>> 
> > > > > > > > >> >>>> 
> > > > > > > > >>>> 
> > > > >>> 
> > >> 
> 

Re: [RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Vova Vysotskyi]

Good news, that issue was resolved so I have published artifacts: 
https://dist.apache.org/repos/dist/release/drill/drill-1.19.0/.

We will have to update the release instruction since now it would work only 
with moving artifacts from dist/dev to dist/release

Kind regards,
Volodymyr Vysotskyi

On 2021/06/09 20:16:12, Ted Dunning  wrote: 
> Thanks.> 
> 
> I figured you would be ahead of me on this.> 
> 
> On Wed, Jun 9, 2021 at 12:27 PM  wrote:> 
> 
> > Hello Ted,> 
> >> 
> > Yes, initially I tried both options.> 
> > I have also left a comment on the ticket, hope it will be resolved soon.> 
> >> 
> > Kind regards,> 
> > Volodymyr Vysotskyi> 
> >> 
> > On 2021/06/09 19:04:02, Ted Dunning  wrote:> 
> > > Vova,>> 
> > >> 
> > > Gavin responded on INFRA-21981 to the effect that upload should go to> 
> > the>> 
> > > dev side and then svn mv should be used to move to the release side.>> 
> > >> 
> > > Is that what you tried to do?>> 
> > >> 
> > >> 
> > >> 
> > > On Wed, Jun 9, 2021 at 10:25 AM  wrote:>> 
> > >> 
> > > > I have some issues, will deploy after>> 
> > > > https://issues.apache.org/jira/browse/INFRA-21981 is fixed.>> 
> > > >>> 
> > > > On 2021/06/09 16:27:12, vo...@apache.org wrote:>> 
> > > > > Hello Laurent,>>> 
> > > > >>> 
> > > > > I’ll publish them later today.>>> 
> > > > >>> 
> > > > > Kind regards,>>> 
> > > > > Volodymyr Vysotskyi>>> 
> > > > >>> 
> > > > >>> 
> > > > > On 2021/06/09 04:39:50, Laurent Goujon  wrote: >>> 
> > > > > > Hi,> >>> 
> > > > > > >>> 
> > > > > > May I kindly ask for a PMC to push the RC1 artifacts to the dist>> 
> > > > repository> >>> 
> > > > > > per instructions at> >>> 
> > > > > > https://github.com/apache/drill/blob/master/docs/dev/Release.md?>> 
> > >>> 
> > > > > > >>> 
> > > > > > The artifacts are available at> >>> 
> > > > > > https://home.apache.org/~laurent/drill/releases/1.19.0/rc1/> >>> 
> > > > > > >>> 
> > > > > > Laurent> >>> 
> > > > > > >>> 
> > > > > > On Tue, Jun 8, 2021 at 9:36 PM Laurent Goujon >> 
> > > > wrote:> >>> 
> > > > > > >>> 
> > > > > > > Hi all,> >>> 
> > > > > > >> >>> 
> > > > > > > The vote passes. Thanks to everyone who has tested the release>> 
> > >>> 
> > > > > > >> >>> 
> > > > > > >>> 
> > > > > > >>> 
> > > > > > > candidate and given their comments and votes. Final tally:> >>> 
> > > > > > >> >>> 
> > > > > > > 3x +1 (binding): Laurent, Ted, Vova> >>> 
> > > > > > >> >>> 
> > > > > > > No 0s or -1s.> >>> 
> > > > > > >> >>> 
> > > > > > > I'll start the process for pushing the release artifacts and> 
> > send>> 
> > > > an> >>> 
> > > > > > > announcement once propagated.> >>> 
> > > > > > >> >>> 
> > > > > > > Kind regards,> >>> 
> > > > > > >> >>> 
> > > > > > > Laurent> >>> 
> > > > > > >> >>> 
> > > > > > >>> 
> > >> 
> 

[jira] [Created] (DRILL-7950) Drill classpath includes both avatica-core:jar:1.15.0 and avatica:jar:1.17.0

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7950:
-

 Summary: Drill classpath includes both avatica-core:jar:1.15.0 and 
avatica:jar:1.17.0
 Key: DRILL-7950
 URL: https://issues.apache.org/jira/browse/DRILL-7950
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi


Drill classpath now includes both jars fat jar for shaded avativa 
(`avatica`:jar:1.17.0) and jar for `avatica-core` with a different version - 
1.15.0.

The latest one should be excluded from the classpath since it conflicts with 
classes from the fat jar and has some vulnerabilities that we attempted to fix 
in DRILL-7918.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7945) Unable to patch Guava classes

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7945:
-

 Summary: Unable to patch Guava classes
 Key: DRILL-7945
 URL: https://issues.apache.org/jira/browse/DRILL-7945
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vitalii Diravka
 Fix For: 1.19.0


When starting Drill in embedded mode, logs have the following line:
{noformat}
2021-06-03 21:59:07,711 [main] WARN  o.a.drill.common.util.GuavaPatcher - 
Unable to patch Guava classes: [source error] 
format(java.lang.String,java.lang.Object[]) not found in 
com.google.common.base.Preconditions
{noformat}

Most likely it is a regression after DRILL-7904.

The importance of this issue is that all 3rd party libraries that initially 
relied on patched methods from {{Preconditions}} will stop working because of 
method not found errors during the runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7929) Update Kafka unit tests to use ClusterTest

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7929:
-

 Summary: Update Kafka unit tests to use ClusterTest
 Key: DRILL-7929
 URL: https://issues.apache.org/jira/browse/DRILL-7929
 Project: Apache Drill
  Issue Type: Task
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7917) Update Kafka to the latest version

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7917:
-

 Summary: Update Kafka to the latest version
 Key: DRILL-7917
 URL: https://issues.apache.org/jira/browse/DRILL-7917
 Project: Apache Drill
  Issue Type: Task
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


Update Kafka to the 2.8.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7814) Add Limit Pushdown to MongoDB Storage Plugin

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7814.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/f05e75adfd04f90852a12cae9e502b204ece47f5

> Add Limit Pushdown to MongoDB Storage Plugin
> 
>
> Key: DRILL-7814
> URL: https://issues.apache.org/jira/browse/DRILL-7814
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - MongoDB
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.19.0
>
>
> Drill's MongoDB plugin does not push limits down to Mongo.  This PR adds this 
> capability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7888) query.json returns an incorrect error message when the query fails

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7888.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/40460be6b958585982c3c7d2b7a35e89ccbfe34a

> query.json returns an incorrect error message when the query fails
> --
>
> Key: DRILL-7888
> URL: https://issues.apache.org/jira/browse/DRILL-7888
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.19.0
>    Reporter: Vova Vysotskyi
>    Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.19.0
>
>
> When submitting a query that should fail to \{{query.json}} endpoint, it 
> always returns the same response regardless of the actual failure reason:
> {code:json}
> {
> "errorMessage": "Current context not Array but root"
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7888) query.json returns an incorrect error message when the query fails

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7888:
-

 Summary: query.json returns an incorrect error message when the 
query fails
 Key: DRILL-7888
 URL: https://issues.apache.org/jira/browse/DRILL-7888
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


When submitting a query that should fail to \{{query.json}} endpoint, it always 
returns the same response regardless of the actual failure reason:
{code:json}
{
"errorMessage": "Current context not Array but root"
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7887) JDBC plugin returns incorrect precision for SUM(DECIMAL) queries

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7887:
-

 Summary: JDBC plugin returns incorrect precision for SUM(DECIMAL) 
queries
 Key: DRILL-7887
 URL: https://issues.apache.org/jira/browse/DRILL-7887
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


The following query:
{code:sql}
SELECT sum(decimal_field * smallint_field) AS `order_total`
FROM mysql.`drill_mysql_test`.person e
{code}
returns {{order_total}} field with {{DECIMAL(42, 2)}} type, but Drill supports 
max presision of 38.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7868) Restore correct master branch state

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7868:
-

 Summary: Restore correct master branch state
 Key: DRILL-7868
 URL: https://issues.apache.org/jira/browse/DRILL-7868
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


Restore correct master branch state after 
https://github.com/apache/drill/commit/54a0ec6225ff275844ab055cfc8af1db2c429904 
commit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7867) Add EMPTY_IF_NULL null handling for complex UDFs

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7867:
-

 Summary: Add EMPTY_IF_NULL null handling for complex UDFs
 Key: DRILL-7867
 URL: https://issues.apache.org/jira/browse/DRILL-7867
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


Introduce {{NullHandling.EMPTY_IF_NULL}} to be used for UDFs that return 
{{ComplexWriter}}, so it wouldn't be required to introduce two versions of UDF 
nor nullable and non-nullable inputs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7865) Unable to apply DrillPushLimitToScanRule due to NPE

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7865:
-

 Summary: Unable to apply DrillPushLimitToScanRule due to NPE
 Key: DRILL-7865
 URL: https://issues.apache.org/jira/browse/DRILL-7865
 Project: Apache Drill
  Issue Type: Bug
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


At some point of planning, DrillPushLimitToScanRule fails with NPE, here is the 
stack trace:
{noformat}
WARN  o.a.d.e.p.l.DrillPushLimitToScanRule - Exception while using the pruned 
partitions.
java.lang.NullPointerException: null
at 
org.apache.drill.exec.store.dfs.easy.EasyGroupScan.initFromSelection(EasyGroupScan.java:192)
at 
org.apache.drill.exec.store.dfs.easy.EasyGroupScan.access$1300(EasyGroupScan.java:77)
at 
org.apache.drill.exec.store.dfs.easy.EasyGroupScan$EasyGroupScanFilterer.build(EasyGroupScan.java:414)
at 
org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.applyLimit(AbstractGroupScanWithMetadata.java:470)
at 
org.apache.drill.exec.planner.logical.DrillPushLimitToScanRule.doOnMatch(DrillPushLimitToScanRule.java:98)
at 
org.apache.drill.exec.planner.logical.DrillPushLimitToScanRule$1.onMatch(DrillPushLimitToScanRule.java:56)
at 
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:561)
at 
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
at 
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:257)
at 
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at 
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:216)
at 
org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:203)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:394)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:351)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:338)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:252)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:308)
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:173)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
This stack trace may be observed for example for 
{{TestImageRecordReader.testLimitPushdown()}} test.
Though the query is not failed, it may prevent applying limit pushdown for some 
cases.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7857) Fix misc GitHub Actions failures

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7857:
-

 Summary: Fix misc GitHub Actions failures
 Key: DRILL-7857
 URL: https://issues.apache.org/jira/browse/DRILL-7857
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


GitHub Action CI jobs fail with several types of errors:
{noformat}
2021-02-06T10:01:33.1192508Z [ERROR] 
org.apache.drill.exec.server.rest.TestRestJson Time elapsed: 0.024 s <<< ERROR! 
2021-02-06T10:01:33.1202149Z java.lang.IllegalStateException: Cluster fixture 
setup failed 2021-02-06T10:01:33.1211983Z at 
org.apache.drill.exec.server.rest.TestRestJson.setup(TestRestJson.java:88) 
2021-02-06T10:01:33.1272734Z Caused by: java.net.BindException: Address already 
in use 2021-02-06T10:01:33.1277495Z at 
org.apache.drill.exec.server.rest.TestRestJson.setup(TestRestJson.java:88)
{noformat}
{noformat}
2021-02-03T05:31:09.2293240Z [ERROR] Tests run: 4, Failures: 1, Errors: 0, 
Skipped: 0, Time elapsed: 0.837 s <<< FAILURE! - in 
org.apache.drill.exec.util.ExecutorServiceUtilTest 2021-02-03T05:31:09.2305753Z 
[ERROR] org.apache.drill.exec.util.ExecutorServiceUtilTest.testCancelExecution 
Time elapsed: 0.25 s <<< FAILURE! 2021-02-03T05:31:09.2323911Z 
java.lang.AssertionError 2021-02-03T05:31:09.2329693Z at 
org.apache.drill.exec.util.ExecutorServiceUtilTest.testCancelExecution(ExecutorServiceUtilTest.java:203)
{noformat}
{noformat}
[INFO] Running org.apache.drill.exec.store.cassandra.CassandraQueryTest
13473
= Using optimized driver!!! =
13474
Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 28.87 s 
<<< FAILURE! - in org.apache.drill.exec.store.cassandra.CassandraQueryTest
13475
Error:  org.apache.drill.exec.store.cassandra.CassandraQueryTest  Time elapsed: 
0.016 s  <<< ERROR!
13476
{noformat}
{noformat}
Error occurred in starting fork, check output in log
11360
Error:  Process Exit Code: 137
11361
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: 
ExecutionException The forked VM terminated without properly saying goodbye. VM 
crash or System.exit called?
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7855) Provide a secure mechanism for specifying storage plugin credentials

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7855:
-

 Summary: Provide a secure mechanism for specifying storage plugin 
credentials
 Key: DRILL-7855
 URL: https://issues.apache.org/jira/browse/DRILL-7855
 Project: Apache Drill
  Issue Type: New Feature
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


Drill provides only a single way for specifying credentials for storage plugins.
Though all storage plugin credentials may be stored in Zookeeper, it may be 
unsafe to specify them directly in the plugin configs.

The aim of this ticket is to provide a more secured mechanism for specifying 
storage plugin credentials.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7550) Add Storage Plugin for Cassandra

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7550.
---
Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/75e4877b46380188b6545b99f188b764b2a576aa

> Add Storage Plugin for Cassandra
> 
>
> Key: DRILL-7550
> URL: https://issues.apache.org/jira/browse/DRILL-7550
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.18.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.19.0
>
>
> Apache Cassandra is a free and open-source, distributed, wide column store, 
> NoSQL database management system designed to handle large amounts of data 
> across many commodity servers, providing high availability with no single 
> point of failure. [1]
> This PR would enable Drill to query Cassandra data stores.
>  
> [1]: https://en.wikipedia.org/wiki/Apache_Cassandra



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7849) Display error message when cannot enable storage plugin

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7849:
-

 Summary: Display error message when cannot enable storage plugin
 Key: DRILL-7849
 URL: https://issues.apache.org/jira/browse/DRILL-7849
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.18.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


Currently, when an exception occurs during enabling the storage plugin, it is 
not shown in the UI, like nothing was done. It may be observed when enabling 
the RDBMS plugin with default configuration from any, storage plugins page or 
plugin configs page.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7848) Github Actions fails due to outdated package repository cache

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7848:
-

 Summary: Github Actions fails due to outdated package repository 
cache
 Key: DRILL-7848
 URL: https://issues.apache.org/jira/browse/DRILL-7848
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.19.0


Recent Github Actions jobs fail with the following error:
{noformat}
Err:122 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 
libsasl2-dev amd64 2.1.27~101-g0780600+dfsg-3ubuntu2.1
235
  404  Not Found [IP: 52.252.75.106 80]
236
E: Failed to fetch 
http://security.ubuntu.com/ubuntu/pool/main/c/cyrus-sasl2/libsasl2-dev_2.1.27~101-g0780600+dfsg-3ubuntu2.1_amd64.deb
  404  Not Found [IP: 52.252.75.106 80]
237
Fetched 24.4 MB in 3s (7162 kB/s)
238
E: Unable to fetch some archives, maybe run apt-get update or try with 
--fix-missing?
{noformat}

It is caused because of the outdated package repository cache for libsasl2-dev.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-5405) Add missing operator types without dependency on protobuf enum

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-5405.
---
Fix Version/s: (was: 2.0.0)
   1.19.0
   Resolution: Fixed

Fixed in 
https://github.com/apache/drill/commit/3d1bc2c07e620187bb251cf61bac1858f4dcceda

> Add missing operator types without dependency on protobuf enum
> --
>
> Key: DRILL-5405
> URL: https://issues.apache.org/jira/browse/DRILL-5405
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>    Assignee: Vova Vysotskyi
>Priority: Minor
> Fix For: 1.19.0
>
> Attachments: maprdb_sub_scan.JPG, unknown_operator.JPG
>
>
> Source - https://github.com/apache/drill/pull/214#discussion_r42687575
> When operators are not added into CoreOperatorType enum, they are displayed 
> on Web UI as UNKNOWN_OPERATOR.
> Screenshots:
> now -> unknown_operator.JPG
> should be -> maprdb_sub_scan.JPG
> Though plugins work fine without this changes, it's still would be nice to 
> display operator type on Web UI.
> But dependency on protobuf enum removes the ability for format plugins to 
> truly be drop-in pluggable.
> In this case as option we may need to refactor this interface to just return 
> a String and not use the protobuf enum, or have a separate "notes" field if 
> we want to have a generic filescan operator ID in this enum with extra data 
> to say what kind of scan we used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7785) Some hive tables fail with UndeclaredThrowableException

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7785.
---
Resolution: Invalid

> Some hive tables fail with UndeclaredThrowableException
> ---
>
> Key: DRILL-7785
> URL: https://issues.apache.org/jira/browse/DRILL-7785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Storage - Hive
>Affects Versions: 1.18.0
>Reporter: Abhishek Girish
>    Assignee: Vova Vysotskyi
>Priority: Major
>
> Query: 
> {code}
> Functional/hive/hive_storage/fileformats/orc/transactional/orc_table_clustered_bucketed.sql
> select * from hive_orc_transactional.orc_table_clustered_bucketed
> {code}
> Exception:
> {code}
> java.sql.SQLException: EXECUTION_ERROR ERROR: 
> java.lang.reflect.UndeclaredThrowableException
> Failed to setup reader: HiveDefaultRecordReader
> Fragment: 0:0
> [Error Id: 323434cc-7bd2-4551-94d4-a5925f6a66af on drill80:31010]
>   (org.apache.drill.common.exceptions.ExecutionSetupException) 
> java.lang.reflect.UndeclaredThrowableException
> 
> org.apache.drill.common.exceptions.ExecutionSetupException.fromThrowable():30
> 
> org.apache.drill.exec.store.hive.readers.HiveDefaultRecordReader.setup():257
> org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas():331
> org.apache.drill.exec.physical.impl.ScanBatch.internalNext():227
> org.apache.drill.exec.physical.impl.ScanBatch.next():298
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():111
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85
> org.apache.drill.exec.record.AbstractRecordBatch.next():170
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():323
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():310
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1669
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():310
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
>   Caused By (java.util.concurrent.ExecutionException) 
> java.lang.reflect.UndeclaredThrowableException
> 
> org.apache.drill.shaded.guava.com.google.common.util.concurrent.AbstractFuture.getDoneValue():553
> 
> org.apache.drill.shaded.guava.com.google.common.util.concurrent.AbstractFuture.get():534
> 
> org.apache.drill.shaded.guava.com.google.common.util.concurrent.FluentFuture$TrustedFuture.get():88
> 
> org.apache.drill.exec.store.hive.readers.HiveDefaultRecordReader.setup():252
> org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas():331
> org.apache.drill.exec.physical.impl.ScanBatch.internalNext():227
> org.apache.drill.exec.physical.impl.ScanBatch.next():298
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():111
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85
> org.apache.drill.exec.record.AbstractRecordBatch.next():170
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():323
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():310
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1669
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():310
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoo

[jira] [Created] (DRILL-7774) IS NOT NULL predicate fails with NPE for the case of non-existing columns with non-deterministic expression

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7774:
-

 Summary: IS NOT NULL predicate fails with NPE for the case of 
non-existing columns with non-deterministic expression
 Key: DRILL-7774
 URL: https://issues.apache.org/jira/browse/DRILL-7774
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.18.0


The following query fails with NPE:
{code:sql}
select count(*) as cnt from cp.`tpch/nation.parquet` where (case when random() 
= 1 then true else null end * t ) is not null
{code}

Error message:
{noformat}
apache drill> select count(*) as cnt from cp.`tpch/nation.parquet` where (case 
when random() = 1 then true else null end * t) is not null;
Error: SYSTEM ERROR: NullPointerException


Please, refer to logs for more information.

[Error Id: cf98ddc3-6e2b-404b-81ed-11188bd11225 on user515050-pc:31016] 
(state=,code=0)
java.sql.SQLException: SYSTEM ERROR: NullPointerException


Please, refer to logs for more information.

[Error Id: cf98ddc3-6e2b-404b-81ed-11188bd11225 on user515050-pc:31016]
at 
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:534)
at 
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:599)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
at 
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at 
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
at 
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
at 
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at 
org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
at sqlline.Commands.executeSingleQuery(Commands.java:1054)
at sqlline.Commands.execute(Commands.java:1003)
at sqlline.Commands.sql(Commands.java:967)
at sqlline.SqlLine.dispatch(SqlLine.java:734)
at sqlline.SqlLine.begin(SqlLine.java:541)
at sqlline.SqlLine.start(SqlLine.java:267)
at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM 
ERROR: NullPointerException


Please, refer to logs for more information.

[Error Id: cf98ddc3-6e2b-404b-81ed-11188bd11225 on user515050-pc:31016]
at 
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at 
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:

[jira] [Created] (DRILL-7761) Drill fails with OOM for the case of large filter conditions

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7761:
-

 Summary: Drill fails with OOM for the case of large filter 
conditions
 Key: DRILL-7761
 URL: https://issues.apache.org/jira/browse/DRILL-7761
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.18.0


Drill fails with OOM for the following query:
{code:sql}
SELECT * FROM (SELECT n.n_name AS name, n.n_nationkey AS  n_nationkey, 
cast(n.n_regionkey AS FLOAT) AS regionkey FROM cp.`/tpch/nation.parquet` n) 
WHERE ((name = 'A' AND ((regionkey >= 0.0 AND regionkey <= 120.0 AND  
n_nationkey = 0.005))) OR (name = 'B' AND ((regionkey >= 0.0  AND regionkey <= 
120.0  AND  n_nationkey = 0.005))) OR (name = 'C' AND ((regionkey >= 0.0  AND 
regionkey <= 120.0  AND  n_nationkey = 0.005))) OR (name = 'D' AND ((regionkey 
>= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 0.005))) OR (name = 'E' AND 
((regionkey >= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 0.005))) OR 
(name = 'F' AND ((regionkey >= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 
0.005))) OR (name = 'G' AND ((regionkey >= 0.0  AND regionkey <= 120.0  AND  
n_nationkey = 0.005))) OR (name = 'I' AND ((regionkey >= 0.0  AND regionkey <= 
120.0  AND  n_nationkey = 0.005))) OR (name = 'J' AND ((regionkey >= 0.0  AND 
regionkey <= 120.0  AND  n_nationkey = 0.005))) OR (name = 'K' AND ((regionkey 
>= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 0.005))) OR (name = 'L' AND 
((regionkey >= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 0.005))) OR 
(name = 'M' AND ((regionkey >= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 
0.005))) OR (name = 'N' AND ((regionkey >= 0.0  AND regionkey <= 120.0  AND  
n_nationkey = 0.005))) OR (name = 'O' AND ((regionkey >= 0.0  AND regionkey <= 
120.0  AND  n_nationkey = 0.005))) OR (name = 'P' AND ((regionkey >= 0.0  AND 
regionkey <= 120.0  AND  n_nationkey = 0.005))) OR (name = 'Q' AND ((regionkey 
>= 0.0  AND regionkey <= 120.0  AND  n_nationkey = 0.005
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7750) Drill fails to read KeyStore password from Credential provider

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7750.
---
Resolution: Fixed

Merged in 1b95c0a8cfce23e11596353a821a5216fd1a983d

> Drill fails to read KeyStore password from Credential provider
> --
>
> Key: DRILL-7750
> URL: https://issues.apache.org/jira/browse/DRILL-7750
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Fix For: 1.18
>
>
> When core-site.xml has keystore or truststore specific properties along with 
> Hadoop's CredentialProvider path, e.g.:
> {code}
> 
> 
> 
> ...
>  
>   ssl.server.truststore.location
>   /etc/conf/ssl_truststore
> 
> 
>   ssl.server.truststore.type
>   jks
> 
> 
>   ssl.server.truststore.reload.interval
>   1
> 
> 
>   ssl.server.keystore.location
>   /etc/conf/ssl_keystore
> 
> 
>   ssl.server.keystore.type
>   jks
> 
> 
>   hadoop.security.credential.provider.path
> jceks://file/etc/conf/ssl_server.jceks
> 
> {code}
> Drill fails to start.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7600) Move Compress File Function to Test Utility

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-7600.
---
Resolution: Fixed

> Move Compress File Function to Test Utility
> ---
>
> Key: DRILL-7600
> URL: https://issues.apache.org/jira/browse/DRILL-7600
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Minor
> Fix For: 1.18.0
>
>
> Several format plugins have unit tests to validate that the function works 
> with compressed files and there is a utility function which has been 
> cut/pasted in multiple plugins.  This story will create a utility function 
> that is accessible to all unit tests and modify unit tests to use the utility 
> function. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-6580) Upgrade Curator libraries

[Vova Vysotskyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vova Vysotskyi resolved DRILL-6580.
---
Resolution: Fixed

Fixed in DRILL-7702

> Upgrade Curator libraries
> -
>
> Key: DRILL-6580
> URL: https://issues.apache.org/jira/browse/DRILL-6580
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Bohdan Kazydub
>    Assignee: Vova Vysotskyi
>Priority: Major
>
> As of [DRILL-6534|https://issues.apache.org/jira/browse/DRILL-6534] Drill 
> uses explicit version ZooKeeper (ZK) and curator. ZK's version was upgraded 
> so it is better to upgrade Curator as it manages ZK.
> Note: I've tried to upgrade Curator to 2.12.0, 3.3.0 and 4.0.1 (has hard 
> dependency on ZK 3.5.x) and everything seems to work except that size of 
> curator-client artifact is much larger than the current one's (~2+ MB and 70 
> kB respectively) and as a result jdbc-all maxSize should be increased.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7741) Columns are missing when using convert_from function

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7741:
-

 Summary: Columns are missing when using convert_from function
 Key: DRILL-7741
 URL: https://issues.apache.org/jira/browse/DRILL-7741
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.18.0


The query returns wrong results when {{convert_from}} function and additional 
columns may appear in new batches. Here is a simple test case for this issue:
{code:java}
  @Test
  public void testConvertFromJson() throws Exception {
String fileName = "table.tsv";
try (BufferedWriter writer = new BufferedWriter(new FileWriter(new 
File(dirTestWatcher.getRootDir(), fileName {
  for (int i = 0; i < JSONRecordReader.DEFAULT_ROWS_PER_BATCH; i++) {
writer.write("{\"id\":\"1\"}\n");
  }
  writer.write("{\"id\":\"2\",\"v\":[\"abc\"]}");
}

String sql = "SELECT t.m.id AS id, t.m.v[0] v FROM \n" +
"(SELECT convert_from(columns[0], 'json') AS m FROM dfs.`%s`) t\n" +
"where t.m.id='2'";

testBuilder()
.sqlQuery(sql, fileName)
.unOrdered()
.baselineColumns("id", "v")
.baselineValues("2", "abc")
.go();
  }
{code}
Currently, the {{"v"}} column is null since {{OK_NEW_SCHEMA}} wasn't returned 
from the project when the complex writer was used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7739) Allow implicit casts from required to nullable data type

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7739:
-

 Summary: Allow implicit casts from required to nullable data type
 Key: DRILL-7739
 URL: https://issues.apache.org/jira/browse/DRILL-7739
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi


UDFs that accept nullable vector should be able to handle data from the 
required vector. Drill already has UDFs for converting required to nullable 
holders, so just function calls should be inserved where possible.

It will be also useful for vararg UDFs, where previously was required having 
the same data mode for all vararg arguments.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7738) Fix TestDynamicUDFSupport failure for GitHub Actions

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7738:
-

 Summary: Fix TestDynamicUDFSupport failure for GitHub Actions
 Key: DRILL-7738
 URL: https://issues.apache.org/jira/browse/DRILL-7738
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.18.0
Reporter: Vova Vysotskyi
Assignee: Vova Vysotskyi
 Fix For: 1.18.0


{{TestDynamicUDFSupport}} started failing again on CI.
This test uses Drill 1.13 jars for tests, but these jars were using 
http://repo.dremio.com/release/ repository, which is unavailable for now, so 
failure is seen when maven tries to fetch something from this repo.
Here is a relevant error with enabled logging:

{noformat}
2020-05-11T20:17:54.8044724Z 20:17:54.803 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - Failed to execute goal on project 
drill-udf: Could not resolve dependencies for project 
org.apache.drill.udf:drill-udf:jar:1.0: Failed to collect dependencies at 
org.apache.drill.exec:drill-java-exec:jar:1.13.0 -> 
org.apache.calcite:calcite-core:jar:1.15.0-drill-r0: Failed to read artifact 
descriptor for org.apache.calcite:calcite-core:jar:1.15.0-drill-r0: Could not 
transfer artifact org.apache.calcite:calcite-core:pom:1.15.0-drill-r0 from/to 
dremio-releases (http://repo.dremio.com/release/): Transfer failed for 
http://repo.dremio.com/release/org/apache/calcite/calcite-core/1.15.0-drill-r0/calcite-core-1.15.0-drill-r0.pom:
 Unknown host repo.dremio.com: Name or service not known -> [Help 1]
2020-05-11T20:17:54.8139658Z 20:17:54.813 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - 
2020-05-11T20:17:54.8144837Z 20:17:54.814 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - To see the full stack trace of the 
errors, re-run Maven with the -e switch.
2020-05-11T20:17:54.8148867Z 20:17:54.814 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - Re-run Maven using the -X switch to 
enable full debug logging.
2020-05-11T20:17:54.8151736Z 20:17:54.814 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - 
2020-05-11T20:17:54.8155634Z 20:17:54.815 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - For more information about the errors and 
possible solutions, please read the following articles:
2020-05-11T20:17:54.8160874Z 20:17:54.815 [main] ERROR 
o.a.d.exec.udf.dynamic.JarBuilder$1 - [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
2020-05-11T20:17:56.4581196Z [INFO] Running 
org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport
2020-05-11T20:17:56.4860577Z [ERROR] Tests run: 1, Failures: 1, Errors: 0, 
Skipped: 0, Time elapsed: 0.017 s <<< FAILURE! - in 
org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport
2020-05-11T20:17:56.4867090Z [ERROR] 
org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport  Time elapsed: 0.012 s  
<<< FAILURE!
2020-05-11T20:17:56.4871764Z java.lang.AssertionError: Build should be 
successful. expected:<0> but was:<1>
2020-05-11T20:17:56.4878002Zat 
org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport.buildJars(TestDynamicUDFSupport.java:909)
2020-05-11T20:17:56.4884892Zat 
org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport.buildAndStoreDefaultJars(TestDynamicUDFSupport.java:93)
{noformat}

This repository was removed in DRILL-6470, so using 1.14 or newer version 
should fix the failure.
But in fix will be used 1.17 to reduce dependencies versioning differences.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: compile issue with MapR repo is being worked

[Vova Vysotskyi]

Hi Ted,

Thanks for your help! It looks like for http protocol this issue was
resolved.

Kind regards,
Volodymyr Vysotskyi


On Mon, May 4, 2020 at 4:19 AM Charles Givre  wrote:

> Hi Ted,
> Thanks for your help.  You can view the logs here:
> https://github.com/apache/drill/pull/2067 <
> https://github.com/apache/drill/pull/2067> in the CI stuff.
> -- C
>
>
>
>
> > On May 3, 2020, at 9:16 PM, Ted Dunning  wrote:
> >
> > I will pass the word.
> >
> > Do you have logs?
> >
> >
> > On Sun, May 3, 2020 at 4:15 PM Charles Givre  wrote:
> >
> >> Hi Ted,
> >> Thanks for looking into this so quickly.  Unfortunately, I re-ran the CI
> >> jobs from github and it is still producing the same errors.
> >> Best,
> >> --C
> >>
> >>> On May 3, 2020, at 5:58 PM, Ted Dunning  wrote:
> >>>
> >>> It appears that the certificate issue is resolved.
> >>>
> >>> Can somebody verify this by doing a compilation?
> >>>
> >>> I have to add that based on the number of off-line and on-list pings I
> >> got
> >>> about this issue I can say that there were quite a few people compiling
> >>> Drill on a Sunday morning. That bodes well, I think, for community
> >> health.
> >>>
> >>>
> >>>
> >>> On Sun, May 3, 2020 at 11:27 AM Ted Dunning 
> >> wrote:
> >>>
> 
>  I just got word back that the team is looking at the issue.
> 
>  Not surprisingly, their first look indicates that the issue isn't what
> >> it
>  appears to be (i.e. not a bad cert)
> 
> 
> 
> >>
> >>
>
>

Re: More CI Issues

[Vova Vysotskyi]

Hi Charles,

Here is a Jira for migrating repositories to https: DRILL-7270
, but as you may see from
its comment, the repository owner was informed about certificates issues.
So it looks like some work for fixing this issue is in the progress and it
caused the unavailability of the repository.

Hope it will be fixed soon.

Kind regards,
Volodymyr Vysotskyi


On Sun, May 3, 2020 at 6:14 PM Charles Givre  wrote:

> One more thing...
> I tried changing the HTTP to HTTPS (which we should do anyway) and now it
> is getting certificate errors:
>
> [ERROR] Failed to execute goal on project drill-java-exec: Could not
> resolve dependencies for project
> org.apache.drill.exec:drill-java-exec:jar:1.18.0-SNAPSHOT: Failed to
> collect dependencies at org.kohsuke:libpam4j:jar:1.8-rev2: Failed to read
> artifact descriptor for org.kohsuke:libpam4j:jar:1.8-rev2: Could not
> transfer artifact org.kohsuke:libpam4j:pom:1.8-rev2 from/to mapr-releases (
> https://repository.mapr.com/maven/): Transfer failed for
> https://repository.mapr.com/maven/org/kohsuke/libpam4j/1.8-rev2/libpam4j-1.8-rev2.pom:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target -> [Help 1]
> 35540
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35540>[ERROR]
>
> 35541
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35541>[ERROR]
> To see the full stack trace of the errors, re-run Maven with the -e switch.
> 35542
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35542>[ERROR]
> Re-run Maven using the -X switch to enable full debug logging.
> 35543
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35543>[ERROR]
>
> 35544
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35544>[ERROR]
> For more information about the errors and possible solutions, please read
> the following articles:
> 35545
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35545>[ERROR]
> [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> 35546
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35546>[ERROR]
>
> 35547
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35547>[ERROR]
> After correcting the problems, you can resume the build with the command
> 35548
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35548>[ERROR]
>  mvn  -rf :drill-java-exec
> 35549
>  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640659179#step:6:35549>##[error]Process
> completed with exit code 1.
>
>
>
> > On May 3, 2020, at 11:02 AM, Charles Givre  wrote:
> >
> > Hello all,
> > In a recent PR that had nothing to do with this, I saw that the CI is
> now failing due to the following error.  I did some digging and found that
> the URL http://repository.mapr.com/maven/ <
> http://repository.mapr.com/maven/> is no longer accessible.  However with
> SSL https://repository.mapr.com/maven/ 
> IS available but with bad certificates.
> >
> > I'm not a maven with maven, but is there some way to easily fix this so
> that Drill isn't trying to pull from non-existent or broken MapR repos?
> > Best,
> > -- C
> >
> >
> > [ERROR] Failed to execute goal on project drill-java-exec: Could not
> resolve dependencies for project
> org.apache.drill.exec:drill-java-exec:jar:1.18.0-SNAPSHOT: Failed to
> collect dependencies at org.kohsuke:libpam4j:jar:1.8-rev2: Failed to read
> artifact descriptor for org.kohsuke:libpam4j:jar:1.8-rev2: Could not
> transfer artifact org.kohsuke:libpam4j:pom:1.8-rev2 from/to mapr-releases (
> http://repository.mapr.com/maven/ ):
> Transfer failed for
> http://repository.mapr.com/maven/org/kohsuke/libpam4j/1.8-rev2/libpam4j-1.8-rev2.pom
> <
> http://repository.mapr.com/maven/org/kohsuke/libpam4j/1.8-rev2/libpam4j-1.8-rev2.pom>
> 500 Proxy Error -> [Help 1]
> > 1094
> >  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640588163#step:6:1094>[ERROR]
>
> > 1095
> >  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640588163#step:6:1095>[ERROR]
> To see the full stack trace of the errors, re-run Maven with the -e switch.
> > 1096
> >  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640588163#step:6:1096>[ERROR]
> Re-run Maven using the -X switch to enable full debug logging.
> > 1097
> >  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640588163#step:6:1097>[ERROR]
>
> > 1098
> >  <
> https://github.com/apache/drill/pull/2067/checks?check_run_id=640588163#step:6:1098>[ERROR]
> For more 

Re: [QUESTION]: httpclient dependency

[Vova Vysotskyi]

e
> [INFO] |  |  \- com.googlecode.json-simple:json-simple:jar:1.1.1:compile
> [INFO] |  +- org.reflections:reflections:jar:0.9.10:compile
> [INFO] |  |  \- com.google.code.findbugs:annotations:jar:2.0.1:compile
> [INFO] |  +- org.javassist:javassist:jar:3.27.0-GA:compile
> [INFO] |  +-
> com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:compile
> [INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.10.3:compile
> [INFO] |  +- io.dropwizard.metrics:metrics-core:jar:4.0.2:compile
> [INFO] |  +- io.dropwizard.metrics:metrics-servlets:jar:4.0.2:compile
> [INFO] |  |  +-
> io.dropwizard.metrics:metrics-healthchecks:jar:4.0.2:compile
> [INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:4.0.2:compile
> [INFO] |  |  \- com.papertrail:profiler:jar:1.0.2:compile
> [INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:4.0.2:compile
> [INFO] |  \- io.dropwizard.metrics:metrics-jmx:jar:4.0.2:compile
> [INFO] +- commons-codec:commons-codec:jar:1.14:compile
> [INFO] +- io.netty:netty-handler:jar:4.0.48.Final:compile
> [INFO] |  +- io.netty:netty-buffer:jar:4.0.48.Final:compile
> [INFO] |  +- io.netty:netty-transport:jar:4.0.48.Final:compile
> [INFO] |  \- io.netty:netty-codec:jar:4.0.48.Final:compile
> [INFO] +- io.netty:netty-common:jar:4.0.48.Final:compile
> [INFO] +- org.apache.drill:drill-shaded-guava:jar:28.2-jre:compile
> [INFO] |  +-
> com.google.guava:listenablefuture:jar:.0-empty-to-avoid-conflict-with-guava:compile
> [INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
> [INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
> [INFO] |  +-
> com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
> [INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
> [INFO] +- org.slf4j:slf4j-api:jar:1.7.26:compile
> [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile
> [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile
> [INFO] +- org.slf4j:log4j-over-slf4j:jar:1.7.26:compile
> [INFO] +- commons-io:commons-io:jar:2.4:compile
> [INFO] +- org.jmockit:jmockit:jar:1.47:test
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] +- org.mockito:mockito-core:jar:2.23.4:test
> [INFO] |  +- net.bytebuddy:byte-buddy:jar:1.9.3:test
> [INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.9.3:test
> [INFO] |  \- org.objenesis:objenesis:jar:2.6:test
> [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:test
> [INFO] +- ch.qos.logback:logback-core:jar:1.2.3:test
> [INFO] +-
> de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic:jar:0.9.44:test
> [INFO] |  +- de.huxhorn.sulky:de.huxhorn.sulky.codec:jar:0.9.17:test
> [INFO] |  |  \- de.huxhorn.sulky:de.huxhorn.sulky.io:jar:0.9.17:test
> [INFO] |  +-
> de.huxhorn.lilith:de.huxhorn.lilith.logback.converter-classic:jar:0.9.44:test
> [INFO] |  |  +-
> de.huxhorn.lilith:de.huxhorn.lilith.data.converter:jar:0.9.44:test
> [INFO] |  |  +-
> de.huxhorn.lilith:de.huxhorn.lilith.logback.classic:jar:0.9.44:test
> [INFO] |  |  \-
> de.huxhorn.lilith:de.huxhorn.lilith.data.logging:jar:0.9.44:test
> [INFO] |  |+-
> de.huxhorn.sulky:de.huxhorn.sulky.formatting:jar:0.9.17:test
> [INFO] |  |\-
> de.huxhorn.lilith:de.huxhorn.lilith.data.eventsource:jar:0.9.44:test
> [INFO] |  +-
> de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-core:jar:0.9.44:test
> [INFO] |  |  \- de.huxhorn.lilith:de.huxhorn.lilith.sender:jar:0.9.44:test
> [INFO] |  \-
> de.huxhorn.lilith:de.huxhorn.lilith.data.logging.protobuf:jar:0.9.44:test
> [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
> [INFO] +- org.apache.kerby:kerb-client:jar:1.0.0:test
> [INFO] |  +- org.apache.kerby:kerby-config:jar:1.0.0:test
> [INFO] |  +- org.apache.kerby:kerb-common:jar:1.0.0:test
> [INFO] |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.0:test
> [INFO] |  \- org.apache.kerby:kerb-util:jar:1.0.0:test
> [INFO] +- org.apache.kerby:kerb-core:jar:1.0.0:test
> [INFO] |  \- org.apache.kerby:kerby-pkix:jar:1.0.0:test
> [INFO] |+- org.apache.kerby:kerby-asn1:jar:1.0.0:test
> [INFO] |\- org.apache.kerby:kerby-util:jar:1.0.0:test
> [INFO] \- org.apache.kerby:kerb-simplekdc:jar:1.0.0:test
> [INFO]\- org.apache.kerby:kerb-admin:jar:1.0.0:test
> [INFO]  +- org.apache.kerby:kerb-server:jar:1.0.0:test
> [INFO]  |  \- org.apache.kerby:kerb-identity:jar:1.0.0:test
> [INFO]  \- org.apache.kerby:kerby-xdr:jar:1.0.0:test
> [INFO]
> 
> [INFO] BUILD SUCCESS
> [INFO]
> 
> [INFO] Total time:  1.402 s
> [INFO] Finished at: 2020-04-23T20:30:50-04:00
> [INFO]
> 

Re: [QUESTION]: httpclient dependency

[Vova Vysotskyi]

Hi Charles,

Could you please share more details about your problem, for example, which
one dependency uses commons-httpclient, which errors you see, etc.

Kind regards,
Volodymyr Vysotskyi


On Wed, Apr 22, 2020 at 5:23 PM Charles Givre  wrote:

> Hi Volodymyr,
> Thanks for the response.  Do you have any suggestions as to how to fix
> this?  I don't really know enough Maven to fix this.  I've tried manually
> including the library in the pom, removing exclusions etc.
> Thanks,
> -- C
>
>
>
> > On Apr 20, 2020, at 4:15 AM, Vova Vysotskyi  wrote:
> >
> > Hi Charles,
> >
> > *commons-httpclient* was excluded since it has some vulnerabilities. But
> > this library was renamed to *org.apache.httpcomponents:httpclient* and
> > dependency to its latest version was added to the project, so it should
> be
> > used everywhere instead of the old one.
> >
> > By the way, *org.elasticsearch.client:elasticsearch-rest-client* 7.6.2
> has
> > a dependency on *org.apache.httpcomponents:httpclient*, but not
> > *commons-httpclient*, so it is strange that you have some issues with the
> > older library.
> >
> > Kind regards,
> > Volodymyr Vysotskyi
> >
> >
> > On Mon, Apr 20, 2020 at 4:11 AM Charles Givre  wrote:
> >
> >> Hello all,
> >> I am doing some work for an Elasticsearch (ES) storage plugin.  I
> recently
> >> rebased on the current master and found that broke my plugin due to a
> >> dependency exclusion.  Basically, the ES modules need the
> >> commons-httpclient which seems to be excluded by the HBase plugin. My
> >> question is can I "un-exclude" it or do something in Maven so that the
> code
> >> gets these dependencies?
> >> Thanks,
> >> -- C
>
>

Re: [QUESTION]: httpclient dependency

[Vova Vysotskyi]

Hi Charles,

*commons-httpclient* was excluded since it has some vulnerabilities. But
this library was renamed to *org.apache.httpcomponents:httpclient* and
dependency to its latest version was added to the project, so it should be
used everywhere instead of the old one.

By the way, *org.elasticsearch.client:elasticsearch-rest-client* 7.6.2 has
a dependency on *org.apache.httpcomponents:httpclient*, but not
*commons-httpclient*, so it is strange that you have some issues with the
older library.

Kind regards,
Volodymyr Vysotskyi


On Mon, Apr 20, 2020 at 4:11 AM Charles Givre  wrote:

> Hello all,
> I am doing some work for an Elasticsearch (ES) storage plugin.  I recently
> rebased on the current master and found that broke my plugin due to a
> dependency exclusion.  Basically, the ES modules need the
> commons-httpclient which seems to be excluded by the HBase plugin. My
> question is can I "un-exclude" it or do something in Maven so that the code
> gets these dependencies?
> Thanks,
> -- C