[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688612#comment-17688612 ] ASF GitHub Bot commented on DRILL-8117: --- kingswanwho commented on PR #2756: URL: https://github.com/apache/drill/pull/2756#issuecomment-1430077393 > @jnturton @kingswanwho Should we close the other PR? Yes, I closed another PR > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688611#comment-17688611 ] ASF GitHub Bot commented on DRILL-8117: --- kingswanwho closed pull request #2499: DRILL-8117: Upgrade unit tests to the cluster fixture framework URL: https://github.com/apache/drill/pull/2499 > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8401) Skip nested MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688526#comment-17688526 ] ASF GitHub Bot commented on DRILL-8401: --- jnturton opened a new pull request, #2757: URL: https://github.com/apache/drill/pull/2757 # [DRILL-8401](https://issues.apache.org/jira/browse/DRILL-8401): Skip nested MAP column without children when creating parquet tables ## Description This extends the work of [DRILL-8272](https://issues.apache.org/jira/browse/DRILL-8272) in order to handle nested empty MAPs which currently also break the Parquet writer. ## Documentation N/A ## Testing Extension of TestParquetWriter#testResultWithEmptyMap > Skip nested MAP column without children when creating parquet tables > > > Key: DRILL-8401 > URL: https://issues.apache.org/jira/browse/DRILL-8401 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.20.3 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > This extends the work of DRILL-8272 in order to handle nested empty MAPs > which currently also break the Parquet writer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688495#comment-17688495 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2756: URL: https://github.com/apache/drill/pull/2756#issuecomment-1429701817 One other question. Should we document this in the developer documentation? > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688493#comment-17688493 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2756: URL: https://github.com/apache/drill/pull/2756#issuecomment-1429699470 @jnturton @kingswanwho Should we close the other PR? > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688476#comment-17688476 ] ASF GitHub Bot commented on DRILL-8117: --- jnturton opened a new pull request, #2756: URL: https://github.com/apache/drill/pull/2756 # [DRILL-8117](https://issues.apache.org/jira/browse/DRILL-8117): Upgrade unit tests to the cluster fixture framework ## Description Upgrades various unit tests to the cluster fixture framework and replaces other instances of deprecated code usage. Methods client.alterSession() and client.alterSystem() run sql silently which would not throw exception, so it doesn't match the purpose for some test cases. And I still keep using run() here for consistence. For `TestInboundImpersonation` Because cluster fixture save client properties even client closed, use a dedicated cluster fixture for each test case to ensure the client fixture has clean properties. For `TestImpersonationMetadata` Only use one cluster from test start to the end will induce buffer overflow. So change BeforeClass to Before to start a new cluster for each test case. ## Documentation N/A ## Testing Existing unit tests. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688234#comment-17688234 ] ASF GitHub Bot commented on DRILL-8117: --- kingswanwho commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428986001 > Details Hi Charles, Thanks for the remind! Need James' help to investigate. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688230#comment-17688230 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428975963 @kingswanwho @jnturton It looks like Java 8 is still not liking the Hive unit tests. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688096#comment-17688096 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428502853 @kingswanwho Looks like we're still getting some errors on the Hive unit tests. I'm rerunning to see if it was a fluke or not. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688028#comment-17688028 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428284278 @jnturton @kingswanwho I'll review once CI passes later today. Looks good so far! > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688014#comment-17688014 ] ASF GitHub Bot commented on DRILL-8117: --- kingswanwho commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428239825 > The Hive unit tests are quite problematic for me locally due to more problems than you'd expect when I try to do test runs under JDK 8. Anyway, I think that I've fixed the two broken Hive impersonation tests and rebased on master in [this new PR to your branch](https://github.com/kingswanwho/drill/pull/3). I have merged and pushed, it should work well this time : -) > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688006#comment-17688006 ] ASF GitHub Bot commented on DRILL-8117: --- jnturton commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1428211191 The Hive unit tests are quite problematic for me locally due to more problems than you'd expect when I try to do test runs under JDK 8. Anyway, I think that I've fixed the two broken Hive impersonation tests and rebased on master in [this new PR to your branch](https://github.com/kingswanwho/drill/pull/3). > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688003#comment-17688003
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff commented on PR #2747:
URL: https://github.com/apache/drill/pull/2747#issuecomment-1428198156
@cgivre I'm not sure if this change is appropriate. I'd love to hear your
opinion.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
> Fix For: 1.21.0
>
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json";, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687930#comment-17687930 ] ASF GitHub Bot commented on DRILL-8117: --- jnturton commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1427913119 > > @jnturton Did you do a rebase with DRILL-8398 and DRILL-8400? I did think of that and now have, thanks. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687921#comment-17687921 ] ASF GitHub Bot commented on DRILL-8117: --- cgivre commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1427881798 > Let me see what's gone awry with the Hive tests... @jnturton Did you do a rebase with DRILL-8398 and DRILL-8400? > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687897#comment-17687897 ] ASF GitHub Bot commented on DRILL-8117: --- jnturton commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1427809955 Let me see what's gone awry with the Hive tests... > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[ https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687865#comment-17687865 ] ASF GitHub Bot commented on DRILL-8117: --- jnturton commented on PR #2499: URL: https://github.com/apache/drill/pull/2499#issuecomment-1427686523 I think this one is licked! I've requested a review from @cgivre to get it a once over from a pair of eyes belonging to a non-author. > Upgrade unit tests to the cluster fixture framework > --- > > Key: DRILL-8117 > URL: https://issues.apache.org/jira/browse/DRILL-8117 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.1 >Reporter: Jingchuan Hu >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > Upgrade various unit tests to the cluster fixture framework and replace other > instances of deprecated code usage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8398) Fix GitHub Actions to use proper JDK version
[ https://issues.apache.org/jira/browse/DRILL-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687703#comment-17687703 ] ASF GitHub Bot commented on DRILL-8398: --- jnturton merged PR #2754: URL: https://github.com/apache/drill/pull/2754 > Fix GitHub Actions to use proper JDK version > > > Key: DRILL-8398 > URL: https://issues.apache.org/jira/browse/DRILL-8398 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.0 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Blocker > Fix For: 1.21.0 > > > In one of the changes GitHub Actions `actions/setup-java@v2` action was > removed, and after that provided java version (11) was used instead of the > specified one in the > matrix.java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687621#comment-17687621 ] ASF GitHub Bot commented on DRILL-4232: --- cgivre merged PR #2599: URL: https://github.com/apache/drill/pull/2599 > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687620#comment-17687620 ] ASF GitHub Bot commented on DRILL-4232: --- cgivre commented on PR #2599: URL: https://github.com/apache/drill/pull/2599#issuecomment-1427120271 @Leon-WTF Thanks for this. @vvysotskyi Thanks for the review. Merging now! > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8316) Convert Druid Storage Plugin to EVF & V2 JSON Reader
[ https://issues.apache.org/jira/browse/DRILL-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687600#comment-17687600 ] ASF GitHub Bot commented on DRILL-8316: --- cgivre commented on PR #2657: URL: https://github.com/apache/drill/pull/2657#issuecomment-1427088906 @Z0ltrix Would you mind reviewing this? The unit tests pass, but I'm not 100% certain this is handling large data sets correctly. > Convert Druid Storage Plugin to EVF & V2 JSON Reader > > > Key: DRILL-8316 > URL: https://issues.apache.org/jira/browse/DRILL-8316 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Druid >Affects Versions: 1.20.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8399) MS Access Reader Misinterprets Data Types
[ https://issues.apache.org/jira/browse/DRILL-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687512#comment-17687512 ] ASF GitHub Bot commented on DRILL-8399: --- cgivre merged PR #2755: URL: https://github.com/apache/drill/pull/2755 > MS Access Reader Misinterprets Data Types > - > > Key: DRILL-8399 > URL: https://issues.apache.org/jira/browse/DRILL-8399 > Project: Apache Drill > Issue Type: Bug > Components: Format - MS Access >Affects Versions: 1.21.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 1.21.0 > > > The MS Access reader was assigning certain data types incorrectly, resulting > in various errors. This minor PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687511#comment-17687511
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1103724674
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
> Drill doesn't use streaming aggregate for distinct calls, so only hash agg
should be enough.
@vvysotskyi I see it checks aggregate.containsDistinctCall() in
StreamAggPrule, but It will generate steam agg for sql like "select a,b,c from
foo group by a,b,c".
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687510#comment-17687510
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1103724674
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
> Drill doesn't use streaming aggregate for distinct calls, so only hash agg
should be enough.
I see it checks aggregate.containsDistinctCall() in StreamAggPrule, but It
will generate steam agg for sql like "select a,b,c from foo group by a,b,c".
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687491#comment-17687491
]
ASF GitHub Bot commented on DRILL-4232:
---
vvysotskyi commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1103702656
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
Drill doesn't use streaming aggregate for distinct calls, so only hash agg
should be enough.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8399) MS Access Reader Misinterprets Data Types
[ https://issues.apache.org/jira/browse/DRILL-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687208#comment-17687208 ] ASF GitHub Bot commented on DRILL-8399: --- cgivre opened a new pull request, #2755: URL: https://github.com/apache/drill/pull/2755 # [DRILL-8399](https://issues.apache.org/jira/browse/DRILL-8399): MS Access Reader Misinterprets Data Types ## Description Minor bug fix. Drill was misinterpreting some data types in MS Access files. This fixes that. ## Documentation N/A ## Testing Added unit test. > MS Access Reader Misinterprets Data Types > - > > Key: DRILL-8399 > URL: https://issues.apache.org/jira/browse/DRILL-8399 > Project: Apache Drill > Issue Type: Bug > Components: Format - MS Access >Affects Versions: 1.21.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 1.21.0 > > > The MS Access reader was assigning certain data types incorrectly, resulting > in various errors. This minor PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687062#comment-17687062 ] ASF GitHub Bot commented on DRILL-4232: --- cgivre commented on PR #2599: URL: https://github.com/apache/drill/pull/2599#issuecomment-1425760414 > @Leon-WTF This weekend would be great! Very excited to get this merged. > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686882#comment-17686882
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1102305963
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
One more question about moving converting rule to physical phase, I need to
add physical agg rel node, so needs to add both hash(distribute by all
keys/single key) and stream agg, right?
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+pa
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686876#comment-17686876 ] ASF GitHub Bot commented on DRILL-4232: --- Leon-WTF commented on PR #2599: URL: https://github.com/apache/drill/pull/2599#issuecomment-1425212100 > Hey @Leon-WTF Any chance you could address @vvysotskyi 's comments soon. This is one of the last PRs slated to be merged for the next release. I will do it by this weekend, is that ok? > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8396) Update checkstyle version
[ https://issues.apache.org/jira/browse/DRILL-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686824#comment-17686824 ] ASF GitHub Bot commented on DRILL-8396: --- cgivre merged PR #2753: URL: https://github.com/apache/drill/pull/2753 > Update checkstyle version > - > > Key: DRILL-8396 > URL: https://issues.apache.org/jira/browse/DRILL-8396 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Minor > > Update com.puppycrawl.tools:checkstyle version to the latest one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8398) Fix GitHub Actions to use proper JDK version
[ https://issues.apache.org/jira/browse/DRILL-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686810#comment-17686810 ] ASF GitHub Bot commented on DRILL-8398: --- vvysotskyi opened a new pull request, #2754: URL: https://github.com/apache/drill/pull/2754 # [DRILL-8398](https://issues.apache.org/jira/browse/DRILL-8398): Fix GitHub Actions to use proper JDK version ## Description Added `actions/setup-java` action to use dedicated java version for tests instead of using provided with the instance one. This PR is based on the commit for DRILL-8396 to avoid early failures for JDK 8. Need to fix hive tests that were broken earlier (because they run only with JDK 8). ## Documentation NA ## Testing TBA > Fix GitHub Actions to use proper JDK version > > > Key: DRILL-8398 > URL: https://issues.apache.org/jira/browse/DRILL-8398 > Project: Apache Drill > Issue Type: Bug >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > > In one of the changes GitHub Actions `actions/setup-java@v2` action was > removed, and after that provided java version (11) was used instead of the > specified one in the > matrix.java. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8396) Update checkstyle version
[ https://issues.apache.org/jira/browse/DRILL-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686766#comment-17686766 ] ASF GitHub Bot commented on DRILL-8396: --- vvysotskyi opened a new pull request, #2753: URL: https://github.com/apache/drill/pull/2753 # [DRILL-8396](https://issues.apache.org/jira/browse/DRILL-8396): Checkstyle plugin works only with JDK 11+ ## Description As it turned out, Checkstyle 10.x version is supported only on JDK 11+ (https://checkstyle.org/#JRE_and_JDK), so it causes failures when attempting to build Drill on JDK 8. We didn't notice it because CI doesn't use JDK 8 (though it should). To fix it, I have replaced it with the `checkstyle-backport-jre8` library which works fine on JDK 8 and is referenced in the official docs: https://checkstyle.org/#Backport ## Documentation NA ## Testing Checked manually. > Update checkstyle version > - > > Key: DRILL-8396 > URL: https://issues.apache.org/jira/browse/DRILL-8396 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Minor > > Update com.puppycrawl.tools:checkstyle version to the latest one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8397) Drill prints warnings to console when starting it
[
https://issues.apache.org/jira/browse/DRILL-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686605#comment-17686605
]
ASF GitHub Bot commented on DRILL-8397:
---
jnturton merged PR #2752:
URL: https://github.com/apache/drill/pull/2752
> Drill prints warnings to console when starting it
> -
>
> Key: DRILL-8397
> URL: https://issues.apache.org/jira/browse/DRILL-8397
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> When starting the drill in embedded mode, it prints the following warnings:
> {noformat}
> 11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This
> is logback-classic version 1.4.5
> 11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
> NOT find resource [logback-test.xml]
> 11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
> resource [logback.xml] at
> [file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 73
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 78
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 91
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [STDOUT]
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 11:19:55,658 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [QUERY]
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,697 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No
> compression will be used
> 11:19:55,699 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [FILE]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No
> compression will be used
> 11:19:55,701 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log]
> 11:19:55,702 |-INFO in
> ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting
> additivity of logger [org.apache.drill] to false
> 11:19:55,702 |-INFO in
> ch.qos.logback.classic.model.processor.LevelModelHandler - org.apache.drill
> level set to INFO
> 11:19:55,702 |-INFO in
> ch.qos.logback.core.model.processor.AppenderR
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686441#comment-17686441 ] ASF GitHub Bot commented on DRILL-4232: --- cgivre commented on PR #2599: URL: https://github.com/apache/drill/pull/2599#issuecomment-1424152751 Hey @Leon-WTF Any chance you could address @vvysotskyi 's comments soon. This is one of the last PRs slated to be merged for the next release. > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8397) Drill prints warnings to console when starting it
[
https://issues.apache.org/jira/browse/DRILL-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686438#comment-17686438
]
ASF GitHub Bot commented on DRILL-8397:
---
cgivre commented on PR #2752:
URL: https://github.com/apache/drill/pull/2752#issuecomment-1424151052
>
Given the flakyness of that test, I'm fine with marking that as a "run
manually" test. I think you are correct that the indexing doesn't happen
immediately.
> Drill prints warnings to console when starting it
> -
>
> Key: DRILL-8397
> URL: https://issues.apache.org/jira/browse/DRILL-8397
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> When starting the drill in embedded mode, it prints the following warnings:
> {noformat}
> 11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This
> is logback-classic version 1.4.5
> 11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
> NOT find resource [logback-test.xml]
> 11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
> resource [logback.xml] at
> [file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 73
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 78
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 91
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [STDOUT]
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 11:19:55,658 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [QUERY]
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,697 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No
> compression will be used
> 11:19:55,699 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [FILE]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No
> compression will be used
> 11:19:55,701 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log]
> 11:19:55,702 |-INFO in
> ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting
> additivity of logger [org.apache.drill] to f
[jira] [Commented] (DRILL-8397) Drill prints warnings to console when starting it
[
https://issues.apache.org/jira/browse/DRILL-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686399#comment-17686399
]
ASF GitHub Bot commented on DRILL-8397:
---
jnturton commented on PR #2752:
URL: https://github.com/apache/drill/pull/2752#issuecomment-1424101087
In any case, that failure is certainly unrelated to this PR which in my
opinion may be merged.
> Drill prints warnings to console when starting it
> -
>
> Key: DRILL-8397
> URL: https://issues.apache.org/jira/browse/DRILL-8397
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> When starting the drill in embedded mode, it prints the following warnings:
> {noformat}
> 11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This
> is logback-classic version 1.4.5
> 11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
> NOT find resource [logback-test.xml]
> 11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
> resource [logback.xml] at
> [file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 73
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 78
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 91
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [STDOUT]
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 11:19:55,658 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [QUERY]
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,697 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No
> compression will be used
> 11:19:55,699 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [FILE]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No
> compression will be used
> 11:19:55,701 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log]
> 11:19:55,702 |-INFO in
> ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting
> additivity of logger [org.apache.drill] to false
> 11:19:55,702 |-INFO in
> ch.qos.logback.classic.model.processor.L
[jira] [Commented] (DRILL-8397) Drill prints warnings to console when starting it
[
https://issues.apache.org/jira/browse/DRILL-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686398#comment-17686398
]
ASF GitHub Bot commented on DRILL-8397:
---
jnturton commented on PR #2752:
URL: https://github.com/apache/drill/pull/2752#issuecomment-1424093562
Regarding the sporadic failures of the unit test
testBasicCTASWithScalarDataTypes that have been showing in recent CI runs, and
now also here, it looks to me like [the Thread.sleep(15_000) that it
uses](https://github.com/jnturton/drill/blob/83bf04cd815e6c6d22d9ef04550cd8113bd9f905/contrib/storage-splunk/src/test/java/org/apache/drill/exec/store/splunk/SplunkWriterTest.java#L113)
to wait for an INSERT to go through is not always enough to ensure that Splunk
will return the record needed for the subsequent schema validation to pass.
Locally I've been able to reproduce both successes and failures of this test
with a range of sleep durations from 6_000 to 60_000 so I'm not sure if there
is a magic number here, or what it might be. Stepping back further, I don't
know if Splunk ever takes a transactional approach to loading data - maybe it
makes no guarantees beyond "Thanks, I'll index this later"?
CC @cgivre.
> Drill prints warnings to console when starting it
> -
>
> Key: DRILL-8397
> URL: https://issues.apache.org/jira/browse/DRILL-8397
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> When starting the drill in embedded mode, it prints the following warnings:
> {noformat}
> 11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This
> is logback-classic version 1.4.5
> 11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
> NOT find resource [logback-test.xml]
> 11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
> resource [logback.xml] at
> [file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 73
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 78
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 91
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [STDOUT]
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 11:19:55,658 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [QUERY]
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,697 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No
> compression will be used
> 11:19:55,699 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [FILE]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No
> compression wil
[jira] [Commented] (DRILL-8397) Drill prints warnings to console when starting it
[
https://issues.apache.org/jira/browse/DRILL-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686311#comment-17686311
]
ASF GitHub Bot commented on DRILL-8397:
---
vvysotskyi opened a new pull request, #2752:
URL: https://github.com/apache/drill/pull/2752
# [DRILL-8397](https://issues.apache.org/jira/browse/DRILL-8397): Drill
prints warnings to console when starting it
## Description
Replaced the `level` tag with the attribute, so no extra output is printed
to the console.
## Documentation
NA
## Testing
Checked manually
> Drill prints warnings to console when starting it
> -
>
> Key: DRILL-8397
> URL: https://issues.apache.org/jira/browse/DRILL-8397
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> When starting the drill in embedded mode, it prints the following warnings:
> {noformat}
> 11:19:55,482 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This
> is logback-classic version 1.4.5
> 11:19:55,499 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
> NOT find resource [logback-test.xml]
> 11:19:55,503 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
> resource [logback.xml] at
> [file:/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/conf/logback.xml]
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 73
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 78
> 11:19:55,607 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> element is deprecated. Near [level] on line 91
> 11:19:55,608 |-WARN in ch.qos.logback.classic.joran.action.LevelAction -
> Please use "level" attribute within or elements instead.
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [STDOUT]
> 11:19:55,652 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
> 11:19:55,658 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [QUERY]
> 11:19:55,689 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,697 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@66ac5762 - No
> compression will be used
> 11:19:55,699 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json
> 11:19:55,699 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[QUERY]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline_queries.json]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - Processing
> appender named [FILE]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.model.processor.AppenderModelHandler - About to
> instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
> 11:19:55,700 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@797cf65c - No
> compression will be used
> 11:19:55,701 |-INFO in
> ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default
> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder]
> property
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - Active log file name:
> /tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log
> 11:19:55,701 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE]
> - File property is set to
> [/tmp/drill/distribution/target/apache-drill-1.21.0-SNAPSHOT/apache-drill-1.21.0-SNAPSHOT/log/sqlline.log]
> 11:19:55,702 |-
[jira] [Commented] (DRILL-8309) Upgrade slf4j to 2.0.x
[ https://issues.apache.org/jira/browse/DRILL-8309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686165#comment-17686165 ] ASF GitHub Bot commented on DRILL-8309: --- jnturton commented on PR #2750: URL: https://github.com/apache/drill/pull/2750#issuecomment-1423573873 @pjfanning the upgrade wasn't as smooth as it looks, I fought with it for hours. I should have left this PR in draft before retiring yesterday because I did want to come back and expand my commentary. The seemingly unrelated upgrades and removals came from my chasing after fat jars that bundle slf4j < 2.0 (and were breaking this upgrade). In the end it was the refactoring of the Avatica dependency that really counted. > Upgrade slf4j to 2.0.x > -- > > Key: DRILL-8309 > URL: https://issues.apache.org/jira/browse/DRILL-8309 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.2 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > * logback 1.2.11 → 1.4.5. > * log4j 2.18.0 → 2.19.0. > * slf4j 1.7.26 → 2.0.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8309) Upgrade slf4j to 2.0.x
[ https://issues.apache.org/jira/browse/DRILL-8309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686157#comment-17686157 ] ASF GitHub Bot commented on DRILL-8309: --- cgivre merged PR #2750: URL: https://github.com/apache/drill/pull/2750 > Upgrade slf4j to 2.0.x > -- > > Key: DRILL-8309 > URL: https://issues.apache.org/jira/browse/DRILL-8309 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.2 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > * logback 1.2.11 → 1.4.5. > * log4j 2.18.0 → 2.19.0. > * slf4j 1.7.26 → 2.0.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8396) Update checkstyle version
[ https://issues.apache.org/jira/browse/DRILL-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686154#comment-17686154 ] ASF GitHub Bot commented on DRILL-8396: --- cgivre merged PR #2751: URL: https://github.com/apache/drill/pull/2751 > Update checkstyle version > - > > Key: DRILL-8396 > URL: https://issues.apache.org/jira/browse/DRILL-8396 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Minor > > Update com.puppycrawl.tools:checkstyle version to the latest one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-7722) CREATE VIEW with LATERAL UNNEST creates an invalid view
[
https://issues.apache.org/jira/browse/DRILL-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686139#comment-17686139
]
ASF GitHub Bot commented on DRILL-7722:
---
cgivre commented on PR #2083:
URL: https://github.com/apache/drill/pull/2083#issuecomment-1423478202
The underlying issue was fixed in https://github.com/apache/drill/pull/2602.
> CREATE VIEW with LATERAL UNNEST creates an invalid view
> ---
>
> Key: DRILL-7722
> URL: https://issues.apache.org/jira/browse/DRILL-7722
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.17.0
>Reporter: Matevž Bradač
>Assignee: Vova Vysotskyi
>Priority: Blocker
>
> Creating a view from a query containing LATERAL UNNEST results in a view that
> cannot be parsed by the engine. The generated view contains superfluous
> parentheses, thus the failed parsing.
> {code:bash|title=a simple JSON database}
> $ cat /tmp/t.json
> [{"name": "item_1", "related": ["id1"]}, {"name": "item_2", "related":
> ["id1", "id2"]}, {"name": "item_3", "related": ["id2"]}]
> {code}
> {code:SQL|title=drill query, working}
> SELECT
> item.name,
> relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
> SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
> name rels
> 0 item_1 id1
> 1 item_2 id1
> 2 item_2 id2
> 3 item_3 id2
> {code}
> {code:SQL|title=create a drill view from the above query}
> CREATE VIEW dfs.tmp.unnested_view AS
> SELECT
> item.name,
> relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
> SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
> {code}
> {code:bash|title=contents of view file}
> # note the extra parentheses near LATERAL and FROM
> $ cat /tmp/unnested_view.view.drill
> {
> "name" : "unnested_view",
> "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS
> `item`\nINNER JOIN LATERAL((SELECT *\nFROM (UNNEST(`item`.`related`)) AS `i`
> (`rels`))) AS `relations` ON TRUE",
> "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
> }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ ]
> }
> {code}
> {code:SQL|title=query the view}
> SELECT * FROM dfs.tmp.unnested_view
> PARSE ERROR: Failure parsing a view your query is dependent upon.
> SQL Query: SELECT `item`.`name`, `relations`.*
> FROM `dfs`.`tmp`.`t.json` AS `item`
> INNER JOIN LATERAL((SELECT *
> FROM (UNNEST(`item`.`related`)) AS `i` (`rels`))) AS `relations` ON TRUE
> ^
> [Error Id: fd816a27-c2c5-4c2a-b6bf-173ab37eb693 ]
> {code}
> If the view is "fixed" by editing the generated JSON and removing the extra
> parentheses, e.g.
> {code:bash|title=fixed view}
> $ cat /tmp/fixed_unnested_view.view.drill
> {
> "name" : "fixed_unnested_view",
> "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS
> `item`\nINNER JOIN LATERAL(SELECT *\nFROM UNNEST(`item`.`related`) AS `i`
> (`rels`)) AS `relations` ON TRUE",
> "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
> }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ ]
> }
> {code}
> then querying works as expected:
> {code:sql|title=fixed view query}
> SELECT * FROM dfs.tmp.fixed_unnested_view
> name rels
> 0 item_1 id1
> 1 item_2 id1
> 2 item_2 id2
> 3 item_3 id2
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-7722) CREATE VIEW with LATERAL UNNEST creates an invalid view
[
https://issues.apache.org/jira/browse/DRILL-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686140#comment-17686140
]
ASF GitHub Bot commented on DRILL-7722:
---
cgivre closed pull request #2083: DRILL-7722: Add reproducer unit test
URL: https://github.com/apache/drill/pull/2083
> CREATE VIEW with LATERAL UNNEST creates an invalid view
> ---
>
> Key: DRILL-7722
> URL: https://issues.apache.org/jira/browse/DRILL-7722
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.17.0
>Reporter: Matevž Bradač
>Assignee: Vova Vysotskyi
>Priority: Blocker
>
> Creating a view from a query containing LATERAL UNNEST results in a view that
> cannot be parsed by the engine. The generated view contains superfluous
> parentheses, thus the failed parsing.
> {code:bash|title=a simple JSON database}
> $ cat /tmp/t.json
> [{"name": "item_1", "related": ["id1"]}, {"name": "item_2", "related":
> ["id1", "id2"]}, {"name": "item_3", "related": ["id2"]}]
> {code}
> {code:SQL|title=drill query, working}
> SELECT
> item.name,
> relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
> SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
> name rels
> 0 item_1 id1
> 1 item_2 id1
> 2 item_2 id2
> 3 item_3 id2
> {code}
> {code:SQL|title=create a drill view from the above query}
> CREATE VIEW dfs.tmp.unnested_view AS
> SELECT
> item.name,
> relations.*
> FROM dfs.tmp.`t.json` item
> JOIN LATERAL(
> SELECT * FROM UNNEST(item.related) i(rels)
> ) relations
> ON TRUE
> {code}
> {code:bash|title=contents of view file}
> # note the extra parentheses near LATERAL and FROM
> $ cat /tmp/unnested_view.view.drill
> {
> "name" : "unnested_view",
> "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS
> `item`\nINNER JOIN LATERAL((SELECT *\nFROM (UNNEST(`item`.`related`)) AS `i`
> (`rels`))) AS `relations` ON TRUE",
> "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
> }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ ]
> }
> {code}
> {code:SQL|title=query the view}
> SELECT * FROM dfs.tmp.unnested_view
> PARSE ERROR: Failure parsing a view your query is dependent upon.
> SQL Query: SELECT `item`.`name`, `relations`.*
> FROM `dfs`.`tmp`.`t.json` AS `item`
> INNER JOIN LATERAL((SELECT *
> FROM (UNNEST(`item`.`related`)) AS `i` (`rels`))) AS `relations` ON TRUE
> ^
> [Error Id: fd816a27-c2c5-4c2a-b6bf-173ab37eb693 ]
> {code}
> If the view is "fixed" by editing the generated JSON and removing the extra
> parentheses, e.g.
> {code:bash|title=fixed view}
> $ cat /tmp/fixed_unnested_view.view.drill
> {
> "name" : "fixed_unnested_view",
> "sql" : "SELECT `item`.`name`, `relations`.*\nFROM `dfs`.`tmp`.`t.json` AS
> `item`\nINNER JOIN LATERAL(SELECT *\nFROM UNNEST(`item`.`related`) AS `i`
> (`rels`)) AS `relations` ON TRUE",
> "fields" : [ {
> "name" : "name",
> "type" : "ANY",
> "isNullable" : true
> }, {
> "name" : "rels",
> "type" : "ANY",
> "isNullable" : true
> } ],
> "workspaceSchemaPath" : [ ]
> }
> {code}
> then querying works as expected:
> {code:sql|title=fixed view query}
> SELECT * FROM dfs.tmp.fixed_unnested_view
> name rels
> 0 item_1 id1
> 1 item_2 id1
> 2 item_2 id2
> 3 item_3 id2
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8396) Update checkstyle version
[ https://issues.apache.org/jira/browse/DRILL-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686115#comment-17686115 ] ASF GitHub Bot commented on DRILL-8396: --- vvysotskyi opened a new pull request, #2751: URL: https://github.com/apache/drill/pull/2751 # [DRILL-8396](https://issues.apache.org/jira/browse/DRILL-8396): Update checkstyle version ## Description Update checkstyle version to the latest one ## Documentation NA ## Testing Unit tests pass > Update checkstyle version > - > > Key: DRILL-8396 > URL: https://issues.apache.org/jira/browse/DRILL-8396 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Minor > > Update com.puppycrawl.tools:checkstyle version to the latest one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8395) Add Support for INSERT and Drop Table to GoogleSheets Plugin
[ https://issues.apache.org/jira/browse/DRILL-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686099#comment-17686099 ] ASF GitHub Bot commented on DRILL-8395: --- cgivre merged PR #2748: URL: https://github.com/apache/drill/pull/2748 > Add Support for INSERT and Drop Table to GoogleSheets Plugin > > > Key: DRILL-8395 > URL: https://issues.apache.org/jira/browse/DRILL-8395 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - GoogleSheets >Affects Versions: 1.20.3 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.0 > > > This PR adds support for INSERT queries which allow a user to append data to > an existing GoogleSheets tab. It also: > * Adds support for DROP TABLE queries which were not implemented > * Modifies CTAS queries so that if a user executes a CTAS query with a file > token, Drill will add a new tab to an existing document, but if the user > executes a CTAS with a file name, it will create an entirely new document. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685882#comment-17685882 ] ASF GitHub Bot commented on DRILL-8379: --- cgivre merged PR #2749: URL: https://github.com/apache/drill/pull/2749 > Update Calcite to 1.33.0 > > > Key: DRILL-8379 > URL: https://issues.apache.org/jira/browse/DRILL-8379 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[
https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685824#comment-17685824
]
ASF GitHub Bot commented on DRILL-8379:
---
vvysotskyi commented on code in PR #2749:
URL: https://github.com/apache/drill/pull/2749#discussion_r1099920934
##
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestAnalyze.java:
##
@@ -332,7 +332,7 @@ public void testUseStatistics() throws Exception {
query = " select emp.employee_id from dfs.tmp.employeeUseStat emp join
dfs.tmp.departmentUseStat dept"
+ " on emp.department_id = dept.department_id "
+ " group by emp.employee_id";
- String[] expectedPlan8 = {"HashAgg\\(group=\\[\\{0\\}\\]\\).*rowcount =
730.2832515526484,.*",
+ String[] expectedPlan8 = {"HashAgg\\(group=\\[\\{0\\}\\]\\).*rowcount =
730.2832515526.*",
Review Comment:
Calcite has slightly updated the formula to calculate it. I have removed
these digits to avoid failures for further potential updates.
> Update Calcite to 1.33.0
>
>
> Key: DRILL-8379
> URL: https://issues.apache.org/jira/browse/DRILL-8379
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[
https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685820#comment-17685820
]
ASF GitHub Bot commented on DRILL-8379:
---
jnturton commented on code in PR #2749:
URL: https://github.com/apache/drill/pull/2749#discussion_r1099912943
##
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestAnalyze.java:
##
@@ -332,7 +332,7 @@ public void testUseStatistics() throws Exception {
query = " select emp.employee_id from dfs.tmp.employeeUseStat emp join
dfs.tmp.departmentUseStat dept"
+ " on emp.department_id = dept.department_id "
+ " group by emp.employee_id";
- String[] expectedPlan8 = {"HashAgg\\(group=\\[\\{0\\}\\]\\).*rowcount =
730.2832515526484,.*",
+ String[] expectedPlan8 = {"HashAgg\\(group=\\[\\{0\\}\\]\\).*rowcount =
730.2832515526.*",
Review Comment:
Did we drop the last three digits here to prevent spurious test failures
resulting from rounding errors?
> Update Calcite to 1.33.0
>
>
> Key: DRILL-8379
> URL: https://issues.apache.org/jira/browse/DRILL-8379
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vova Vysotskyi
>Assignee: Vova Vysotskyi
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8309) Upgrade slf4j to 2.0.x
[ https://issues.apache.org/jira/browse/DRILL-8309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685769#comment-17685769 ] ASF GitHub Bot commented on DRILL-8309: --- jnturton opened a new pull request, #2750: URL: https://github.com/apache/drill/pull/2750 # [DRILL-8309](https://issues.apache.org/jira/browse/DRILL-8309): Upgrade slf4j to 2.0.x ## Description - logback 1.2.11 -> 1.4.5. - log4j 2.18.0 -> 2.19.0. - slf4j 1.7.26 -> 2.0.6. ## Documentation N/A ## Testing Run embedded Drill and check its log output. > Upgrade slf4j to 2.0.x > -- > > Key: DRILL-8309 > URL: https://issues.apache.org/jira/browse/DRILL-8309 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.20.2 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.0 > > > * logback 1.2.11 -> 1.4.5. > * log4j 2.18.0 -> 2.19.0. > * slf4j 1.7.26 -> 2.0.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8372) Unfreed buffers when running a LIMIT 0 query over delimited text
[
https://issues.apache.org/jira/browse/DRILL-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685735#comment-17685735
]
ASF GitHub Bot commented on DRILL-8372:
---
jnturton merged PR #2728:
URL: https://github.com/apache/drill/pull/2728
> Unfreed buffers when running a LIMIT 0 query over delimited text
>
>
> Key: DRILL-8372
> URL: https://issues.apache.org/jira/browse/DRILL-8372
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.21.0
>Reporter: James Turton
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> With the following data layout
> {code:java}
> /tmp/foo
> ├──a
> │ └──test_data.csvh
> └──b
> └──test_data.csvh {code}
> a LIMIT 0 query over it results in unfreed buffer errors.
> {code:java}
> apache drill (dfs.tmp)> select * from `foo` limit 0;
> Error: SYSTEM ERROR: IllegalStateException: Allocator[op:0:0:4:EasySubScan]
> closed with outstanding buffers allocated (3).
> Allocator(op:0:0:4:EasySubScan) 100/299008/3182592/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 3
> ledger[113] allocator: op:0:0:4:EasySubScan), isOwning: true, size:
> 262144, references: 1, life: 277785186322881..0, allocatorManager: [109,
> life: 277785186258906..0] holds 1 buffers.
> DrillBuf[142], udle: [110 0..262144]
> ledger[114] allocator: op:0:0:4:EasySubScan), isOwning: true, size:
> 32768, references: 1, life: 277785186463824..0, allocatorManager: [110, life:
> 277785186414654..0] holds 1 buffers.
> DrillBuf[143], udle: [111 0..32768]
> ledger[112] allocator: op:0:0:4:EasySubScan), isOwning: true, size: 4096,
> references: 1, life: 277785186046095..0, allocatorManager: [108, life:
> 277785185921147..0] holds 1 buffers.
> DrillBuf[141], udle: [109 0..4096]
> reservations: 0 {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8290) Short cut recursive file listings for LIMIT 0 queries
[ https://issues.apache.org/jira/browse/DRILL-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685734#comment-17685734 ] ASF GitHub Bot commented on DRILL-8290: --- jnturton merged PR #2636: URL: https://github.com/apache/drill/pull/2636 > Short cut recursive file listings for LIMIT 0 queries > - > > Key: DRILL-8290 > URL: https://issues.apache.org/jira/browse/DRILL-8290 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning & Optimization >Affects Versions: 1.20.2 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.0 > > > The existing LIMIT 0 query optimisations do not prevent a query run against > the top of a deep DFS directory tree from recursively listing FileStatuses > for everything within it using a pool of worker threads. This Issue proposes > a new optimisation whereby such queries will recurse into the directory tree > on a single thread that returns as soon as any single FileStatus has been > obtained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8290) Short cut recursive file listings for LIMIT 0 queries
[
https://issues.apache.org/jira/browse/DRILL-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685716#comment-17685716
]
ASF GitHub Bot commented on DRILL-8290:
---
jnturton commented on code in PR #2636:
URL: https://github.com/apache/drill/pull/2636#discussion_r1099737668
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java:
##
@@ -645,10 +646,18 @@ public Void visitOp(PhysicalOperator op,
Collection collection
}
return null;
}
-
}
protected Pair validateNode(SqlNode sqlNode) throws
ValidationException, RelConversionException, ForemanSetupException {
+if (context.getOptions().getOption(ExecConstants.FILE_LISTING_LIMIT0_OPT))
{
+ // Check for a LIMIT 0 in the root portion of the query before validation
+ // because validation of the query's FROM clauses will already trigger
+ // the recursive listing files to which FILE_LISTING_LIMIT0_OPT is meant
+ // to apply.
+ boolean rootSelectLimit0 = FindLimit0SqlVisitor.containsLimit0(sqlNode);
+ context.getPlannerSettings().setRootSelectLimit0(rootSelectLimit0);
Review Comment:
@vvysotskyi I've made use of a query option, how do things look now?
> Short cut recursive file listings for LIMIT 0 queries
> -
>
> Key: DRILL-8290
> URL: https://issues.apache.org/jira/browse/DRILL-8290
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 1.21.0
>
>
> The existing LIMIT 0 query optimisations do not prevent a query run against
> the top of a deep DFS directory tree from recursively listing FileStatuses
> for everything within it using a pool of worker threads. This Issue proposes
> a new optimisation whereby such queries will recurse into the directory tree
> on a single thread that returns as soon as any single FileStatus has been
> obtained.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685601#comment-17685601 ] ASF GitHub Bot commented on DRILL-8379: --- vvysotskyi commented on PR #2749: URL: https://github.com/apache/drill/pull/2749#issuecomment-1421566035 > Thanks @vvysotskyi This looked like it wasn't quite as painless as going from 1.31 to 1.32, but didn't look like it required any major surgery. Agree, this update wasn't trivial and forced us to sacrifice some temporary table functionality in https://github.com/apache/drill/pull/2733, but still, it was much simpler than jumping from 1.21.0 to 1.31.0. > Update Calcite to 1.33.0 > > > Key: DRILL-8379 > URL: https://issues.apache.org/jira/browse/DRILL-8379 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685477#comment-17685477 ] ASF GitHub Bot commented on DRILL-8379: --- cgivre commented on PR #2749: URL: https://github.com/apache/drill/pull/2749#issuecomment-1421470624 Thanks @vvysotskyi This looked like it wasn't quite as painless as going from 1.31 to 1.32, but didn't look like it required any major surgery. > Update Calcite to 1.33.0 > > > Key: DRILL-8379 > URL: https://issues.apache.org/jira/browse/DRILL-8379 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685411#comment-17685411 ] ASF GitHub Bot commented on DRILL-8379: --- vvysotskyi commented on code in PR #2749: URL: https://github.com/apache/drill/pull/2749#discussion_r1099017177 ## exec/java-exec/src/main/codegen/data/Parser.tdd: ## @@ -118,780 +118,824 @@ # List of keywords from "keywords" section that are not reserved. # Copied from calcite-core config.fmpp # For details please see comment under CALCITE-2405. - nonReservedKeywords: [ -"A" -"ABSENT" -"ABSOLUTE" -"ACTION" -"ADA" -"ADD" -"ADMIN" -"AFTER" -"ALIAS" -"ALIASES" -"ALWAYS" -"APPLY" -"ASC" -"ASSERTION" -"ASSIGNMENT" -"ATTRIBUTE" -"ATTRIBUTES" -"BEFORE" -"BERNOULLI" -"BREADTH" -"C" -"CASCADE" -"CATALOG" -"CATALOG_NAME" -"CENTURY" -"CHAIN" -"CHARACTER_SET_CATALOG" -"CHARACTER_SET_NAME" -"CHARACTER_SET_SCHEMA" -"CHARACTERISTICS" -"CHARACTERS" -"CLASS_ORIGIN" -"COBOL" -"COLLATION" -"COLLATION_CATALOG" -"COLLATION_NAME" -"COLLATION_SCHEMA" -"COLUMN_NAME" -"COMMAND_FUNCTION" -"COMMAND_FUNCTION_CODE" -"COMMITTED" -"CONDITION_NUMBER" -"CONNECTION" -"CONNECTION_NAME" -"CONSTRAINT_CATALOG" -"CONSTRAINT_NAME" -"CONSTRAINT_SCHEMA" -"CONSTRAINTS" -"CONSTRUCTOR" -"CONTINUE" -"CURSOR_NAME" -"DATA" -"DATABASE" -"DATETIME_INTERVAL_CODE" -"DATETIME_INTERVAL_PRECISION" -"DECADE" -"DEFAULTS" -"DEFERRABLE" -"DEFERRED" -"DEFINED" -"DEFINER" -"DEGREE" -"DEPTH" -"DERIVED" -"DESC" -"DESCRIPTION" -"DESCRIPTOR" -"DIAGNOSTICS" -"DISPATCH" -"DOMAIN" -"DOW" -"DOY" -"DYNAMIC_FUNCTION" -"DYNAMIC_FUNCTION_CODE" -"ENCODING" -"EPOCH" -"ERROR" -"EXCEPTION" -"EXCLUDE" -"EXCLUDING" -"FINAL" -"FIRST" -"FOLLOWING" -"FORMAT" -"FORTRAN" -"FOUND" -"FRAC_SECOND" -"G" -"GENERAL" -"GENERATED" -"GEOMETRY" -"GO" -"GOTO" -"GRANTED" -"HIERARCHY" -"IMMEDIATE" -"IMMEDIATELY" -"IMPLEMENTATION" -"INCLUDING" -"INCREMENT" -"INITIALLY" -"INPUT" -"INSTANCE" -"INSTANTIABLE" -"INVOKER" -"ISODOW" -"ISOYEAR" -"ISOLATION" -"JAVA" -"JSON" -"K" -"KEY" -"KEY_MEMBER" -"KEY_TYPE" -"LABEL" -"LAST" -"LENGTH" -"LEVEL" -"LIBRARY" -"LOCATOR" -"M" -"MAP" -"MATCHED" -"MAXVALUE" -"MICROSECOND" -"MESSAGE_LENGTH" -"MESSAGE_OCTET_LENGTH" -"MESSAGE_TEXT" -"MILLISECOND" -"MILLENNIUM" -"MINVALUE" -"MONTHS" -"MORE_" -"MUMPS" -"NAME" -"NAMES" -"NANOSECOND" -"NESTING" -"NORMALIZED" -"NULLABLE" -"NULLS" -"NUMBER" -"OBJECT" -"OCTETS" -"OPTION" -"OPTIONS" -"ORDERING" -"ORDINALITY" -"OTHERS" -"OUTPUT" -"OVERRIDING" -"PAD" -"PARAMETER_MODE" -"PARAMETER_NAME" -"PARAMETER_ORDINAL_POSITION" -"PARAMETER_SPECIFIC_CATALOG" -"PARAMETER_SPECIFIC_NAME" -"PARAMETER_SPECIFIC_SCHEMA" -"PARTIAL" -"PASCAL" -"PASSING" -"PASSTHROUGH" -"PAST" -"PATH" -"PLACING" -"PLAN" -"PLI" -"PRECEDING" -"PRESERVE" -"PRIOR" -"PRIVILEGES" -"PUBLIC" -"QUARTER" -"READ" -"RELATIVE" -"REPEATABLE" -"REPLACE" -"RESTART" -"RESTRICT" -"RETURNED_CARDINALITY" -"RETURNED_LENGTH" -"RETURNED_OCTET_LENGTH" -"RETURNED_SQLSTATE" -"ROLE" -"ROUTINE" -"ROUTINE_CATALOG" -"ROUTINE_NAME" -"ROUTINE_SCHEMA" -"ROW_COUNT" -"SCALE" -"SCHEMA" -"SCHEMA_NAME" -"SCOPE_CATALOGS" -"SCOPE_NAME" -"SCOPE_SCHEMA" -"SECTION" -"SECURITY" -"SELF" -"SEQUENCE" -"SERIALIZABLE" -"SERVER" -"SERVER_NAME" -"SESSION" -"SETS" -"SIMPLE" -"SIZE" -"SOURCE" -"SPACE" -
[jira] [Commented] (DRILL-8387) Add Support for User Translation to ElasticSearch Plugin
[ https://issues.apache.org/jira/browse/DRILL-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685270#comment-17685270 ] ASF GitHub Bot commented on DRILL-8387: --- cgivre merged PR #2739: URL: https://github.com/apache/drill/pull/2739 > Add Support for User Translation to ElasticSearch Plugin > > > Key: DRILL-8387 > URL: https://issues.apache.org/jira/browse/DRILL-8387 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - ElasticSearch >Affects Versions: 1.20.3 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.0 > > > Add support for user translation to ElasticSearch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685159#comment-17685159 ] ASF GitHub Bot commented on DRILL-8379: --- jnturton commented on code in PR #2749: URL: https://github.com/apache/drill/pull/2749#discussion_r1098367575 ## exec/java-exec/src/main/codegen/data/Parser.tdd: ## @@ -118,780 +118,824 @@ # List of keywords from "keywords" section that are not reserved. # Copied from calcite-core config.fmpp # For details please see comment under CALCITE-2405. - nonReservedKeywords: [ -"A" -"ABSENT" -"ABSOLUTE" -"ACTION" -"ADA" -"ADD" -"ADMIN" -"AFTER" -"ALIAS" -"ALIASES" -"ALWAYS" -"APPLY" -"ASC" -"ASSERTION" -"ASSIGNMENT" -"ATTRIBUTE" -"ATTRIBUTES" -"BEFORE" -"BERNOULLI" -"BREADTH" -"C" -"CASCADE" -"CATALOG" -"CATALOG_NAME" -"CENTURY" -"CHAIN" -"CHARACTER_SET_CATALOG" -"CHARACTER_SET_NAME" -"CHARACTER_SET_SCHEMA" -"CHARACTERISTICS" -"CHARACTERS" -"CLASS_ORIGIN" -"COBOL" -"COLLATION" -"COLLATION_CATALOG" -"COLLATION_NAME" -"COLLATION_SCHEMA" -"COLUMN_NAME" -"COMMAND_FUNCTION" -"COMMAND_FUNCTION_CODE" -"COMMITTED" -"CONDITION_NUMBER" -"CONNECTION" -"CONNECTION_NAME" -"CONSTRAINT_CATALOG" -"CONSTRAINT_NAME" -"CONSTRAINT_SCHEMA" -"CONSTRAINTS" -"CONSTRUCTOR" -"CONTINUE" -"CURSOR_NAME" -"DATA" -"DATABASE" -"DATETIME_INTERVAL_CODE" -"DATETIME_INTERVAL_PRECISION" -"DECADE" -"DEFAULTS" -"DEFERRABLE" -"DEFERRED" -"DEFINED" -"DEFINER" -"DEGREE" -"DEPTH" -"DERIVED" -"DESC" -"DESCRIPTION" -"DESCRIPTOR" -"DIAGNOSTICS" -"DISPATCH" -"DOMAIN" -"DOW" -"DOY" -"DYNAMIC_FUNCTION" -"DYNAMIC_FUNCTION_CODE" -"ENCODING" -"EPOCH" -"ERROR" -"EXCEPTION" -"EXCLUDE" -"EXCLUDING" -"FINAL" -"FIRST" -"FOLLOWING" -"FORMAT" -"FORTRAN" -"FOUND" -"FRAC_SECOND" -"G" -"GENERAL" -"GENERATED" -"GEOMETRY" -"GO" -"GOTO" -"GRANTED" -"HIERARCHY" -"IMMEDIATE" -"IMMEDIATELY" -"IMPLEMENTATION" -"INCLUDING" -"INCREMENT" -"INITIALLY" -"INPUT" -"INSTANCE" -"INSTANTIABLE" -"INVOKER" -"ISODOW" -"ISOYEAR" -"ISOLATION" -"JAVA" -"JSON" -"K" -"KEY" -"KEY_MEMBER" -"KEY_TYPE" -"LABEL" -"LAST" -"LENGTH" -"LEVEL" -"LIBRARY" -"LOCATOR" -"M" -"MAP" -"MATCHED" -"MAXVALUE" -"MICROSECOND" -"MESSAGE_LENGTH" -"MESSAGE_OCTET_LENGTH" -"MESSAGE_TEXT" -"MILLISECOND" -"MILLENNIUM" -"MINVALUE" -"MONTHS" -"MORE_" -"MUMPS" -"NAME" -"NAMES" -"NANOSECOND" -"NESTING" -"NORMALIZED" -"NULLABLE" -"NULLS" -"NUMBER" -"OBJECT" -"OCTETS" -"OPTION" -"OPTIONS" -"ORDERING" -"ORDINALITY" -"OTHERS" -"OUTPUT" -"OVERRIDING" -"PAD" -"PARAMETER_MODE" -"PARAMETER_NAME" -"PARAMETER_ORDINAL_POSITION" -"PARAMETER_SPECIFIC_CATALOG" -"PARAMETER_SPECIFIC_NAME" -"PARAMETER_SPECIFIC_SCHEMA" -"PARTIAL" -"PASCAL" -"PASSING" -"PASSTHROUGH" -"PAST" -"PATH" -"PLACING" -"PLAN" -"PLI" -"PRECEDING" -"PRESERVE" -"PRIOR" -"PRIVILEGES" -"PUBLIC" -"QUARTER" -"READ" -"RELATIVE" -"REPEATABLE" -"REPLACE" -"RESTART" -"RESTRICT" -"RETURNED_CARDINALITY" -"RETURNED_LENGTH" -"RETURNED_OCTET_LENGTH" -"RETURNED_SQLSTATE" -"ROLE" -"ROUTINE" -"ROUTINE_CATALOG" -"ROUTINE_NAME" -"ROUTINE_SCHEMA" -"ROW_COUNT" -"SCALE" -"SCHEMA" -"SCHEMA_NAME" -"SCOPE_CATALOGS" -"SCOPE_NAME" -"SCOPE_SCHEMA" -"SECTION" -"SECURITY" -"SELF" -"SEQUENCE" -"SERIALIZABLE" -"SERVER" -"SERVER_NAME" -"SESSION" -"SETS" -"SIMPLE" -"SIZE" -"SOURCE" -"SPACE" -
[jira] [Commented] (DRILL-8379) Update Calcite to 1.33.0
[ https://issues.apache.org/jira/browse/DRILL-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685080#comment-17685080 ] ASF GitHub Bot commented on DRILL-8379: --- vvysotskyi opened a new pull request, #2749: URL: https://github.com/apache/drill/pull/2749 # [DRILL-8379](https://issues.apache.org/jira/browse/DRILL-8379): Update Calcite to 1.33.0 ## Description Updated Caclite and Avatica versions. Had to update Jackson to avoid errors with ES Calcite connector. ## Documentation NA ## Testing All unit tests pass. > Update Calcite to 1.33.0 > > > Key: DRILL-8379 > URL: https://issues.apache.org/jira/browse/DRILL-8379 > Project: Apache Drill > Issue Type: Task >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685050#comment-17685050
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1097151371
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, I pushed a new commit which refactor updateClient from ClusterTest
to ClientFixture.
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684549#comment-17684549
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1097151371
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, I pushed a new commit which refactor updateClient from ClusterTest
to ClientFixture, and I can pass the client into updateClient so that we don't
need to client.close for each updateClient call, however method signature needs
add another variable specifically for close client.
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684548#comment-17684548
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1097151371
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, I pushed a new commit which refactor updateClient from ClusterTest
to ClientFixture, and I can pass the client into updateClient so that we don't
need to client.close for each updateClient, how method signature needs add
another variable specifically for close client.
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684546#comment-17684546
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1097148776
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
oh, yes, I will refactor that.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684500#comment-17684500
]
ASF GitHub Bot commented on DRILL-4232:
---
vvysotskyi commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1097057876
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
Oh, ok, if it is specific to our implementation of except operator,
aggregation added here possibly could be removed by other Calcite rules which
assume that results would be already distinct.
I think it would be better to add an aggregation when converting it to
physical rel nodes.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684454#comment-17684454
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1097007805
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
It's not distinct for left table after except operator. I choosed to reuse
an aggregate operator to do the distinct.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684451#comment-17684451
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1097007805
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
It's not distinct for left table after except operator.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684447#comment-17684447
]
ASF GitHub Bot commented on DRILL-4232:
---
vvysotskyi commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1097000816
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
But output values should be already distinct after the execution of except
operator, so the aggregate will do nothing.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684433#comment-17684433
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1096965727
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
@vvysotskyi It's for performance, if the data cardinality is high, aggregate
before except may not reduce many data, if the data after except left are few,
aggregate after except will only handle few data which is faster than before
except. This may be choosen by statistics info + CBO automaticlly in the future.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684432#comment-17684432
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1096965727
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
+ }
+
+ private boolean findAggRel(RelNode relNode) {
+if (relNode instanceof HepRelVertex) {
+ return findAggRel(((HepRelVertex) relNode).getCurrentRel());
+}
+if (relNode instanceof DrillAggregateRel) {
+ return true;
+}
+if (relNode.getInputs().size() == 1 && relNode.getInput(0) != null) {
+ return findAggRel(relNode.getInput(0));
+}
+return false;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillExceptRel drillExceptRel = call.rel(0);
+boolean addAggBelow =
PrelUtil.getPlannerSettings(call.getPlanner()).getOptions().getOption(EXCEPT_ADD_AGG_BELOW);
+if (addAggBelow) {
+ RelNode aggNode = new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.getInput(0),
+ImmutableBitSet.range(0,
drillExceptRel.getInput(0).getRowType().getFieldList().size()),
ImmutableList.of(), ImmutableList.of());
+ call.transformTo(drillExceptRel.copy(ImmutableList.of(aggNode,
drillExceptRel.getInput(1)), true));
+} else {
+ call.transformTo(new DrillAggregateRel(drillExceptRel.getCluster(),
drillExceptRel.getTraitSet(), drillExceptRel.copy(true),
Review Comment:
It's for performance, if the data cardinality is high, aggregate before
except may not reduce many data, if the data after except left are few,
aggregate after except will only handle few data which is faster than before
except.
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
[jira] [Commented] (DRILL-8395) Add Support for INSERT and Drop Table to GoogleSheets Plugin
[ https://issues.apache.org/jira/browse/DRILL-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684322#comment-17684322 ] ASF GitHub Bot commented on DRILL-8395: --- cgivre opened a new pull request, #2748: URL: https://github.com/apache/drill/pull/2748 # [DRILL-8395](https://issues.apache.org/jira/browse/DRILL-8395): Add Support for INSERT and Drop Table to GoogleSheets Plugin ## Description This PR adds support for INSERT queries which allow a user to append data to an existing GoogleSheets tab. It also: * Adds support for DROP TABLE queries which were not implemented * Modifies CTAS queries so that if a user executes a CTAS query with a file token, Drill will add a new tab to an existing document, but if the user executes a CTAS with a file name, it will create an entirely new document. ## Documentation Updated README. ## Testing Added unit tests and tested manually. > Add Support for INSERT and Drop Table to GoogleSheets Plugin > > > Key: DRILL-8395 > URL: https://issues.apache.org/jira/browse/DRILL-8395 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - GoogleSheets >Affects Versions: 1.20.3 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.0 > > > This PR adds support for INSERT queries which allow a user to append data to > an existing GoogleSheets tab. It also: > * Adds support for DROP TABLE queries which were not implemented > * Modifies CTAS queries so that if a user executes a CTAS query with a file > token, Drill will add a new tab to an existing document, but if the user > executes a CTAS with a file name, it will create an entirely new document. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684286#comment-17684286
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1096674511
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Thanks James! I have tried cluster.addClientFixture(), due to this method
doesn't copy all the client properties from ClusterFixture, so after create a
new ClientFixture, it still has problem to connect with ClusterFixture. And I
also tried to modify updateClient method signature to return a ClientFixture,
however due to the ClusterFixture and ClientFixture started by startCluster()
are hold by two static variables, and updateClient should be a non-static
method to refactor into ClientFixture, seems a non-static method return a value
to static variable made some errors here, and I am still figuring it out. I
think the feasible way should be started ClientFixture and ClusterFixture
separately in each test case, so that those two variables are both non-static.
The drawback here is the code change should be a little too much, but I can
submit a commit to see whether it works
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684285#comment-17684285
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1096674511
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Thanks James! I have tried cluster.addClientFixture(), due to this method
doesn't copy all the client properties from ClusterFixture, so after create a
new ClientFixture, it still has problem to connect with cluster. And I also
tried to modify updateClient method signature to return a client, however due
to the cluster started by startCluster() is hold by a static variable, and
updateClient should be a non-static method to refactor to ClientFixture, and
seems a non-static method return a value to static variable made some error
here, and I am still figuring it out. I think the feasible way should be start
ClientFixture and ClusterFixture separately in each case, so that those two
variables are both non-static. The drawback the is the code change should be a
little too much, but I can submit it commit to see whether it works
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8290) Short cut recursive file listings for LIMIT 0 queries
[
https://issues.apache.org/jira/browse/DRILL-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684215#comment-17684215
]
ASF GitHub Bot commented on DRILL-8290:
---
vvysotskyi commented on code in PR #2636:
URL: https://github.com/apache/drill/pull/2636#discussion_r1096586279
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java:
##
@@ -645,10 +646,18 @@ public Void visitOp(PhysicalOperator op,
Collection collection
}
return null;
}
-
}
protected Pair validateNode(SqlNode sqlNode) throws
ValidationException, RelConversionException, ForemanSetupException {
+if (context.getOptions().getOption(ExecConstants.FILE_LISTING_LIMIT0_OPT))
{
+ // Check for a LIMIT 0 in the root portion of the query before validation
+ // because validation of the query's FROM clauses will already trigger
+ // the recursive listing files to which FILE_LISTING_LIMIT0_OPT is meant
+ // to apply.
+ boolean rootSelectLimit0 = FindLimit0SqlVisitor.containsLimit0(sqlNode);
+ context.getPlannerSettings().setRootSelectLimit0(rootSelectLimit0);
Review Comment:
Perhaps with the query option would be simpler to pass the value rather than
extend the planner setting interface and pass it through. And schema config
already has access to query options.
> Short cut recursive file listings for LIMIT 0 queries
> -
>
> Key: DRILL-8290
> URL: https://issues.apache.org/jira/browse/DRILL-8290
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 1.21.0
>
>
> The existing LIMIT 0 query optimisations do not prevent a query run against
> the top of a deep DFS directory tree from recursively listing FileStatuses
> for everything within it using a pool of worker threads. This Issue proposes
> a new optimisation whereby such queries will recurse into the directory tree
> on a single thread that returns as soon as any single FileStatus has been
> obtained.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684213#comment-17684213
]
ASF GitHub Bot commented on DRILL-4232:
---
vvysotskyi commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1096581680
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"DrillAddAggForExceptRule");
+ protected static final Logger tracer = CalciteTrace.getPlannerTracer();
+
+ public DrillAddAggForExceptRule(RelOptRuleOperand operand, String
description) {
+super(operand, description);
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+DrillExceptRel drillExceptRel = call.rel(0);
+return !drillExceptRel.all && !drillExceptRel.isAggAdded() &&
!findAggRel(drillExceptRel.getInput(0));
Review Comment:
I'm not sure whether this check would work properly in some cases. For
example, the volcano planner will use RelSet to wrap nodes, and perhaps there
are some other cases. Instead, I propose using
`RelMetadataQuery.getUniqueKeys()` to ensure that input columns have unique
values, and if it is so, do not add aggregate. It calls methods from
`org.apache.calcite.rel.metadata.RelMdUniqueKeys` for specific node types and
should handle more cases than existing checks. In this case, the `isAggAdded`
field wouldn't be required.
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillAddAggForExceptRule.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.trace.CalciteTrace;
+import org.apache.curator.shaded.com.google.common.collect.ImmutableList;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.slf4j.Logger;
+
+import static org.apache.drill.exec.ExecConstants.EXCEPT_ADD_AGG_BELOW;
+
+/**
+ * Rule that try to add agg for Except set op.
+ */
+public class DrillAddAggForExceptRule extends RelOptRule {
+ public static final RelOptRule INSTANCE = new
DrillAddAggForExceptRule(RelOptHelper.any(DrillExceptRel.class),
"D
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683896#comment-17683896
]
ASF GitHub Bot commented on DRILL-8117:
---
jnturton commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1095876337
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Let me test out a couple of things out locally. If they work I'll send a PR
to you, otherwise we'll be ready to merge as is.
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[ https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683793#comment-17683793 ] ASF GitHub Bot commented on DRILL-4232: --- Leon-WTF commented on PR #2599: URL: https://github.com/apache/drill/pull/2599#issuecomment-1415544424 @vvysotskyi Hi, any more comments on this PR? > Support for EXCEPT set operator > --- > > Key: DRILL-4232 > URL: https://issues.apache.org/jira/browse/DRILL-4232 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Tengfei Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8393) Allow parameters to be passed to headers through SQL in WHERE clause
[
https://issues.apache.org/jira/browse/DRILL-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683705#comment-17683705
]
ASF GitHub Bot commented on DRILL-8393:
---
LYCJeff opened a new pull request, #2747:
URL: https://github.com/apache/drill/pull/2747
# [DRILL-8393](https://issues.apache.org/jira/browse/DRILL-8393): Allow
parameters to be passed to headers through SQL in WHERE clause
## Description
Allow parameters to be passed to headers through SQL in WHERE clause. Use
the _params_ configuration item to control what parameters are allowed in,
passing them into the body and header depending on the prefix.
Config:
`{ "url": "https://api.sunrise-sunset.org/json";, "requireTail": false,
"params": ["body.lat", "body.lng", "body.date", "header.header1"],
"parameterLocation": "json_body" }`
SQL Query:
```
SELECT * FROM api.sunrise
WHERE `body.lat` = 36.7201600
AND `body.lng` = -4.4203400
AND `body.date` = '2019-10-02'
AND `header.header1` = 'value1';
```
Post body:
`{ "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}`
Headers:
`{ "header1": "value1", ……}`
## Documentation
The _params_ configuration mode and SQL parameter passing mode need to be
described in the document.
## Testing
The _params_ configuration and SQL in the original unit test have been
changed and have passed this part of the unit test.
> Allow parameters to be passed to headers through SQL in WHERE clause
>
>
> Key: DRILL-8393
> URL: https://issues.apache.org/jira/browse/DRILL-8393
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - HTTP
>Affects Versions: 1.20.0
>Reporter: Yuchen Liang
>Priority: Major
> Fix For: 1.21.0
>
>
> Some APIs require parameters (e.g. digital signature) in the headers to be
> generated at access time.So I'm wondering if we can pass it in through filter
> statement.
> Perhaps we could design it like the params field in connections parameter.
> For example:
>
> Config:
> { "url": "https://api.sunrise-sunset.org/json";, "requireTail": false,
> "params": ["body.lat", "body.lng", "body.date", "header.header1"],
> "parameterLocation": "json_body" }
>
> SQL Query:
> SELECT * FROM api.sunrise
> WHERE `body.lat` = 36.7201600
> AND `body.lng` = -4.4203400
> AND `body.date` = '2019-10-02'
> AND `header.header1` = 'value1';
>
> Post body:
> { "lat": 36.7201600, "lng": -4.4203400, "date": "2019-10-02"}
>
> Headers:
> { "header1": "value1", ……}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683383#comment-17683383
]
ASF GitHub Bot commented on DRILL-8117:
---
jnturton commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094453622
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Ah, yes you're right. These methods still look a bit out of place. Can
errorMsgTestHelper move to ClientFixture? And can the new updateClient methods
be replaced by usage of cluster.addClientFixture(Properties p) and
cluster.client(int number)?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683312#comment-17683312
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094240724
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, ClientFixture doesn't have a client variable there, so that we
could't close client at first and we need to change updateClient signature from
void to return ClientFixture, besides due to cluster variable here is none
static, we also need to change updateClient from static method to none static
method, and those changes involve lots of code change. Furthermore, there are
still lots of test cases that we could just update client without restart
cluster, and for those cases that do need restart cluster, we could use
startCluster method to config new cluster and client at same time without using
updateClient. So maybe we could still keep those code here?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683313#comment-17683313
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094240724
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, ClientFixture doesn't have a client variable there, so that we
could't close client at first and we need to change updateClient signature from
void to return ClientFixture, besides due to cluster variable here is none
static, we also need to change updateClient from static method to none static
method, and those modifications involve lots of code change. Furthermore, there
are still lots of test cases that we could just update client without restart
cluster, and for those cases that do need restart cluster, we could use
startCluster method to config new cluster and client at same time without using
updateClient. So maybe we could still keep those code here?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683307#comment-17683307
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094240724
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, ClientFixture doesn't have a client variable there, so that we
could't close client at first and we need to change updateClient signature from
void to return ClientFixture, besides due to cluster variable here is none
static, we also need to change updateClient from static method to none static
method. Furthermore, there are still lots of test cases that we could just
update client without restart cluster, and for those cases that do need restart
cluster, we could use startCluster method to config new cluster and client at
same time without using updateClient. So maybe we could still keep those code
here?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683306#comment-17683306
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094240171
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
> Let's move these additions to ClientFixture. It will mean that they're no
longer drop-in replacements but they'll be more at home there.
Hi James, ClientFixture doesn't have a client variable there, so that we
could't close client at first and we need to change updateClient signature from
void to return ClientFixture, besides due to cluster variable here is none
static, we also need to change updateClient from static method to none static
method. Furthermore, there are still lots of test cases that we could just
update client without restart cluster, and for those cases that do need restart
cluster, we could use startCluster method to config new cluster and client at
same time without using updateClient. So maybe we could still keep those code
here?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683304#comment-17683304
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094238769
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, ClientFixture doesn't have a client variable there, so that we
could't close client at first and we need to change updateClient signature from
void to return ClientFixture, besides due to cluster variable here is none
static, we also need to change updateClient from static method to none static
method. Furthermore, there are still lots of test cases that we could just
update client without restart cluster, and for those cases that do need restart
cluster, we could use startCluster method to config new cluster and client at
same time without using updateClient. So maybe we could still keep those code
here?
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683276#comment-17683276
]
ASF GitHub Bot commented on DRILL-8117:
---
kingswanwho commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1094205201
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
HI James, ClientFixture doesn't have client variable, so that we could't
close client at first. And cluster here is a non-static variable, we should
change all updateClient static method to non-static method, should we still
keep changing this?
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Hi James, ClientFixture doesn't have client variable, so that we could't
close client at first. And cluster here is a non-static variable, we should
change all updateClient static method to non-static m
[jira] [Commented] (DRILL-8117) Upgrade unit tests to the cluster fixture framework
[
https://issues.apache.org/jira/browse/DRILL-8117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683044#comment-17683044
]
ASF GitHub Bot commented on DRILL-8117:
---
jnturton commented on code in PR #2499:
URL: https://github.com/apache/drill/pull/2499#discussion_r1093275447
##
exec/java-exec/src/test/java/org/apache/drill/test/ClusterTest.java:
##
@@ -125,4 +132,56 @@ public static void run(String query, Object... args)
throws Exception {
public QueryBuilder queryBuilder( ) {
return client.queryBuilder();
}
+
+ /**
+ * Utility method which tests given query produces a {@link UserException}
and the exception message contains
+ * the given message.
+ * @param testSqlQuery Test query
+ * @param expectedErrorMsg Expected error message.
+ */
+ protected static void errorMsgTestHelper(String testSqlQuery, String
expectedErrorMsg) throws Exception {
+try {
+ run(testSqlQuery);
+ fail("Expected a UserException when running " + testSqlQuery);
+} catch (UserException actualException) {
+ try {
+assertThat("message of UserException when running " + testSqlQuery,
actualException.getMessage(), containsString(expectedErrorMsg));
+ } catch (AssertionError e) {
+e.addSuppressed(actualException);
+throw e;
+ }
+}
+ }
+
+ protected static void updateClient(Properties properties) {
+if (client != null) {
+ client.close();
+ client = null;
+}
+ClientFixture.ClientBuilder clientBuilder = cluster.clientBuilder();
+if (properties != null) {
+ for (final String key : properties.stringPropertyNames()) {
+final String lowerCaseKey = key.toLowerCase();
+clientBuilder.property(lowerCaseKey, properties.getProperty(key));
+ }
+}
+client = clientBuilder.build();
+ }
+
+ protected static void updateClient(final String user) {
+updateClient(user, null);
+ }
+
+ protected static void updateClient(final String user, final String password)
{
+if (client != null) {
+ client.close();
+ client = null;
+}
+final Properties properties = new Properties();
+properties.setProperty(DrillProperties.USER, user);
+if (password != null) {
+ properties.setProperty(DrillProperties.PASSWORD, password);
+}
+updateClient(properties);
+ }
Review Comment:
Let's move these additions to ClientFixture. It will mean that they're no
longer drop-in replacements but they'll be more at home there.
> Upgrade unit tests to the cluster fixture framework
> ---
>
> Key: DRILL-8117
> URL: https://issues.apache.org/jira/browse/DRILL-8117
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.20.1
>Reporter: Jingchuan Hu
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> Upgrade various unit tests to the cluster fixture framework and replace other
> instances of deprecated code usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8394) ANALYZE TABLE ... COMPUTE STATISTICS fails with a trailing slash
[
https://issues.apache.org/jira/browse/DRILL-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682845#comment-17682845
]
ASF GitHub Bot commented on DRILL-8394:
---
jnturton merged PR #2745:
URL: https://github.com/apache/drill/pull/2745
> ANALYZE TABLE ... COMPUTE STATISTICS fails with a trailing slash
>
>
> Key: DRILL-8394
> URL: https://issues.apache.org/jira/browse/DRILL-8394
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.20.3
>Reporter: James Turton
>Assignee: James Turton
>Priority: Trivial
> Fix For: 1.21.0
>
>
> Given a directory cstest containing a single parquet file both of the
> following commands should succeed.
> {code:java}
> apache drill> analyze table dfs.tmp.`cstest` compute statistics;
> Fragment 0_0
> Number of records written 41 row selected (1.754 seconds)
> apache drill> analyze table dfs.tmp.`cstest/` compute statistics;
> ok false
> summary Table cstest/ is not supported by ANALYZE. Support is currently
> limited to directory-based Parquet tables. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8394) ANALYZE TABLE ... COMPUTE STATISTICS fails with a trailing slash
[
https://issues.apache.org/jira/browse/DRILL-8394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682481#comment-17682481
]
ASF GitHub Bot commented on DRILL-8394:
---
jnturton opened a new pull request, #2745:
URL: https://github.com/apache/drill/pull/2745
# [DRILL-8394](https://issues.apache.org/jira/browse/DRILL-8394): ANALYZE
TABLE ... COMPUTE STATISTICS fails with a trailing slash
## Description
1. Add a method to extract the table name from a SqlIdentifier, trimming
trailing slashes in the process.
2. Add a unit test.
3. Fix the spelling error in the class name
org.apache.drill.exec.planner.sql.SchemaUtilites.
## Documentation
N/A
## Testing
New unit test.
> ANALYZE TABLE ... COMPUTE STATISTICS fails with a trailing slash
>
>
> Key: DRILL-8394
> URL: https://issues.apache.org/jira/browse/DRILL-8394
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.20.3
>Reporter: James Turton
>Assignee: James Turton
>Priority: Trivial
> Fix For: 1.21.0
>
>
> Given a directory cstest containing a single parquet file both of the
> following commands should succeed.
> {code:java}
> apache drill> analyze table dfs.tmp.`cstest` compute statistics;
> Fragment 0_0
> Number of records written 41 row selected (1.754 seconds)
> apache drill> analyze table dfs.tmp.`cstest/` compute statistics;
> ok false
> summary Table cstest/ is not supported by ANALYZE. Support is currently
> limited to directory-based Parquet tables. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681659#comment-17681659
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1089851904
##
exec/java-exec/src/test/java/org/apache/drill/TestSetOp.java:
##
@@ -0,0 +1,1093 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchemaBuilder;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.drill.categories.OperatorTest;
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.nio.file.Paths;
+import java.util.List;
+
+@Category({SqlTest.class, OperatorTest.class})
+public class TestSetOp extends ClusterTest {
Review Comment:
Sure, I have added that.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681182#comment-17681182
]
ASF GitHub Bot commented on DRILL-4232:
---
Leon-WTF commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1088599209
##
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/setop/HashSetOpProbeTemplate.java:
##
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.setop;
+
+import org.apache.calcite.sql.SqlKind;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.physical.impl.common.HashPartition;
+import org.apache.drill.exec.physical.impl.join.HashJoinHelper;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.RecordBatch;
+import org.apache.drill.exec.record.RecordBatch.IterOutcome;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.record.VectorWrapper;
+import org.apache.drill.exec.vector.IntVector;
+import org.apache.drill.exec.vector.ValueVector;
+
+import java.util.ArrayList;
+
+import static org.apache.drill.exec.record.JoinBatchMemoryManager.LEFT_INDEX;
+
+public class HashSetOpProbeTemplate implements HashSetOpProbe {
Review Comment:
It's more like HashJoinProbeTemplate, I have refactored it.
> Support for EXCEPT set operator
> ---
>
> Key: DRILL-4232
> URL: https://issues.apache.org/jira/browse/DRILL-4232
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Tengfei Wang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8372) Unfreed buffers when running a LIMIT 0 query over delimited text
[
https://issues.apache.org/jira/browse/DRILL-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680906#comment-17680906
]
ASF GitHub Bot commented on DRILL-8372:
---
paul-rogers commented on code in PR #2728:
URL: https://github.com/apache/drill/pull/2728#discussion_r1087483902
##
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/limit/LimitRecordBatch.java:
##
@@ -75,7 +75,7 @@ public IterOutcome innerNext() {
upStream = next(incoming);
}
// If EMIT that means leaf operator is UNNEST, in this case refresh the
limit states and return EMIT.
- if (upStream == EMIT) {
+ if (upStream == EMIT || upStream == NONE) {
Review Comment:
This doesn't seem to be quite the right solution. This block of code is for
a very particular case: an UNNEST.
Expanding this code, look at the top of the loop:
```java
if (!first && !needMoreRecords(numberOfRecords)) {
```
With a LIMIT 0, we hit the limit on the first batch. I'm not quite sure why
the `!first` is in place. Maybe history would tell us. Perhaps the right answer
is something like:
```java
if ( !needMoreRecords(numberOfRecords)) {
outgoingSv.setRecordCount(0);
VectorAccessibleUtilities.clear(incoming);
return super.innerNext();
}
if (!first) {
...
```
I suspect that the logic actually needs more analysis. What does it do on
the first batch now? What does `super.innerNext()` do, and do we want that if
we've reached the limit?
Generally, the debugger is the best way to sort this out. Try a LIMIT 0, a
LIMIT n where n < size of the first batch, LIMIT n where n > batch size && n <
2 * batch size, etc.
> Unfreed buffers when running a LIMIT 0 query over delimited text
>
>
> Key: DRILL-8372
> URL: https://issues.apache.org/jira/browse/DRILL-8372
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.21.0
>Reporter: James Turton
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.0
>
>
> With the following data layout
>
> {code:java}
> /tmp/foo/bar:
> large_csv.csvh
> /tmp/foo/boo:
> large_csv.csvh
> {code}
> a LIMIT 0 query over it results in unfreed buffer errors.
> {code:java}
> apache drill (dfs.tmp)> select * from `foo` limit 0;
> Error: SYSTEM ERROR: IllegalStateException: Allocator[op:0:0:4:EasySubScan]
> closed with outstanding buffers allocated (3).
> Allocator(op:0:0:4:EasySubScan) 100/299008/3182592/100
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 3
> ledger[113] allocator: op:0:0:4:EasySubScan), isOwning: true, size:
> 262144, references: 1, life: 277785186322881..0, allocatorManager: [109,
> life: 277785186258906..0] holds 1 buffers.
> DrillBuf[142], udle: [110 0..262144]
> ledger[114] allocator: op:0:0:4:EasySubScan), isOwning: true, size:
> 32768, references: 1, life: 277785186463824..0, allocatorManager: [110, life:
> 277785186414654..0] holds 1 buffers.
> DrillBuf[143], udle: [111 0..32768]
> ledger[112] allocator: op:0:0:4:EasySubScan), isOwning: true, size: 4096,
> references: 1, life: 277785186046095..0, allocatorManager: [108, life:
> 277785185921147..0] holds 1 buffers.
> DrillBuf[141], udle: [109 0..4096]
> reservations: 0 {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8392) Empty Tables Causes Index Out of Bounds Exception on PDF Reader
[ https://issues.apache.org/jira/browse/DRILL-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680626#comment-17680626 ] ASF GitHub Bot commented on DRILL-8392: --- cgivre merged PR #2744: URL: https://github.com/apache/drill/pull/2744 > Empty Tables Causes Index Out of Bounds Exception on PDF Reader > --- > > Key: DRILL-8392 > URL: https://issues.apache.org/jira/browse/DRILL-8392 > Project: Apache Drill > Issue Type: Bug > Components: Format - PDF >Affects Versions: 1.20.3 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 1.21.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8392) Empty Tables Causes Index Out of Bounds Exception on PDF Reader
[ https://issues.apache.org/jira/browse/DRILL-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680255#comment-17680255 ] ASF GitHub Bot commented on DRILL-8392: --- cgivre opened a new pull request, #2744: URL: https://github.com/apache/drill/pull/2744 # [DRILL-8392](https://issues.apache.org/jira/browse/DRILL-): Empty Tables Causes Index Out of Bounds Exception on PDF Reader ## Description In certain conditions, Drill will generate an Index Out of Bounds exception. This minor PR fixes that. Basically, when a schema is not provided, Drill will ignore empty cells in rows. ## Documentation No user facing changes. ## Testing Ran existing unit tests and tested with customer data. > Empty Tables Causes Index Out of Bounds Exception on PDF Reader > --- > > Key: DRILL-8392 > URL: https://issues.apache.org/jira/browse/DRILL-8392 > Project: Apache Drill > Issue Type: Bug > Components: Format - PDF >Affects Versions: 1.20.3 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 1.21.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-4232) Support for EXCEPT set operator
[
https://issues.apache.org/jira/browse/DRILL-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679609#comment-17679609
]
ASF GitHub Bot commented on DRILL-4232:
---
vvysotskyi commented on code in PR #2599:
URL: https://github.com/apache/drill/pull/2599#discussion_r1083511030
##
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillSetOpRel.java:
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.calcite.linq4j.Ord;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.InvalidRelException;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.common.logical.data.Union;
+import org.apache.drill.exec.planner.common.DrillSetOpRelBase;
+
+import java.util.List;
+
+/**
+ * SetOp implemented in Drill.
+ */
+public class DrillSetOpRel extends DrillSetOpRelBase implements DrillRel {
Review Comment:
I think instead of having a common class for the intersect and except, it
would be better to differ them as it is done in Calcite and extend their
implementations, so it will help to remove defining custom `estimateRowCount`
methods and allow using more optimizations designed for these operators, like
push down to JDBC and so on.
##
exec/java-exec/src/test/java/org/apache/drill/TestSetOp.java:
##
@@ -0,0 +1,1093 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchemaBuilder;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.shaded.guava.com.google.common.collect.Lists;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.drill.categories.OperatorTest;
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.nio.file.Paths;
+import java.util.List;
+
+@Category({SqlTest.class, OperatorTest.class})
+public class TestSetOp extends ClusterTest {
Review Comment:
It is good that it is handled in the code, but it could also be fine to have
a test that verifies it works as expected, so we will be sure that no future
changes will break it.
##
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/setop/HashSetOpProbeTemplate.java:
##
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you
[jira] [Commented] (DRILL-8391) Set autocomplete="off" on the password field of web UI login forms
[
https://issues.apache.org/jira/browse/DRILL-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679174#comment-17679174
]
ASF GitHub Bot commented on DRILL-8391:
---
jnturton commented on PR #2743:
URL: https://github.com/apache/drill/pull/2743#issuecomment-1398480919
The dang squash and merge mangled the commit message!
> Set autocomplete="off" on the password field of web UI login forms
> --
>
> Key: DRILL-8391
> URL: https://issues.apache.org/jira/browse/DRILL-8391
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.20.3
>Reporter: James Turton
>Assignee: James Turton
>Priority: Trivial
> Fix For: 1.21.0
>
>
> In order to avoid triggering security scanners it is necessary to set
> autocomplete = "off" on the password field in the web UI login form. This
> change probably has no real world security benefit because
> {quote}Even without a master password, in-browser password management is
> generally seen as a net gain for security. Since users do not have to
> remember passwords that the browser stores for them, they are able to choose
> stronger passwords than they would otherwise.
> For this reason, many modern browsers do not support {{autocomplete="off"}}
> for login fields:
> {quote}
> *
>
> {quote}If a site sets {{autocomplete="off"}} for a
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form],
> and the form includes username and password input fields, then the browser
> still offers to remember this login, and if the user agrees, the browser will
> autofill those fields the next time the user visits the page.
> {quote} *
> {quote}If a site sets {{autocomplete="off"}} for username and password
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input]
> fields, then the browser still offers to remember this login, and if the user
> agrees, the browser will autofill those fields the next time the user visits
> the page
> {quote}
> Excerpt taken from [this Mozilla Developer Network
> page|https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8391) Set autocomplete="off" on the password field of web UI login forms
[
https://issues.apache.org/jira/browse/DRILL-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679172#comment-17679172
]
ASF GitHub Bot commented on DRILL-8391:
---
cgivre merged PR #2743:
URL: https://github.com/apache/drill/pull/2743
> Set autocomplete="off" on the password field of web UI login forms
> --
>
> Key: DRILL-8391
> URL: https://issues.apache.org/jira/browse/DRILL-8391
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.20.3
>Reporter: James Turton
>Assignee: James Turton
>Priority: Trivial
> Fix For: 1.21.0
>
>
> In order to avoid triggering security scanners it is necessary to set
> autocomplete = "off" on the password field in the web UI login form. This
> change probably has no real world security benefit because
> {quote}Even without a master password, in-browser password management is
> generally seen as a net gain for security. Since users do not have to
> remember passwords that the browser stores for them, they are able to choose
> stronger passwords than they would otherwise.
> For this reason, many modern browsers do not support {{autocomplete="off"}}
> for login fields:
> {quote}
> *
>
> {quote}If a site sets {{autocomplete="off"}} for a
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form],
> and the form includes username and password input fields, then the browser
> still offers to remember this login, and if the user agrees, the browser will
> autofill those fields the next time the user visits the page.
> {quote} *
> {quote}If a site sets {{autocomplete="off"}} for username and password
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input]
> fields, then the browser still offers to remember this login, and if the user
> agrees, the browser will autofill those fields the next time the user visits
> the page
> {quote}
> Excerpt taken from [this Mozilla Developer Network
> page|https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8290) Short cut recursive file listings for LIMIT 0 queries
[ https://issues.apache.org/jira/browse/DRILL-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679170#comment-17679170 ] ASF GitHub Bot commented on DRILL-8290: --- jnturton commented on PR #2636: URL: https://github.com/apache/drill/pull/2636#issuecomment-1398472306 > For such queries the same QueryComputationHints will be used for both inputs, so it will cause incorrect results. @vvysotskyi the idea here was that only a LIMIT 0 on the _root_ SELECT is detected, in which case the single file optimisation can be done on _all_ inputs so a single flag is sufficient. However, I'm trying to implement a better approach that optimises LIMIT 0s at any level. Since files are first listed very early, during validation (so even before partition pruning 🙁) no RelNode trees are available and the detection will have to be done on the SqlNode tree. > Short cut recursive file listings for LIMIT 0 queries > - > > Key: DRILL-8290 > URL: https://issues.apache.org/jira/browse/DRILL-8290 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning & Optimization >Affects Versions: 1.20.2 >Reporter: James Turton >Priority: Minor > Fix For: 1.21.0 > > > The existing LIMIT 0 query optimisations do not prevent a query run against > the top of a deep DFS directory tree from recursively listing FileStatuses > for everything within it using a pool of worker threads. This Issue proposes > a new optimisation whereby such queries will recurse into the directory tree > on a single thread that returns as soon as any single FileStatus has been > obtained. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8391) Disable auto complete on the password field of web UI login forms
[
https://issues.apache.org/jira/browse/DRILL-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679117#comment-17679117
]
ASF GitHub Bot commented on DRILL-8391:
---
jnturton opened a new pull request, #2743:
URL: https://github.com/apache/drill/pull/2743
# [DRILL-8391](https://issues.apache.org/jira/browse/DRILL-8391): Disable
auto complete on the password field of web UI login forms
## Description
In order to avoid triggering security scanners it is necessary to set
autocomplete = "off" on the password field in the web UI login forms. This
change probably has no real world security benefit because
> Even without a master password, in-browser password management is
generally seen as a net gain for security. Since users do not have to remember
passwords that the browser stores for them, they are able to choose stronger
passwords than they would otherwise.
>
> For this reason, many modern browsers do not support autocomplete="off"
for login fields:
>
> - If a site sets autocomplete="off" for a form, and the form includes
username and password input fields, then the browser still offers to remember
this login, and if the user agrees, the browser will autofill those fields the
next time the user visits the page.
> - If a site sets autocomplete="off" for username and password input
fields, then the browser still offers to remember this login, and if the user
agrees, the browser will autofill those fields the next time the user visits
the page
Excerpt taken from [this Mozilla Developer Network
page](https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion).
## Documentation
N/A
## Testing
Confirm that the attribute assignment `autocomplete="off"` is present on the
password of the web UI login form.
> Disable auto complete on the password field of web UI login forms
> -
>
> Key: DRILL-8391
> URL: https://issues.apache.org/jira/browse/DRILL-8391
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.20.3
>Reporter: James Turton
>Assignee: James Turton
>Priority: Trivial
> Fix For: 1.21.0
>
>
> In order to avoid triggering security scanners it is necessary to set
> autocomplete = "off" on the password field in the web UI login form. This
> change probably has no real world security benefit because
> {quote}Even without a master password, in-browser password management is
> generally seen as a net gain for security. Since users do not have to
> remember passwords that the browser stores for them, they are able to choose
> stronger passwords than they would otherwise.
> For this reason, many modern browsers do not support {{autocomplete="off"}}
> for login fields:
> {quote}
> *
>
> {quote}If a site sets {{autocomplete="off"}} for a
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form],
> and the form includes username and password input fields, then the browser
> still offers to remember this login, and if the user agrees, the browser will
> autofill those fields the next time the user visits the page.
> {quote} *
> {quote}If a site sets {{autocomplete="off"}} for username and password
> [{{}}|https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input]
> fields, then the browser still offers to remember this login, and if the user
> agrees, the browser will autofill those fields the next time the user visits
> the page
> {quote}
> Excerpt taken from [this Mozilla Developer Network
> page|https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8390) Minor Improvements to PDF Reader
[ https://issues.apache.org/jira/browse/DRILL-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678708#comment-17678708 ] ASF GitHub Bot commented on DRILL-8390: --- cgivre merged PR #2742: URL: https://github.com/apache/drill/pull/2742 > Minor Improvements to PDF Reader > > > Key: DRILL-8390 > URL: https://issues.apache.org/jira/browse/DRILL-8390 > Project: Apache Drill > Issue Type: Improvement > Components: Format - PDF >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > > This PR makes some minor improvements to the PDF reader including: > * Fixes a minor bug where certain configurations the first row of data was > skipped > * Fixes a minor bug where empty tables were causing crashes with the > spreadsheet extraction algorithm was used > * Adds a table_count metadata field > * Adds a table_index metadata field to reflect the current table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8390) Minor Improvements to PDF Reader
[ https://issues.apache.org/jira/browse/DRILL-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678473#comment-17678473 ] ASF GitHub Bot commented on DRILL-8390: --- cgivre opened a new pull request, #2742: URL: https://github.com/apache/drill/pull/2742 # [DRILL-8390](https://issues.apache.org/jira/browse/DRILL-8390): Minor Improvements to PDF Reader ## Description This PR makes some minor improvements to the PDF reader including: Fixes a minor bug where certain configurations the first row of data was skipped Fixes a minor bug where empty tables were causing crashes with the spreadsheet extraction algorithm was used Adds a `_table_count` metadata field Adds a `_table_index` metadata field to reflect the current table. ## Documentation See above. Updated README. ## Testing Ran existing unit tests. Manually tested against customer data. > Minor Improvements to PDF Reader > > > Key: DRILL-8390 > URL: https://issues.apache.org/jira/browse/DRILL-8390 > Project: Apache Drill > Issue Type: Improvement > Components: Format - PDF >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > > This PR makes some minor improvements to the PDF reader including: > * Fixes a minor bug where certain configurations the first row of data was > skipped > * Fixes a minor bug where empty tables were causing crashes with the > spreadsheet extraction algorithm was used > * Adds a table_count metadata field > * Adds a table_index metadata field to reflect the current table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-5033) Query on JSON that has null as value for each key
[
https://issues.apache.org/jira/browse/DRILL-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678036#comment-17678036
]
ASF GitHub Bot commented on DRILL-5033:
---
cgivre closed pull request #2731: DRILL-5033: Query on JSON That Has Null as
Value For Each Key
URL: https://github.com/apache/drill/pull/2731
> Query on JSON that has null as value for each key
> -
>
> Key: DRILL-5033
> URL: https://issues.apache.org/jira/browse/DRILL-5033
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> Drill 1.9.0 git commit ID : 83513daf
> Drill returns same result with or without `store.json.all_text_mode`=true
> Note that each key in the JSON has null as its value.
> [root@cent01 null_eq_joins]# cat right_all_nulls.json
> {
> "intKey" : null,
> "bgintKey": null,
> "strKey": null,
> "boolKey": null,
> "fltKey": null,
> "dblKey": null,
> "timKey": null,
> "dtKey": null,
> "tmstmpKey": null,
> "intrvldyKey": null,
> "intrvlyrKey": null
> }
> [root@cent01 null_eq_joins]#
> Querying the above JSON file results in null as query result.
> - We should see each of the keys in the JSON as a column in query result.
> - And in each column the value should be a null value.
> Current behavior does not look right.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from `right_all_nulls.json`;
> +---+
> | * |
> +---+
> | null |
> +---+
> 1 row selected (0.313 seconds)
> {noformat}
> Adding comment from [~julianhyde]
> IMHO it is similar but not the same as DRILL-1256. Worth logging an issue and
> let [~jnadeau] (or someone) put on the record what should be the behavior of
> an empty record (empty JSON map) when it is top-level (as in this case) or in
> a collection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-5033) Query on JSON that has null as value for each key
[
https://issues.apache.org/jira/browse/DRILL-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678035#comment-17678035
]
ASF GitHub Bot commented on DRILL-5033:
---
cgivre commented on PR #2731:
URL: https://github.com/apache/drill/pull/2731#issuecomment-1386304077
I'm going to close this PR. If there is any objection, we can revisit.
> Query on JSON that has null as value for each key
> -
>
> Key: DRILL-5033
> URL: https://issues.apache.org/jira/browse/DRILL-5033
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> Drill 1.9.0 git commit ID : 83513daf
> Drill returns same result with or without `store.json.all_text_mode`=true
> Note that each key in the JSON has null as its value.
> [root@cent01 null_eq_joins]# cat right_all_nulls.json
> {
> "intKey" : null,
> "bgintKey": null,
> "strKey": null,
> "boolKey": null,
> "fltKey": null,
> "dblKey": null,
> "timKey": null,
> "dtKey": null,
> "tmstmpKey": null,
> "intrvldyKey": null,
> "intrvlyrKey": null
> }
> [root@cent01 null_eq_joins]#
> Querying the above JSON file results in null as query result.
> - We should see each of the keys in the JSON as a column in query result.
> - And in each column the value should be a null value.
> Current behavior does not look right.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from `right_all_nulls.json`;
> +---+
> | * |
> +---+
> | null |
> +---+
> 1 row selected (0.313 seconds)
> {noformat}
> Adding comment from [~julianhyde]
> IMHO it is similar but not the same as DRILL-1256. Worth logging an issue and
> let [~jnadeau] (or someone) put on the record what should be the behavior of
> an empty record (empty JSON map) when it is top-level (as in this case) or in
> a collection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
