Re: January Apache Drill board report
+1. Looks good! On Thu, Jan 31, 2019 at 9:15 AM Vitalii Diravka wrote: > +1 > > Kind regards > Vitalii > > > On Thu, Jan 31, 2019 at 6:18 PM Aman Sinha wrote: > > > Thanks for putting this together, Arina. > > The Drill Developer Day and Meetup were separate events, so you can split > > them up. > > - A half day Drill Developer Day was held on Nov 14. A variety of > > technical design issues were discussed. > > - A Drill user meetup was held on the same evening. 2 presentations - > > one on use case for Drill and one about indexing support in Drill were > > presented. > > > > Rest of the report LGTM. > > > > -Aman > > > > > > On Thu, Jan 31, 2019 at 7:58 AM Arina Ielchiieva > wrote: > > > > > Hi all, > > > > > > please take a look at the draft board report for the last quarter and > let > > > me know if you have any comments. > > > > > > Thanks, > > > Arina > > > > > > = > > > > > > ## Description: > > > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > > > Storage. > > > > > > ## Issues: > > > - There are no issues requiring board attention at this time. > > > > > > ## Activity: > > > - Since the last board report, Drill has released version 1.15.0, > > >including the following enhancements: > > >- Add capability to do index based planning and execution > > >- CROSS join support > > >- INFORMATION_SCHEMA FILES and FUNCTIONS were added > > >- Support for TIMESTAMPADD and TIMESTAMPDIFF functions > > >- Ability to secure znodes with custom ACLs > > >- Upgrade to SQLLine 1.6 > > >- Parquet filter pushdown for VARCHAR and DECIMAL data types > > >- Support JPPD (Join Predicate Push Down) > > >- Lateral join functionality was enabled by default > > >- Multiple Web UI improvements to simplify the use of options and > > submit > > > queries > > >- Query performance with the semi-join functionality was improved > > >- Support for aliases in the GROUP BY clause > > >- Options to return null for empty string and prevents Drill from > > > returning > > > a result set for DDL statements > > >- Storage plugin names became case-insensitive > > > > > > - Drill developer meet up was held on November 14, 2018. > > > > > > ## Health report: > > > - The project is healthy. Development activity > > >as reflected in the pull requests and JIRAs is good. > > > - Activity on the dev and user mailing lists are stable. > > > - Three committers were added in the last period. > > > > > > ## PMC changes: > > > > > > - Currently 23 PMC members. > > > - No new PMC members added in the last 3 months > > > - Last PMC addition was Charles Givre on Mon Sep 03 2018 > > > > > > ## Committer base changes: > > > > > > - Currently 51 committers. > > > - New commmitters: > > > - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 > > > - Karthikeyan Manivannan was added as a committer on Fri Dec 07 > 2018 > > > - Salim Achouche was added as a committer on Mon Dec 17 2018 > > > > > > ## Releases: > > > > > > - 1.15.0 was released on Mon Dec 31 2018 > > > > > > ## Mailing list activity: > > > > > > - dev@drill.apache.org: > > > - 415 subscribers (down -12 in the last 3 months): > > > - 2066 emails sent to list (2653 in previous quarter) > > > > > > - iss...@drill.apache.org: > > > - 18 subscribers (up 0 in the last 3 months): > > > - 2480 emails sent to list (3228 in previous quarter) > > > > > > - u...@drill.apache.org: > > > - 592 subscribers (down -5 in the last 3 months): > > > - 249 emails sent to list (310 in previous quarter) > > > > > > > > > ## JIRA activity: > > > > > > - 196 JIRA tickets created in the last 3 months > > > - 171 JIRA tickets closed/resolved in the last 3 months > > > > > >
Re: January Apache Drill board report
Boaz, As much as I appreciate the publicity, I think we mentioned it in the last report. — C > On Jan 31, 2019, at 23:43, Boaz Ben-Zvi wrote: > > The report looks good ; thanks. > > One more item that may deserve a mention is the publishing or Charles' and > Paul's O'REILLY Book "Learning Apache Drill" in Nov. 2018. > > Thanks, > > Boaz > > On 1/31/19 8:18 AM, Aman Sinha wrote: >> Thanks for putting this together, Arina. >> The Drill Developer Day and Meetup were separate events, so you can split >> them up. >> - A half day Drill Developer Day was held on Nov 14. A variety of >> technical design issues were discussed. >> - A Drill user meetup was held on the same evening. 2 presentations - >> one on use case for Drill and one about indexing support in Drill were >> presented. >> >> Rest of the report LGTM. >> >> -Aman >> >> >> On Thu, Jan 31, 2019 at 7:58 AM Arina Ielchiieva wrote: >> >>> Hi all, >>> >>> please take a look at the draft board report for the last quarter and let >>> me know if you have any comments. >>> >>> Thanks, >>> Arina >>> >>> = >>> >>> ## Description: >>> - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud >>> Storage. >>> >>> ## Issues: >>> - There are no issues requiring board attention at this time. >>> >>> ## Activity: >>> - Since the last board report, Drill has released version 1.15.0, >>>including the following enhancements: >>>- Add capability to do index based planning and execution >>>- CROSS join support >>>- INFORMATION_SCHEMA FILES and FUNCTIONS were added >>>- Support for TIMESTAMPADD and TIMESTAMPDIFF functions >>>- Ability to secure znodes with custom ACLs >>>- Upgrade to SQLLine 1.6 >>>- Parquet filter pushdown for VARCHAR and DECIMAL data types >>>- Support JPPD (Join Predicate Push Down) >>>- Lateral join functionality was enabled by default >>>- Multiple Web UI improvements to simplify the use of options and submit >>> queries >>>- Query performance with the semi-join functionality was improved >>>- Support for aliases in the GROUP BY clause >>>- Options to return null for empty string and prevents Drill from >>> returning >>> a result set for DDL statements >>>- Storage plugin names became case-insensitive >>> >>> - Drill developer meet up was held on November 14, 2018. >>> >>> ## Health report: >>> - The project is healthy. Development activity >>>as reflected in the pull requests and JIRAs is good. >>> - Activity on the dev and user mailing lists are stable. >>> - Three committers were added in the last period. >>> >>> ## PMC changes: >>> >>> - Currently 23 PMC members. >>> - No new PMC members added in the last 3 months >>> - Last PMC addition was Charles Givre on Mon Sep 03 2018 >>> >>> ## Committer base changes: >>> >>> - Currently 51 committers. >>> - New commmitters: >>> - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 >>> - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 >>> - Salim Achouche was added as a committer on Mon Dec 17 2018 >>> >>> ## Releases: >>> >>> - 1.15.0 was released on Mon Dec 31 2018 >>> >>> ## Mailing list activity: >>> >>> - dev@drill.apache.org: >>> - 415 subscribers (down -12 in the last 3 months): >>> - 2066 emails sent to list (2653 in previous quarter) >>> >>> - iss...@drill.apache.org: >>> - 18 subscribers (up 0 in the last 3 months): >>> - 2480 emails sent to list (3228 in previous quarter) >>> >>> - u...@drill.apache.org: >>> - 592 subscribers (down -5 in the last 3 months): >>> - 249 emails sent to list (310 in previous quarter) >>> >>> >>> ## JIRA activity: >>> >>> - 196 JIRA tickets created in the last 3 months >>> - 171 JIRA tickets closed/resolved in the last 3 months >>>
Re: January Apache Drill board report
The report looks good ; thanks. One more item that may deserve a mention is the publishing or Charles' and Paul's O'REILLY Book "Learning Apache Drill" in Nov. 2018. Thanks, Boaz On 1/31/19 8:18 AM, Aman Sinha wrote: Thanks for putting this together, Arina. The Drill Developer Day and Meetup were separate events, so you can split them up. - A half day Drill Developer Day was held on Nov 14. A variety of technical design issues were discussed. - A Drill user meetup was held on the same evening. 2 presentations - one on use case for Drill and one about indexing support in Drill were presented. Rest of the report LGTM. -Aman On Thu, Jan 31, 2019 at 7:58 AM Arina Ielchiieva wrote: Hi all, please take a look at the draft board report for the last quarter and let me know if you have any comments. Thanks, Arina = ## Description: - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Since the last board report, Drill has released version 1.15.0, including the following enhancements: - Add capability to do index based planning and execution - CROSS join support - INFORMATION_SCHEMA FILES and FUNCTIONS were added - Support for TIMESTAMPADD and TIMESTAMPDIFF functions - Ability to secure znodes with custom ACLs - Upgrade to SQLLine 1.6 - Parquet filter pushdown for VARCHAR and DECIMAL data types - Support JPPD (Join Predicate Push Down) - Lateral join functionality was enabled by default - Multiple Web UI improvements to simplify the use of options and submit queries - Query performance with the semi-join functionality was improved - Support for aliases in the GROUP BY clause - Options to return null for empty string and prevents Drill from returning a result set for DDL statements - Storage plugin names became case-insensitive - Drill developer meet up was held on November 14, 2018. ## Health report: - The project is healthy. Development activity as reflected in the pull requests and JIRAs is good. - Activity on the dev and user mailing lists are stable. - Three committers were added in the last period. ## PMC changes: - Currently 23 PMC members. - No new PMC members added in the last 3 months - Last PMC addition was Charles Givre on Mon Sep 03 2018 ## Committer base changes: - Currently 51 committers. - New commmitters: - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 - Salim Achouche was added as a committer on Mon Dec 17 2018 ## Releases: - 1.15.0 was released on Mon Dec 31 2018 ## Mailing list activity: - dev@drill.apache.org: - 415 subscribers (down -12 in the last 3 months): - 2066 emails sent to list (2653 in previous quarter) - iss...@drill.apache.org: - 18 subscribers (up 0 in the last 3 months): - 2480 emails sent to list (3228 in previous quarter) - u...@drill.apache.org: - 592 subscribers (down -5 in the last 3 months): - 249 emails sent to list (310 in previous quarter) ## JIRA activity: - 196 JIRA tickets created in the last 3 months - 171 JIRA tickets closed/resolved in the last 3 months
Applications of Apache Drill
Hi everyone As some of you might be aware that the Apache Drill project is into its 5th year of existence and from the rich exchange on the mailing lists, it is clear that it has proven to be of tremendous value to not just enterprise customers using Drill in interesting ways in business-critical applications, but even students who are novices in the space of big data. That said, some of us feel that the a banner page from the community on how Apache Drill is used at various companies in solving different problems is a great way to further generate interest in the project. We're looking to start a 'Power-By-Drill' page on the Apache Drill website, where with the company logo and a short description (< 400 characters) of how Drill is used. If you feel the character limit is restrictive, in addition to the description, you can provide a link to resources that discuss their applications in more detail. Please reach out to me with a reply to this email at the following address ( kunal[at]apache.org ). Please do not hesitate to ask any questions you have as well. Kunal kunal[at]apache.org
Some queries related to writing a custom storage-plugin
Hi All, I am relatively new to Drill and trying to write a custom storage plugin. I have couple of (naive sounding) queries, so mostly need some brief pointers: a) Why do a StoragePlugin have to implement a registerSchemas() (coming from SchemaFactory)? I assumed that drill would discover the data-schema on-the-fly, so that shouldn't be a need for the plugin to register it beforehand. (I created a version of my plugin and skipped implementing the registerSchemas method, assuming it will be discovered, and tried to do a* "SELECT * FROM myplugin.`tableid`"* and it threw a *"VALIDATION_ERROR: Schema [[myplugin]] is not valid with respect to either root schema or current default schema"* --> So, I suspect that might be due to me not implementing registerSchemas(), hence the question) b) Similarly, I see plugins creating their own DrillTable class extending either DrillTable/DynamicDrill and then have to override the RelDataType getRowType(RelDataTypeFactory typeFactory) method, that seems to be converting the relation-item to drill types. But, I see similar type conversion also being done in the RecordReader classes when creating and loading the value-vectors. Am I reading it right, that we are doing it twice? Any pointers will be greatly appreciated. Thanks, Lokendra
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252845060
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertThat;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+
+ private static final String runtimeFilterOption = "ALTER SESSION SET `" +
+ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY + "` = %s";
+
+ private static final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
+ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private static final String runtimeFilterWaitTimeOption = "ALTER SESSION SET
`" +
+ExecConstants.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+client.alterSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY,
true);
+String queryPlan = queryBuilder().sql(sql).explainText();
+assertThat("Query plan doesn't contain RuntimeFilter. This may happen if
plan is not distributed and has no " +
+"exchange operator in it.", queryPlan,
containsString("RuntimeFilter"));
+ }
+
+ /**
+ * Verifies that result of a query with join condition doesn't changes with
and without Runtime Filter for correctness
+ */
+ @Test
+ public void testHashJoinCorrectnessWithRuntimeFilter() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+String allTestQueryOptions = runtimeFilterOption + "; " +
runtimeFilterWaitOption + "; " +
Review comment:
It would be better to introduce a complete string with these options as
constant instead of concatenating options in test methods and remove other
constants.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252845831
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertThat;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+
+ private static final String runtimeFilterOption = "ALTER SESSION SET `" +
Review comment:
Please rename these constants to the upper case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sohami commented on issue #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
sohami commented on issue #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order … URL: https://github.com/apache/drill/pull/1628#issuecomment-459493180 @vvysotskyi - Address your review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sohami commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
sohami commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252827074
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+// don't use client.alterSession since it's an async request and hence
there is no guarantee than runtime filter
Review comment:
Thanks for catching this. So earlier usage was throwing exception which
`runSqlSilently` was consuming and hence during debugging I got an impression
that second request was sent before first one is completed. But as you
mentioned the listener in `run` is waiting for completion to create summary and
hence its not async. Updated to use `alterSession` correctly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (DRILL-7021) HTTPD Throws NPE and Doesn't Recognize Timeformat
Charles Givre created DRILL-7021: Summary: HTTPD Throws NPE and Doesn't Recognize Timeformat Key: DRILL-7021 URL: https://issues.apache.org/jira/browse/DRILL-7021 Project: Apache Drill Issue Type: Bug Components: Storage - Other Affects Versions: 1.16.0 Reporter: Charles Givre Assignee: Charles Givre The HTTPD format plugin throws a NPE when you attempt to query all fields and enumerate them in the query. The query below will throw the NPE: ``` SELECT `request_referer_query`, . . . . . . .semicolon> `request_referer_ref`, . . . . . . .semicolon> `request_firstline_uri_port`, . . . . . . .semicolon> `request_firstline_method`, . . . . . . .semicolon> `request_firstline_uri_protocol`, . . . . . . .semicolon> `request_receive_time_millisecond`, . . . . . . .semicolon> `request_receive_time_day__utc`, . . . . . . .semicolon> `request_receive_time_year__utc`, . . . . . . .semicolon> `request_receive_time_weekofweekyear__utc`, . . . . . . .semicolon> `request_receive_time_second`, . . . . . . .semicolon> `connection_client_host`, . . . . . . .semicolon> `request_referer_userinfo`, . . . . . . .semicolon> `request_referer_path`, . . . . . . .semicolon> `request_referer_host`, . . . . . . .semicolon> `request_receive_time_monthname`, . . . . . . .semicolon> `request_receive_time_month__utc`, . . . . . . .semicolon> `request_firstline_uri_ref`, . . . . . . .semicolon> `request_firstline_protocol_version`, . . . . . . .semicolon> `request_receive_time_minute`, . . . . . . .semicolon> `request_firstline`, . . . . . . .semicolon> `request_receive_time_day`, . . . . . . .semicolon> `request_referer_port`, . . . . . . .semicolon> `request_receive_time_year`, . . . . . . .semicolon> `request_referer_query_$`, . . . . . . .semicolon> `request_firstline_uri_query_$`, . . . . . . .semicolon> `request_firstline_uri`, . . . . . . .semicolon> `request_receive_time_month`, . . . . . . .semicolon> `request_receive_time_weekofweekyear`, . . . . . . .semicolon> `request_firstline_uri_userinfo`, . . . . . . .semicolon> `request_referer`, . . . . . . .semicolon> `request_receive_time_epoch`, . . . . . . .semicolon> `request_referer_protocol`, . . . . . . .semicolon> `request_receive_time_monthname__utc`, . . . . . . .semicolon> `connection_client_logname`, . . . . . . .semicolon> `request_receive_time`, . . . . . . .semicolon> `request_firstline_protocol`, . . . . . . .semicolon> `request_receive_time_hour`, . . . . . . .semicolon> `request_firstline_uri_host`, . . . . . . .semicolon> `request_firstline_uri_path`, . . . . . . .semicolon> `request_user-agent`, . . . . . . .semicolon> `request_receive_time_hour__utc`, . . . . . . .semicolon> `request_receive_time_second__utc`, . . . . . . .semicolon> `request_receive_time_weekyear`, . . . . . . .semicolon> `request_receive_time_timezone`, . . . . . . .semicolon> `request_receive_time_weekyear__utc`, . . . . . . .semicolon> `response_body_bytesclf`, . . . . . . .semicolon> `connection_client_user`, . . . . . . .semicolon> `request_receive_time_millisecond__utc`, . . . . . . .semicolon> `request_status_last`, . . . . . . .semicolon> `request_firstline_uri_query`, . . . . . . .semicolon> `request_receive_time_minute__utc` . . . . . . .semicolon> FROM `dfs.drillclass`.`hackers-access.httpd` ``` The cause for the NPE is that several fields were missing from a type map in the format plugin. Separately, the format plugin is not recognizing the time stamp and is not parsing dates as time formats. Oh... and the unit tests suck. Sorry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Histogram design doc
Hi devs, I have updated DRILL-6992 (histogram support) [1] with a design proposal. Please take a look and provide feedback either in the JIRA or in the doc (make sure to use your email login such that the comments don't show as anonymous). [1] https://issues.apache.org/jira/browse/DRILL-6992 Thanks, Aman
[GitHub] kkhatua commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
kkhatua commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252798223
##
File path: exec/java-exec/src/main/resources/rest/static/js/querySubmission.js
##
@@ -53,7 +53,18 @@ function doSubmitQueryWithUserName() {
//Wrap & Submit Query (invoked directly if impersonation is not enabled)
function wrapAndSubmitQuery() {
//Wrap if required
-wrapQuery();
+var mustWrapWithLimit = $('input[name="forceLimit"]:checked').length > 0;
+//Clear field when submitting if not mustWrapWithLimit
+if (!mustWrapWithLimit) {
+ //Wipe out any numeric entry in the field before
+ $('#autoLimit').attr('value', '');
+} else {
+ let autoLimitValue=document.getElementById("autoLimit").value;
+ if (isNaN(autoLimitValue)) {
+alert(autoLimitValue+ " is not a number. Please fill in a valid
number");
Review comment:
We're already using the alert mechanism for empty username (when
impersonation is enabled). In both cases, we want to indicate that the user
cannot progress because of a missing username or invalid number. An error
message on the query page might get missed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sachouche commented on issue #1630: DRILL-7018: Fixed Parquet buffer overflow when reading timestamp column
sachouche commented on issue #1630: DRILL-7018: Fixed Parquet buffer overflow when reading timestamp column URL: https://github.com/apache/drill/pull/1630#issuecomment-459431892 Thanks Vitali for the review @vdiravka! The loss is due to the way that Parquet INT96 encodes the timestamp information: - 4 bytes used for the julian day and then the rest for nanosecond precision - Formula to convert to unix epoch: (julian_day - 2440588) * (86400 * 1000 * 1000 * 1000) + nanoseconds` - I believe Parquet borrowed this format from Hive / Impala; you can refer to this [PR](https://github.com/apache/parquet-format/pull/49) **NOTE -** My understanding is that INT64 is the preferred format as Spark (few years back) dropped INT96 support in favor to INT64. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: January Apache Drill board report
+1 Kind regards Vitalii On Thu, Jan 31, 2019 at 6:18 PM Aman Sinha wrote: > Thanks for putting this together, Arina. > The Drill Developer Day and Meetup were separate events, so you can split > them up. > - A half day Drill Developer Day was held on Nov 14. A variety of > technical design issues were discussed. > - A Drill user meetup was held on the same evening. 2 presentations - > one on use case for Drill and one about indexing support in Drill were > presented. > > Rest of the report LGTM. > > -Aman > > > On Thu, Jan 31, 2019 at 7:58 AM Arina Ielchiieva wrote: > > > Hi all, > > > > please take a look at the draft board report for the last quarter and let > > me know if you have any comments. > > > > Thanks, > > Arina > > > > = > > > > ## Description: > > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > > Storage. > > > > ## Issues: > > - There are no issues requiring board attention at this time. > > > > ## Activity: > > - Since the last board report, Drill has released version 1.15.0, > >including the following enhancements: > >- Add capability to do index based planning and execution > >- CROSS join support > >- INFORMATION_SCHEMA FILES and FUNCTIONS were added > >- Support for TIMESTAMPADD and TIMESTAMPDIFF functions > >- Ability to secure znodes with custom ACLs > >- Upgrade to SQLLine 1.6 > >- Parquet filter pushdown for VARCHAR and DECIMAL data types > >- Support JPPD (Join Predicate Push Down) > >- Lateral join functionality was enabled by default > >- Multiple Web UI improvements to simplify the use of options and > submit > > queries > >- Query performance with the semi-join functionality was improved > >- Support for aliases in the GROUP BY clause > >- Options to return null for empty string and prevents Drill from > > returning > > a result set for DDL statements > >- Storage plugin names became case-insensitive > > > > - Drill developer meet up was held on November 14, 2018. > > > > ## Health report: > > - The project is healthy. Development activity > >as reflected in the pull requests and JIRAs is good. > > - Activity on the dev and user mailing lists are stable. > > - Three committers were added in the last period. > > > > ## PMC changes: > > > > - Currently 23 PMC members. > > - No new PMC members added in the last 3 months > > - Last PMC addition was Charles Givre on Mon Sep 03 2018 > > > > ## Committer base changes: > > > > - Currently 51 committers. > > - New commmitters: > > - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 > > - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 > > - Salim Achouche was added as a committer on Mon Dec 17 2018 > > > > ## Releases: > > > > - 1.15.0 was released on Mon Dec 31 2018 > > > > ## Mailing list activity: > > > > - dev@drill.apache.org: > > - 415 subscribers (down -12 in the last 3 months): > > - 2066 emails sent to list (2653 in previous quarter) > > > > - iss...@drill.apache.org: > > - 18 subscribers (up 0 in the last 3 months): > > - 2480 emails sent to list (3228 in previous quarter) > > > > - u...@drill.apache.org: > > - 592 subscribers (down -5 in the last 3 months): > > - 249 emails sent to list (310 in previous quarter) > > > > > > ## JIRA activity: > > > > - 196 JIRA tickets created in the last 3 months > > - 171 JIRA tickets closed/resolved in the last 3 months > > >
[GitHub] ihuzenko commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
ihuzenko commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0 URL: https://github.com/apache/drill/pull/1631#discussion_r252737406 ## File path: exec/java-exec/src/main/codegen/data/Parser.tdd ## @@ -89,8 +89,766 @@ ] # List of keywords from "keywords" section that are not reserved. - nonReservedKeywords: [ - ] + # Added due to comment under CALCITE-2405. Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
ihuzenko commented on a change in pull request #1631: DRILL-6862: Update
Calcite to 1.18.0
URL: https://github.com/apache/drill/pull/1631#discussion_r252737282
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet2/TestDrillParquetReader.java
##
@@ -434,4 +434,20 @@ public void testLogicalIntTypes5() throws Exception {
.build().run();
}
+ @Test // DRILL-6856
+ public void testIsTrueOrNullCondition() throws Exception {
+Object[] expectedRowValues = new Object[9];
+// 7 true values and 2 null values in the end
+Arrays.fill(expectedRowValues, 0, expectedRowValues.length - 2, true);
+testBuilder()
+.sqlQuery("SELECT bool_col " +
+"FROM cp.`parquet2/0_0_0.parquet` " +
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
ihuzenko commented on a change in pull request #1631: DRILL-6862: Update
Calcite to 1.18.0
URL: https://github.com/apache/drill/pull/1631#discussion_r252737240
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet2/TestDrillParquetReader.java
##
@@ -434,4 +434,20 @@ public void testLogicalIntTypes5() throws Exception {
.build().run();
}
+ @Test // DRILL-6856
+ public void testIsTrueOrNullCondition() throws Exception {
+Object[] expectedRowValues = new Object[9];
+// 7 true values and 2 null values in the end
+Arrays.fill(expectedRowValues, 0, expectedRowValues.length - 2, true);
+testBuilder()
+.sqlQuery("SELECT bool_col " +
+"FROM cp.`parquet2/0_0_0.parquet` " +
+"WHERE bool_col IS true OR bool_col IS null " +
+"ORDER BY bool_col")
+.ordered()
+.baselineColumns("bool_col")
+.baselineValuesForSingleColumn(expectedRowValues)
+.build().run();
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
Re: January Apache Drill board report
Thanks for putting this together, Arina. The Drill Developer Day and Meetup were separate events, so you can split them up. - A half day Drill Developer Day was held on Nov 14. A variety of technical design issues were discussed. - A Drill user meetup was held on the same evening. 2 presentations - one on use case for Drill and one about indexing support in Drill were presented. Rest of the report LGTM. -Aman On Thu, Jan 31, 2019 at 7:58 AM Arina Ielchiieva wrote: > Hi all, > > please take a look at the draft board report for the last quarter and let > me know if you have any comments. > > Thanks, > Arina > > = > > ## Description: > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > Storage. > > ## Issues: > - There are no issues requiring board attention at this time. > > ## Activity: > - Since the last board report, Drill has released version 1.15.0, >including the following enhancements: >- Add capability to do index based planning and execution >- CROSS join support >- INFORMATION_SCHEMA FILES and FUNCTIONS were added >- Support for TIMESTAMPADD and TIMESTAMPDIFF functions >- Ability to secure znodes with custom ACLs >- Upgrade to SQLLine 1.6 >- Parquet filter pushdown for VARCHAR and DECIMAL data types >- Support JPPD (Join Predicate Push Down) >- Lateral join functionality was enabled by default >- Multiple Web UI improvements to simplify the use of options and submit > queries >- Query performance with the semi-join functionality was improved >- Support for aliases in the GROUP BY clause >- Options to return null for empty string and prevents Drill from > returning > a result set for DDL statements >- Storage plugin names became case-insensitive > > - Drill developer meet up was held on November 14, 2018. > > ## Health report: > - The project is healthy. Development activity >as reflected in the pull requests and JIRAs is good. > - Activity on the dev and user mailing lists are stable. > - Three committers were added in the last period. > > ## PMC changes: > > - Currently 23 PMC members. > - No new PMC members added in the last 3 months > - Last PMC addition was Charles Givre on Mon Sep 03 2018 > > ## Committer base changes: > > - Currently 51 committers. > - New commmitters: > - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 > - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 > - Salim Achouche was added as a committer on Mon Dec 17 2018 > > ## Releases: > > - 1.15.0 was released on Mon Dec 31 2018 > > ## Mailing list activity: > > - dev@drill.apache.org: > - 415 subscribers (down -12 in the last 3 months): > - 2066 emails sent to list (2653 in previous quarter) > > - iss...@drill.apache.org: > - 18 subscribers (up 0 in the last 3 months): > - 2480 emails sent to list (3228 in previous quarter) > > - u...@drill.apache.org: > - 592 subscribers (down -5 in the last 3 months): > - 249 emails sent to list (310 in previous quarter) > > > ## JIRA activity: > > - 196 JIRA tickets created in the last 3 months > - 171 JIRA tickets closed/resolved in the last 3 months >
Re: January Apache Drill board report
Looks good, +1 Kind regards, Volodymyr Vysotskyi On Thu, Jan 31, 2019 at 5:58 PM Arina Ielchiieva wrote: > Hi all, > > please take a look at the draft board report for the last quarter and let > me know if you have any comments. > > Thanks, > Arina > > = > > ## Description: > - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud > Storage. > > ## Issues: > - There are no issues requiring board attention at this time. > > ## Activity: > - Since the last board report, Drill has released version 1.15.0, >including the following enhancements: >- Add capability to do index based planning and execution >- CROSS join support >- INFORMATION_SCHEMA FILES and FUNCTIONS were added >- Support for TIMESTAMPADD and TIMESTAMPDIFF functions >- Ability to secure znodes with custom ACLs >- Upgrade to SQLLine 1.6 >- Parquet filter pushdown for VARCHAR and DECIMAL data types >- Support JPPD (Join Predicate Push Down) >- Lateral join functionality was enabled by default >- Multiple Web UI improvements to simplify the use of options and submit > queries >- Query performance with the semi-join functionality was improved >- Support for aliases in the GROUP BY clause >- Options to return null for empty string and prevents Drill from > returning > a result set for DDL statements >- Storage plugin names became case-insensitive > > - Drill developer meet up was held on November 14, 2018. > > ## Health report: > - The project is healthy. Development activity >as reflected in the pull requests and JIRAs is good. > - Activity on the dev and user mailing lists are stable. > - Three committers were added in the last period. > > ## PMC changes: > > - Currently 23 PMC members. > - No new PMC members added in the last 3 months > - Last PMC addition was Charles Givre on Mon Sep 03 2018 > > ## Committer base changes: > > - Currently 51 committers. > - New commmitters: > - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 > - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 > - Salim Achouche was added as a committer on Mon Dec 17 2018 > > ## Releases: > > - 1.15.0 was released on Mon Dec 31 2018 > > ## Mailing list activity: > > - dev@drill.apache.org: > - 415 subscribers (down -12 in the last 3 months): > - 2066 emails sent to list (2653 in previous quarter) > > - iss...@drill.apache.org: > - 18 subscribers (up 0 in the last 3 months): > - 2480 emails sent to list (3228 in previous quarter) > > - u...@drill.apache.org: > - 592 subscribers (down -5 in the last 3 months): > - 249 emails sent to list (310 in previous quarter) > > > ## JIRA activity: > > - 196 JIRA tickets created in the last 3 months > - 171 JIRA tickets closed/resolved in the last 3 months >
[GitHub] arina-ielchiieva commented on issue #1615: DRILL-6964: Implement CREATE / DROP SCHEMA commands
arina-ielchiieva commented on issue #1615: DRILL-6964: Implement CREATE / DROP SCHEMA commands URL: https://github.com/apache/drill/pull/1615#issuecomment-459402669 @vvysotskyi thanks, addressed code review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
January Apache Drill board report
Hi all, please take a look at the draft board report for the last quarter and let me know if you have any comments. Thanks, Arina = ## Description: - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Since the last board report, Drill has released version 1.15.0, including the following enhancements: - Add capability to do index based planning and execution - CROSS join support - INFORMATION_SCHEMA FILES and FUNCTIONS were added - Support for TIMESTAMPADD and TIMESTAMPDIFF functions - Ability to secure znodes with custom ACLs - Upgrade to SQLLine 1.6 - Parquet filter pushdown for VARCHAR and DECIMAL data types - Support JPPD (Join Predicate Push Down) - Lateral join functionality was enabled by default - Multiple Web UI improvements to simplify the use of options and submit queries - Query performance with the semi-join functionality was improved - Support for aliases in the GROUP BY clause - Options to return null for empty string and prevents Drill from returning a result set for DDL statements - Storage plugin names became case-insensitive - Drill developer meet up was held on November 14, 2018. ## Health report: - The project is healthy. Development activity as reflected in the pull requests and JIRAs is good. - Activity on the dev and user mailing lists are stable. - Three committers were added in the last period. ## PMC changes: - Currently 23 PMC members. - No new PMC members added in the last 3 months - Last PMC addition was Charles Givre on Mon Sep 03 2018 ## Committer base changes: - Currently 51 committers. - New commmitters: - Hanumath Rao Maduri was added as a committer on Thu Nov 01 2018 - Karthikeyan Manivannan was added as a committer on Fri Dec 07 2018 - Salim Achouche was added as a committer on Mon Dec 17 2018 ## Releases: - 1.15.0 was released on Mon Dec 31 2018 ## Mailing list activity: - dev@drill.apache.org: - 415 subscribers (down -12 in the last 3 months): - 2066 emails sent to list (2653 in previous quarter) - iss...@drill.apache.org: - 18 subscribers (up 0 in the last 3 months): - 2480 emails sent to list (3228 in previous quarter) - u...@drill.apache.org: - 592 subscribers (down -5 in the last 3 months): - 249 emails sent to list (310 in previous quarter) ## JIRA activity: - 196 JIRA tickets created in the last 3 months - 171 JIRA tickets closed/resolved in the last 3 months
[GitHub] ihuzenko edited a comment on issue #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko edited a comment on issue #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query URL: https://github.com/apache/drill/pull/1608#issuecomment-459396389 Hello @kkhatua . Thanks for fixing issues, I've checked changes and added review comments. Also I'd like to suggest remember user's choice and don't reset autoLimit checkbox and value after query submission. I think you could try [Window.sessionStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage) for this. Also please squash commits into one with name ```DRILL-6960: Auto Limit Wrapping should not apply to non-select query``` and description of changes below, like [here](https://github.com/apache/drill/pull/1631/commits/48f60f9a923222b4f8e42808085844f8fbf0be64) cc: @vdiravka This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on issue #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on issue #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query URL: https://github.com/apache/drill/pull/1608#issuecomment-459396389 Hello @kkhatua . Thanks for fixing issues, I've checked changes and added review comments. Also I'd like to suggest remember user's choice and don't reset autoLimit checkbox and value after query submission. I think you could try [Window.sessionStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage) for this. Also please squash commits into one with name ```DRILL-6960: Auto Limit Wrapping should not apply to non-select query``` and description of changes below, like [here](https://github.com/apache/drill/pull/1631/commits/48f60f9a923222b4f8e42808085844f8fbf0be64) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252707797
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryResources.java
##
@@ -83,10 +83,14 @@ public QueryResult submitQueryJSON(QueryWrapper query)
throws Exception {
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.TEXT_HTML)
public Viewable submitQuery(@FormParam("query") String query,
- @FormParam("queryType") String queryType) throws
Exception {
+ @FormParam("queryType") String queryType,
+ @FormParam("autoLimit") String autoLimit
+ ) throws Exception {
try {
final String trimmedQueryString =
CharMatcher.is(';').trimTrailingFrom(query.trim());
- final QueryResult result = submitQueryJSON(new
QueryWrapper(trimmedQueryString, queryType));
+ //Extract option of AutoLimit
+ String autoLimitValue = (autoLimit != null &&
autoLimit.matches("[0-9]+")) ? autoLimit : "";
Review comment:
To avoid double pattern check (here and in ```QueryWrapper```'s
constructor), simply pass ```autoLimitValue``` to constructor of
```QueryWrapper``` and then within constructor set :
```java
this.autoLimitRowCount = autoLimit != null && autoLimit.matches("[0-9]+") ?
new Integer(autoLimit) : null;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252693016
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java
##
@@ -69,14 +75,24 @@ public QueryType getType() {
return QueryType.valueOf(queryType);
}
+ public Integer getAutoLimitRowCount() {
+return autoLimitRowCount;
+ }
+
+ public boolean isAutoLimitEnabled() {
Review comment:
could you please check again whether we actually need this method, and
remove it if not
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252684099
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
##
@@ -135,7 +135,7 @@ private static PhysicalPlan getQueryPlan(QueryContext
context, String sql, Point
final SqlConverter parser = new SqlConverter(context);
injector.injectChecked(context.getExecutionControls(), "sql-parsing",
ForemanSetupException.class);
-final SqlNode sqlNode = parser.parse(sql);
+final SqlNode sqlNode = wrapWithAutoLimit(parser, context, sql);
Review comment:
Could you replace the line with:
```suggestion
final SqlNode sqlNode = parser.parse(sql);
if (context.isAutoLimitEnabled()) {
boolean applied = applyAutoLimit(sqlNode, sqlNode.getKind(),
context.getAutoLimitRowCount());
if (!applied) {
context.disableAutoLimit();
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252682700
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
##
@@ -273,6 +279,21 @@ public RemoteFunctionRegistry getRemoteFunctionRegistry()
{
return drillbitContext.getRemoteFunctionRegistry();
}
+ public boolean isAutoLimitEnabled() {
+return (autoLimitRowCount != null);
Review comment:
please remove redundant parenthesis and add javadoc
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252694825
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java
##
@@ -136,15 +136,31 @@
*/
public Foreman(final WorkerBee bee, final DrillbitContext drillbitContext,
final UserClientConnection connection, final QueryId queryId, final
RunQuery queryRequest) {
+this(bee, drillbitContext, connection, queryId, queryRequest, null);
+ }
+
+ /**
+ * Constructor. Sets up the Foreman, but does not initiate any execution.
+ *
+ * @param bee work manager (runs fragments)
+ * @param drillbitContext drillbit context
+ * @param connection connection
+ * @param queryId the id for the query
+ * @param queryRequest the query to execute
+ * @param autoLimitRowCount the number of rows to limit in execution (used
for WebRequests)
+ */
+ public Foreman(final WorkerBee bee, final DrillbitContext drillbitContext,
+ final UserClientConnection connection, final QueryId queryId, final
RunQuery queryRequest,
+ final Integer autoLimitRowCount) {
this.queryId = queryId;
this.queryIdString = QueryIdHelper.getQueryId(queryId);
this.queryRequest = queryRequest;
this.drillbitContext = drillbitContext;
this.initiatingClient = connection;
this.closeFuture = initiatingClient.getChannelClosureFuture();
closeFuture.addListener(closeListener);
-
-this.queryContext = new QueryContext(connection.getSession(),
drillbitContext, queryId);
+this.queryContext = new QueryContext(connection.getSession(),
drillbitContext, queryId,
+(autoLimitRowCount == null) ? null : autoLimitRowCount);
Review comment:
this null check is redundant here, just pass ```autoLimitRowCount```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252707385
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java
##
@@ -46,15 +46,21 @@
private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
private final String query;
-
private final String queryType;
+ private final Integer autoLimitRowCount;
private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
@JsonCreator
- public QueryWrapper(@JsonProperty("query") String query,
@JsonProperty("queryType") String queryType) {
+ public QueryWrapper(@JsonProperty("query") String query,
@JsonProperty("queryType") String queryType, @JsonProperty("autoLimit") String
autoLimit) {
this.query = query;
this.queryType = queryType.toUpperCase();
+this.autoLimitRowCount = autoLimit.matches("[0-9]+") ? new
Integer(autoLimit) : null;
+ }
+
+ @JsonCreator
+ public QueryWrapper(@JsonProperty("query") String query,
@JsonProperty("queryType") String queryType) {
+this(query, queryType, "");
Review comment:
to avoid redundant pattern check :
```suggestion
this(query, queryType, null);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252687106
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
##
@@ -183,4 +183,24 @@ private static PhysicalPlan getQueryPlan(QueryContext
context, String sql, Point
return handler.getPlan(sqlNode);
}
+
Review comment:
I'd like to replace the constant and method defined below, with:
```suggestion
private static boolean applyAutoLimit(SqlNode sqlNode, SqlKind kind,
Integer autoLimit) {
if (kind == SqlKind.SELECT) {
SqlNumericLiteral fetch =
SqlNumericLiteral.createExactNumeric(autoLimit.toString(),
sqlNode.getParserPosition());
((SqlSelect) sqlNode).setFetch(fetch);
return true;
} else if (sqlNode instanceof SqlWith) {
SqlNode withBody = ((SqlWith) sqlNode).body;
return applyAutoLimit(withBody, withBody.getKind(), autoLimit);
}
return false;
}
```
Suggested method avoids double parsing of SQL string and handles WITH
queries.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252707066
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java
##
@@ -46,15 +46,21 @@
private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
private final String query;
-
private final String queryType;
+ private final Integer autoLimitRowCount;
private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
@JsonCreator
- public QueryWrapper(@JsonProperty("query") String query,
@JsonProperty("queryType") String queryType) {
+ public QueryWrapper(@JsonProperty("query") String query,
@JsonProperty("queryType") String queryType, @JsonProperty("autoLimit") String
autoLimit) {
this.query = query;
this.queryType = queryType.toUpperCase();
+this.autoLimitRowCount = autoLimit.matches("[0-9]+") ? new
Integer(autoLimit) : null;
Review comment:
As suggested in previous comment :
```suggestion
this.autoLimitRowCount = autoLimit != null && autoLimit.matches("[0-9]+")
? new Integer(autoLimit) : null;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252694100
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java
##
@@ -69,14 +75,24 @@ public QueryType getType() {
return QueryType.valueOf(queryType);
}
+ public Integer getAutoLimitRowCount() {
+return autoLimitRowCount;
+ }
+
+ public boolean isAutoLimitEnabled() {
+return getAutoLimitRowCount() != null;
+ }
+
public QueryResult run(final WorkManager workManager, final
WebUserConnection webUserConnection) throws Exception {
final RunQuery runQuery = RunQuery.newBuilder().setType(getType())
.setPlan(getQuery())
.setResultsMode(QueryResultsMode.STREAM_FULL)
.build();
// Submit user query to Drillbit work queue.
-final QueryId queryId =
workManager.getUserWorker().submitWork(webUserConnection, runQuery);
+final QueryId queryId = isAutoLimitEnabled() ?
Review comment:
I think this check is redundant here, because when deciding to apply limit
you're using method from ```QueryContext``` . Here you can just invoke
``` workManager.getUserWorker().submitWork(webUserConnection, runQuery,
autoLimitRowCount)``` .
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252681891
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
##
@@ -85,8 +86,13 @@
private DrillOperatorTable table;
public QueryContext(final UserSession session, final DrillbitContext
drillbitContext, QueryId queryId) {
+this(session, drillbitContext, queryId, null);
+ }
+
+ public QueryContext(final UserSession session, final DrillbitContext
drillbitContext, QueryId queryId, Integer autoLimit) {
Review comment:
please remove redundant ```final``` modifiers for parameters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252696632
##
File path: exec/java-exec/src/main/resources/rest/static/js/querySubmission.js
##
@@ -53,7 +53,18 @@ function doSubmitQueryWithUserName() {
//Wrap & Submit Query (invoked directly if impersonation is not enabled)
function wrapAndSubmitQuery() {
//Wrap if required
-wrapQuery();
+var mustWrapWithLimit = $('input[name="forceLimit"]:checked').length > 0;
+//Clear field when submitting if not mustWrapWithLimit
+if (!mustWrapWithLimit) {
+ //Wipe out any numeric entry in the field before
+ $('#autoLimit').attr('value', '');
+} else {
+ let autoLimitValue=document.getElementById("autoLimit").value;
+ if (isNaN(autoLimitValue)) {
+alert(autoLimitValue+ " is not a number. Please fill in a valid
number");
Review comment:
Can you show the error message somewhere on query page instead of alerting
it ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit Wrapping should not apply to non-select query
ihuzenko commented on a change in pull request #1608: DRILL-6960: Auto Limit
Wrapping should not apply to non-select query
URL: https://github.com/apache/drill/pull/1608#discussion_r252681985
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
##
@@ -85,8 +86,13 @@
private DrillOperatorTable table;
public QueryContext(final UserSession session, final DrillbitContext
drillbitContext, QueryId queryId) {
Review comment:
please remove redundant ```final``` modifiers for parameters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1615: DRILL-6964: Implement CREATE / DROP SCHEMA commands
vvysotskyi commented on a change in pull request #1615: DRILL-6964: Implement
CREATE / DROP SCHEMA commands
URL: https://github.com/apache/drill/pull/1615#discussion_r252704706
##
File path:
exec/java-exec/src/test/java/org/apache/drill/TestSchemaCommands.java
##
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.metadata.schema.PathSchemaProvider;
+import org.apache.drill.exec.record.metadata.schema.SchemaContainer;
+import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.apache.hadoop.fs.Path;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+@Category(SqlTest.class)
+public class TestSchemaCommands extends ClusterTest {
+
+ @Rule
+ public ExpectedException thrown = ExpectedException.none();
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testCreateWithoutSchema() throws Exception {
+thrown.expect(UserException.class);
+thrown.expectMessage("PARSE ERROR: Lexical error");
+
+run("create schema for");
+ }
+
+ @Test
+ public void testCreateWithForAndPath() throws Exception {
+thrown.expect(UserException.class);
+thrown.expectMessage("PARSE ERROR: Encountered \"path\"");
+
+run("create schema ( col1 int, col2 int) for table tbl path
'/tmp/schema.file'");
+ }
+
+ @Test
+ public void testCreateWithPathAndOrReplace() throws Exception {
+thrown.expect(UserException.class);
+thrown.expectMessage("PARSE ERROR: cannot be used with
property");
+
+run("create or replace schema (col1 int, col2 int) path
'/tmp/schema.file'");
+ }
+
+ @Test
+ public void testCreateForMissingTable() throws Exception {
+String table = "dfs.tmp.tbl";
+thrown.expect(UserException.class);
+thrown.expectMessage("VALIDATION ERROR: Table [tbl] was not found");
+
+run("create schema (col1 int, col2 int) for table %s", table);
+ }
+
+ @Test
+ public void testCreateForTemporaryTable() throws Exception {
+String table = "temp_create";
+try {
+ run("create temporary table %s as select 'a' as c from (values(1))",
table);
+ thrown.expect(UserException.class);
+ thrown.expectMessage(String.format("VALIDATION ERROR: Indicated table
[%s] is temporary table", table));
+
+ run("create schema (col1 int, col2 int) for table %s", table);
+} finally {
+ run("drop table if exists %s", table);
+}
+ }
+
+ @Test
+ public void testCreateForImmutableSchema() throws Exception {
+String table = "sys.version";
+thrown.expect(UserException.class);
+thrown.expectMessage("VALIDATION ERROR: Unable to create or drop objects.
Schema [sys] is immutable");
+
+run("create schema (col1 int, col2 int) for table %s", table);
+ }
+
+ @Test
+ public void testMissingDirectory() throws Exception {
+File tmpDir = dirTestWatcher.getTmpDir();
+Path schema = new Path(Paths.get(tmpDir.getPath(),
"missing_parent_directory", "file.schema").toFile().getPath());
+
+thrown.expect(UserException.class);
+
[GitHub] vvysotskyi commented on a change in pull request #1615: DRILL-6964: Implement CREATE / DROP SCHEMA commands
vvysotskyi commented on a change in pull request #1615: DRILL-6964: Implement CREATE / DROP SCHEMA commands URL: https://github.com/apache/drill/pull/1615#discussion_r252651556 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SchemaHandler.java ## @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.planner.sql.handlers; + +import org.apache.calcite.schema.Schema; +import org.apache.calcite.schema.SchemaPlus; +import org.apache.calcite.schema.Table; +import org.apache.calcite.sql.SqlNode; +import org.apache.commons.io.IOUtils; +import org.apache.drill.common.exceptions.UserException; +import org.apache.drill.exec.ExecConstants; +import org.apache.drill.exec.physical.PhysicalPlan; +import org.apache.drill.exec.planner.sql.DirectPlan; +import org.apache.drill.exec.planner.sql.SchemaUtilites; +import org.apache.drill.exec.planner.sql.parser.SqlCreateType; +import org.apache.drill.exec.planner.sql.parser.SqlSchema; +import org.apache.drill.exec.record.metadata.schema.FsMetastoreSchemaProvider; +import org.apache.drill.exec.record.metadata.schema.PathSchemaProvider; +import org.apache.drill.exec.record.metadata.schema.SchemaProvider; +import org.apache.drill.exec.store.AbstractSchema; +import org.apache.drill.exec.store.StorageStrategy; +import org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory; +import org.apache.drill.exec.util.ImpersonationUtil; +import org.apache.drill.shaded.guava.com.google.common.base.Preconditions; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; + +import java.io.IOException; +import java.io.InputStream; +import java.util.List; + +/** + * Parent class for CREATE / DROP SCHEMA handles. Review comment: `handles` -> `handlers` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0 URL: https://github.com/apache/drill/pull/1631#discussion_r252630557 ## File path: exec/java-exec/src/main/codegen/data/Parser.tdd ## @@ -89,8 +89,766 @@ ] # List of keywords from "keywords" section that are not reserved. - nonReservedKeywords: [ - ] + # Added due to comment under CALCITE-2405. Review comment: Could you please rephrase this comment: ``` Copied from calcite-core config.fmpp For details please see comment under CALCITE-2405. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update
Calcite to 1.18.0
URL: https://github.com/apache/drill/pull/1631#discussion_r252631999
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet2/TestDrillParquetReader.java
##
@@ -434,4 +434,20 @@ public void testLogicalIntTypes5() throws Exception {
.build().run();
}
+ @Test // DRILL-6856
+ public void testIsTrueOrNullCondition() throws Exception {
+Object[] expectedRowValues = new Object[9];
+// 7 true values and 2 null values in the end
+Arrays.fill(expectedRowValues, 0, expectedRowValues.length - 2, true);
+testBuilder()
+.sqlQuery("SELECT bool_col " +
+"FROM cp.`parquet2/0_0_0.parquet` " +
+"WHERE bool_col IS true OR bool_col IS null " +
+"ORDER BY bool_col")
+.ordered()
+.baselineColumns("bool_col")
+.baselineValuesForSingleColumn(expectedRowValues)
+.build().run();
Review comment:
`.build().run()` -> `.go()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update Calcite to 1.18.0
vvysotskyi commented on a change in pull request #1631: DRILL-6862: Update
Calcite to 1.18.0
URL: https://github.com/apache/drill/pull/1631#discussion_r252634318
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet2/TestDrillParquetReader.java
##
@@ -434,4 +434,20 @@ public void testLogicalIntTypes5() throws Exception {
.build().run();
}
+ @Test // DRILL-6856
+ public void testIsTrueOrNullCondition() throws Exception {
+Object[] expectedRowValues = new Object[9];
+// 7 true values and 2 null values in the end
+Arrays.fill(expectedRowValues, 0, expectedRowValues.length - 2, true);
+testBuilder()
+.sqlQuery("SELECT bool_col " +
+"FROM cp.`parquet2/0_0_0.parquet` " +
Review comment:
Is it possible to use an existing table or at least create it in the test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ihuzenko commented on issue #1631: DRILL-6862: Update Calcite to 1.18.0
ihuzenko commented on issue #1631: DRILL-6862: Update Calcite to 1.18.0 URL: https://github.com/apache/drill/pull/1631#issuecomment-459308293 @vvysotskyi could you review please ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ihuzenko opened a new pull request #1631: DRILL-6862: Update Calcite to 1.18.0
ihuzenko opened a new pull request #1631: DRILL-6862: Update Calcite to 1.18.0 URL: https://github.com/apache/drill/pull/1631 1. Moved Calcite dependency from profile hadoop-default to general dependency managment 2. Updated Calcite version to 1.18.0-drill-r0 and Avatica version to 1.13.0 3. Hook.REL_BUILDER_SIMPLIFY moved to static block, cause now it can't be removed (fixes DRILL-6830) 4. Removed WrappedAccessor, since it was workaround fixed in CALCITE-1408 5. Fixed setting of multiple options in TestBuilder 6. Timstampadd type inference aligned with CALCITE-2699 7. Dependency update caused 417 kB increase of jdb-all jar size, so the maxsize limit was increased from 39.5 to 40 MB 8. Added test into TestDrillParquetReader to ensure that DRILL-6856 was fixed by Calcite update This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252619645
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+// don't use client.alterSession since it's an async request and hence
there is no guarantee than runtime filter
+// option will be enabled before sending the actual query
+client.runQueriesAndLog(String.format(runtimeFilterOption, "true"));
+UserProtos.QueryPlanFragments planFragments = client.planQuery(sql);
Review comment:
The easier way of obtaining plan string is using
`queryBuilder().sql(query).explainText()` method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252622522
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
Review comment:
Please remove `@throws Exception` line, since IDE highlights it as the
warning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252621395
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+// don't use client.alterSession since it's an async request and hence
there is no guarantee than runtime filter
+// option will be enabled before sending the actual query
+client.runQueriesAndLog(String.format(runtimeFilterOption, "true"));
+UserProtos.QueryPlanFragments planFragments = client.planQuery(sql);
+assertTrue(planFragments.toString().contains("runtime-filter"));
+ }
+
+ /**
+ * Verifies that result of a query with join condition doesn't changes with
and without Runtime Filter for correctness
+ * @throws Exception
+ */
+ @Test
+ public void testHashJoinCorrectnessWithRuntimeFilter() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+String allTestQueryOptions = runtimeFilterOption + "; " +
runtimeFilterWaitOption + "; " +
+ runtimeFilterWaitTimeOption;
+client.testBuilder()
Review comment:
No need to specify `client.testBuilder()`, `ClusterTest` has method
`testBuilder()` which already calls it on the client.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252620923
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+// don't use client.alterSession since it's an async request and hence
there is no guarantee than runtime filter
+// option will be enabled before sending the actual query
+client.runQueriesAndLog(String.format(runtimeFilterOption, "true"));
+UserProtos.QueryPlanFragments planFragments = client.planQuery(sql);
+assertTrue(planFragments.toString().contains("runtime-filter"));
Review comment:
Could you please change
`assertTrue(planFragments.toString().contains("runtime-filter"));` to
`assertThat("some useful message", plan, containsString("runtime-filter"));` so
in the case when this test will fail, assertion error message will contain the
plan and expected string instead of "Expected true but was false".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252617363
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
++ "` = %s";
+
+ private final String runtimeFilterWaitOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY + "` = %s";
+
+ private final String runtimeFilterWaitTimeOption = "ALTER SESSION SET `" +
ExecConstants
+.HASHJOIN_RUNTIME_FILTER_MAX_WAITING_TIME_KEY + "` = %d";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+dirTestWatcher.copyResourceToRoot(Paths.get("tpchmulti"));
+// Reduce the slice target so that there are multiple minor fragments with
exchange, otherwise RuntimeFilter
+// will not be inserted in the plan
+startCluster(ClusterFixture.builder(dirTestWatcher)
+ .clusterSize(2)
+ .maxParallelization(1)
+ .systemOption(ExecConstants.SLICE_TARGET, 10));
+ }
+
+ @After
+ public void tearDown() {
+client.resetSession(ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY);
+
client.resetSession(ExecConstants.HASHJOIN_RUNTIME_FILTER_WAITING_ENABLE_KEY);
+ }
+
+ /**
+ * Test to make sure runtime filter is inserted in the plan. This is to
ensure that with current cluster setup and
+ * system options distributed plan is generated rather than single fragment
plan. Since in later case RuntimeFilter
+ * will not be inserted.
+ * @throws Exception
+ */
+ @Test
+ public void testRuntimeFilterPresentInPlan() throws Exception {
+String sql = "SELECT l.n_name, r.r_name FROM dfs.`tpchmulti/nation` l,
dfs.`tpchmulti/region/` r where " +
+ "l.n_regionkey = r.r_regionkey";
+
+// don't use client.alterSession since it's an async request and hence
there is no guarantee than runtime filter
Review comment:
Could you please explain, why `client.alterSession` is an async request?
It uses `ClientFixture.runSqlSilently()` method, which calls
`queryBuilder().sql(sql).run()` which executres the query and returns its
`QuerySummary`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252617514
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
Review comment:
Please remove the commented out line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query result with RuntimeFilter enabled when order …
vvysotskyi commented on a change in pull request #1628: DRILL-7016: Wrong query
result with RuntimeFilter enabled when order …
URL: https://github.com/apache/drill/pull/1628#discussion_r252617717
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestHashJoinJPPDCorrectness.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.join;
+
+import org.apache.drill.categories.SqlTest;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserProtos;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.junit.After;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import static org.junit.Assert.assertTrue;
+import java.nio.file.Paths;
+
+@Category(SqlTest.class)
+public class TestHashJoinJPPDCorrectness extends ClusterTest {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestHashJoinJPPDCorrectness.class);
+
+ private final String runtimeFilterOption = "ALTER SESSION SET `" +
ExecConstants.HASHJOIN_ENABLE_RUNTIME_FILTER_KEY
Review comment:
static?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
