[GitHub] drill issue #258: DRILL-4091: Support for additional gis operations in gis c...
Github user amansinha100 commented on the issue: https://github.com/apache/drill/pull/258 Somehow these extensions to the GIS module never got reviewed ! (the original GIS functions are in the contrib). @k255 would you be able to rebase your PR onto the latest master branch ? Hopefully someone familiar with the semantics of these functions can do a quick review and then we will merge it in. Thanks for your contribution ! ---
[GitHub] drill issue #258: DRILL-4091: Support for additional gis operations in gis c...
Github user joeauty commented on the issue: https://github.com/apache/drill/pull/258 I'm super excited about this! Just wondering, is there plans for ST_ASGeoJSON? ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147262655 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java --- @@ -59,6 +59,9 @@ @Ignore public class ExampleTest { + @Rule + public final DirTestWatcher dirTestWatcher = new DirTestWatcher(); --- End diff -- I added java doc, and added example of how it's used in secondTest ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147261203 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/ClusterFixture.java --- @@ -226,43 +217,16 @@ private void createConfig(FixtureBuilder builder) throws Exception { serviceSet = null; usesZk = true; - isLocal = false; } else { // Embedded Drillbit. serviceSet = RemoteServiceSet.getLocalServiceSet(); - isLocal = true; } } - private void startDrillbits(FixtureBuilder builder) throws Exception { -//// Ensure that Drill uses the log directory determined here rather than -//// it's hard-coded defaults. WIP: seems to be needed some times but -//// not others. -// -//String logDir = null; -//if (builder.tempDir != null) { -// logDir = builder.tempDir.getAbsolutePath(); -//} -//if (logDir == null) { -// logDir = config.getString(ExecConstants.DRILL_TMP_DIR); -// if (logDir != null) { -//logDir += "/drill/log"; -// } -//} -//if (logDir == null) { -// logDir = "/tmp/drill"; -//} -//new File(logDir).mkdirs(); -//System.setProperty("drill.log-dir", logDir); - -dfsTestTempDir = makeTempDir("dfs-test"); - -// Clean up any files that may have been left from the -// last run. - -preserveLocalFiles = builder.preserveLocalFiles; -removeLocalFiles(); + private void startDrillbits() throws Exception { +dfsTestTempDir = new File(getRootDir(), "dfs-test"); +dfsTestTempDir.mkdirs(); --- End diff -- This is possible through the BaseDirTestWatcher. It is configured through the BaseDireTestWatcher's constructor. If you pass it false it will not delete your directories at the end of each test. ---
Re: [ANNOUNCE] New Committer: Kamesh Bhallamudi
Congratulations, Kamesh! On Thu, Oct 26, 2017 at 1:17 PM, Parth Chandrawrote: > The Project Management Committee (PMC) for Apache Drill has invited Kamesh > Bhallamudi to become a committer, and we are pleased to announce that he > has accepted. > > Kamesh (along with Anil Kumar) is the developer of the MongoDB storage > plugin and is also working on the Kafka plugin. > > Welcome Kamesh, and thank you for your contributions. Keep up the good > work ! > > - Parth > (on behalf of the Apache Drill PMC) >
Re: [ANNOUNCE] New Committer: Anil Kumar Batchu
Congratulations, Anil! On Thu, Oct 26, 2017 at 1:17 PM, Parth Chandrawrote: > The Project Management Committee (PMC) for Apache Drill has invited Anil > Kumar Batchu to become a committer, and we are pleased to announce that he > has accepted. > > Anil Kumar (along with Kamesh) is the developer of the MongoDB storage > plugin and is also working on the Kafka plugin. > > Welcome Anil, and thank you for your contributions. Keep up the good work > ! > > - Parth > (on behalf of the Apache Drill PMC) >
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147258230 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/ClusterFixture.java --- @@ -213,7 +204,7 @@ private void configureZk(FixtureBuilder builder) { } } - private void createConfig(FixtureBuilder builder) throws Exception { + private void createConfig() throws Exception { --- End diff -- They should be. The only change I made is that I didn't require passing the FixtureBuilder as an arg, because the class is already holding a reference to it. ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147257244 --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java --- @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.base.Charsets; +import org.apache.commons.io.FileUtils; +import org.apache.drill.common.util.TestTools; +import org.apache.drill.exec.util.TestUtilities; +import org.junit.runner.Description; + +import java.io.File; +import java.io.IOException; + +public class BaseDirTestWatcher extends DirTestWatcher { + public enum DirType { +ROOT, +TEST_TMP; + } + + private File tmpDir; + private File storeDir; + private File dfsTestTmpParentDir; + private File dfsTestTmpDir; + private File rootDir; + + public BaseDirTestWatcher() { +super(); + } + + @Override + protected void starting(Description description) { +super.starting(description); + +rootDir = makeSubDir("root"); +tmpDir = new File(rootDir, "tmp"); +storeDir = new File(rootDir, "store"); +dfsTestTmpParentDir = new File(rootDir, "dfsTestTmp"); + +tmpDir.mkdirs(); +storeDir.mkdirs(); +dfsTestTmpParentDir.mkdirs(); + } + + public File getTmpDir() { +return tmpDir; + } + + public File getStoreDir() { +return storeDir; + } + + public File getDfsTestTmpParentDir() { +return dfsTestTmpParentDir; + } + + public File getDfsTestTmpDir() { +return dfsTestTmpDir; + } + + public String getDfsTestTmpDirPath() { +return dfsTestTmpDir.getAbsolutePath().replaceAll("/\\./", "/"); + } + + public File getRootDir() { +return rootDir; + } + + public String getRootDirPath() { +return rootDir.getAbsolutePath().replaceAll("/\\./", "/"); + } + + public void newDfsTestTmpDir() { +dfsTestTmpDir = TestUtilities.createTempDir(BaseTestQuery.dirTestWatcher.getDfsTestTmpParentDir()); + } + + private File getDir(DirType type) { +switch (type) { + case ROOT: +return rootDir; + case TEST_TMP: +return dfsTestTmpDir; + default: +throw new IllegalArgumentException(String.format("Unsupported type %s", type)); +} + } + + public File makeRootSubDir(String relPath) { --- End diff -- Fair enough. I would prefer to use File objects to represent concrete files or directories. Java.nio.Path objects are useful because they can represent relative paths, and provide a bunch of methods for manipulating paths. I'll update the methods I introduced to use jva.io.Files and java.nio.Paths ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147256621 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java --- @@ -147,7 +152,7 @@ public void sortOneKeyDescendingExternalSortLegacy() throws Throwable { } private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws Throwable { -FixtureBuilder builder = ClusterFixture.builder() +FixtureBuilder builder = ClusterFixture.builder(dirTestWatcher) --- End diff -- Is there a case where the tmp directories are optional? The Drillbit will always have to be configured with a drill.tmp-dir at minimum, so I think it should be required to use a BaseDirTestWatcher with a ClusterFixture. I could move passing the dirTestWatcher to the build method instead of the constructor. Let me know what you think. ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147255211 --- Diff: exec/java-exec/src/test/java/org/apache/drill/TestAltSortQueries.java --- @@ -19,24 +19,33 @@ import org.apache.drill.categories.OperatorTest; import org.apache.drill.categories.SqlTest; +import org.apache.drill.test.BaseTestQuery; +import org.junit.BeforeClass; import org.junit.Test; import org.junit.experimental.categories.Category; @Category({SqlTest.class, OperatorTest.class}) -public class TestAltSortQueries extends BaseTestQuery{ +public class TestAltSortQueries extends BaseTestQuery { static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(TestAltSortQueries.class); + @BeforeClass + public static void setupTestFiles() { +dirTestWatcher.copyFileToRoot("sample-data/region.parquet"); +dirTestWatcher.copyFileToRoot("sample-data/regionsSF"); +dirTestWatcher.copyFileToRoot("sample-data/nation.parquet"); + } + @Test public void testOrderBy() throws Exception{ test("select R_REGIONKEY " + - "from dfs_test.`[WORKING_PATH]/../../sample-data/region.parquet` " + + "from dfs_test.`/sample-data/region.parquet` " + --- End diff -- Updated package-info.java and ExampleTest.java ---
[ANNOUNCE] New Committer: Kamesh Bhallamudi
The Project Management Committee (PMC) for Apache Drill has invited Kamesh Bhallamudi to become a committer, and we are pleased to announce that he has accepted. Kamesh (along with Anil Kumar) is the developer of the MongoDB storage plugin and is also working on the Kafka plugin. Welcome Kamesh, and thank you for your contributions. Keep up the good work ! - Parth (on behalf of the Apache Drill PMC)
[ANNOUNCE] New Committer: Anil Kumar Batchu
The Project Management Committee (PMC) for Apache Drill has invited Anil Kumar Batchu to become a committer, and we are pleased to announce that he has accepted. Anil Kumar (along with Kamesh) is the developer of the MongoDB storage plugin and is also working on the Kafka plugin. Welcome Anil, and thank you for your contributions. Keep up the good work ! - Parth (on behalf of the Apache Drill PMC)
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147229671 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java --- @@ -76,43 +78,50 @@ //- detecting corrupt values must be deferred to actual data page reading //- one from 1.4, where there is a proper created-by, but the corruption is present private static final String MIXED_CORRUPTED_AND_CORRECT_DATES_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/mixed_drill_versions"; + "/parquet/4203_corrupt_dates/mixed_drill_versions"; // partitioned with 1.2.0, no certain metadata that these were written with Drill // the value will be checked to see that they look corrupt and they will be corrected // by default. Users can use the format plugin option autoCorrectCorruptDates to disable // this behavior if they have foreign parquet files with valid rare date values that are // in the similar range as Drill's corrupt values + private static final String PARTITIONED_1_2_FOLDER = "partitioned_with_corruption_4203_1_2"; private static final String CORRUPTED_PARTITIONED_DATES_1_2_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/partitioned_with_corruption_4203_1_2"; + Paths.get("/parquet/4203_corrupt_dates", PARTITIONED_1_2_FOLDER).toString(); // partitioned with 1.4.0, no certain metadata regarding the date corruption status. // The same detection approach of the corrupt date values as for the files partitioned with 1.2.0 + private static final String PARTITIONED_1_4_FOLDER = "partitioned_with_corruption_4203"; private static final String CORRUPTED_PARTITIONED_DATES_1_4_0_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/partitioned_with_corruption_4203"; + Paths.get("/parquet/4203_corrupt_dates", PARTITIONED_1_4_FOLDER).toString(); private static final String PARQUET_DATE_FILE_WITH_NULL_FILLED_COLS = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/null_date_cols_with_corruption_4203.parquet"; + "/parquet/4203_corrupt_dates/null_date_cols_with_corruption_4203.parquet"; + private static final String PARTITIONED_1_9_FOLDER = "1_9_0_partitioned_no_corruption"; private static final String CORRECT_PARTITIONED_DATES_1_9_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/1_9_0_partitioned_no_corruption"; + Paths.get("/parquet/4203_corrupt_dates", PARTITIONED_1_9_FOLDER).toString(); private static final String VARCHAR_PARTITIONED = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/fewtypes_varcharpartition"; + "/parquet/4203_corrupt_dates/fewtypes_varcharpartition"; private static final String DATE_PARTITIONED = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/fewtypes_datepartition"; + "/parquet/4203_corrupt_dates/fewtypes_datepartition"; private static final String EXCEPTION_WHILE_PARSING_CREATED_BY_META = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/hive1dot2_fewtypes_null"; + "/parquet/4203_corrupt_dates/hive1dot2_fewtypes_null"; private static final String CORRECT_DATES_1_6_0_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/correct_dates_and_old_drill_parquet_writer.parquet"; - private static final String PARTITIONED_1_2_FOLDER = "partitioned_with_corruption_4203_1_2"; + "/parquet/4203_corrupt_dates/correct_dates_and_old_drill_parquet_writer.parquet"; private static final String MIXED_CORRUPTED_AND_CORRECT_PARTITIONED_FOLDER = "mixed_partitioned"; - @BeforeClass public static void initFs() throws Exception { +dirTestWatcher.copyResourceToRoot("/parquet/4203_corrupt_dates"); --- End diff -- fixed ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147229627 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java --- @@ -76,43 +78,50 @@ //- detecting corrupt values must be deferred to actual data page reading //- one from 1.4, where there is a proper created-by, but the corruption is present private static final String MIXED_CORRUPTED_AND_CORRECT_DATES_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/mixed_drill_versions"; + "/parquet/4203_corrupt_dates/mixed_drill_versions"; // partitioned with 1.2.0, no certain metadata that these were written with Drill // the value will be checked to see that they look corrupt and they will be corrected // by default. Users can use the format plugin option autoCorrectCorruptDates to disable // this behavior if they have foreign parquet files with valid rare date values that are // in the similar range as Drill's corrupt values + private static final String PARTITIONED_1_2_FOLDER = "partitioned_with_corruption_4203_1_2"; private static final String CORRUPTED_PARTITIONED_DATES_1_2_PATH = - "[WORKING_PATH]/src/test/resources/parquet/4203_corrupt_dates/partitioned_with_corruption_4203_1_2"; + Paths.get("/parquet/4203_corrupt_dates", PARTITIONED_1_2_FOLDER).toString(); --- End diff -- Updated all paths to be relative in the tests ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147229478 --- Diff: exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java --- @@ -33,8 +35,11 @@ @Category(UnlikelyTest.class) public class TestBugFixes extends BaseTestQuery { private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(TestBugFixes.class); - private static final String WORKING_PATH = TestTools.getWorkingPath(); - private static final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources"; + + @BeforeClass + public static void setupTestFiles() { +dirTestWatcher.copyResourceToRoot("/bugs/DRILL-4192"); --- End diff -- I have made all the path references relative in the tests now ---
[GitHub] drill pull request #984: DRILL-5783 Made a unit test for generated Priority ...
Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/984#discussion_r147229235 --- Diff: common/src/test/java/org/apache/drill/test/SubDirTestWatcher.java --- @@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.drill.test; + +import com.google.common.base.Preconditions; +import com.google.common.collect.Lists; +import org.junit.rules.TestWatcher; +import org.junit.runner.Description; + +import java.io.File; +import java.util.List; + +public class SubDirTestWatcher extends TestWatcher { --- End diff -- Done ---
Capturing in-flight batches
Hi All, Yesterday, in a conversation, Salim mentioned it would be handy to be able to capture and replay in-flight batches in a Drill query in order to diagnose problems. As it turns out, we have most of the pieces readily available; we just need someone to assemble them. First, we have the IteratorValidatorBatchIterator class which sits on top of each operator and validates that operator’s state. We extended it a while back to validate vector internals to catch a few cases of offset vector corruption. This class could be extended to capture in-flight batches for selected operators. Second, we have the VectorAccessibleSerializable class (and the recently added VectorSerializer wrapper class) that writes batches to, and reads batches from disk. This class is the foundation of our spilling support. Third, we have the EasyFormatPlugin class that lets us easily create a new disk-based reader. Combine them and we can use the validator to write batches using the vector serializer. Then, we create a new easy format plugin to read these files again using the vector serializer. The good news is that most of these classes have been around since the early days, so any technique built using them should work for any older versions of Drill we need to debug. (Though, of course, we’d have to rebuild that old version to include the batch intercept code…) Thanks, - Paul
[GitHub] drill pull request #1011: Drill 1170: Drill-on-YARN
GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/1011 Drill 1170: Drill-on-YARN Provides Drill integration with YARN. Runs Drill as a long-running task under YARN. Monitors the Drill cluster, restarting failed Drillbits. Provides a command-line UI to start, stop and resize the cluster. Provides a web-based UI to monitor the cluster. The Drill-on-YARN (DoY) code has been in use by commercial users for over a year, since Drill 1.8 and has proven quite stable. Usage has been on MapR's version of YARN, we seek feedback from users of the Apache and other versions of YARN. See [DRILL-1170](https://issues.apache.org/jira/browse/DRILL-1170) for design information. See the included `README.md`` for internals information and `USAGE.md` for a detailed user guide. This is a large PR; it will take time to review. The key goal at this moment is to allow interested users to download the PR, build DoY, and try it out in their environments. The DoY code is mostly independent of Drill itself. The DoY code can be used to launch any version of Drill since 1.8. See the usage guide for information. It has been suggested that the code move to the `contrib` directory. That change will be made. But, since the code works successfully in its current location; we'll leave it their for now to ensure users are successful if they choose to try it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-1170 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1011.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1011 commit e26012bc56cad3bf2819dff3bbdf70664de34955 Author: Paul RogersDate: 2017-10-26T07:24:00Z DRILL-1170: YARN integration for Drill This commit includes documentation files. commit 3a6ffe78d9fe0e9a5beacd100e2e0ee6b40c7f34 Author: Paul Rogers Date: 2017-10-26T07:25:34Z Client app commit 509410c9710fc1ff23a4a13f51320a4c153f9328 Author: Paul Rogers Date: 2017-10-26T07:26:50Z Files common to several modules commit 36b8d323118077115ef1097ba8b23f6fc4a5390a Author: Paul Rogers Date: 2017-10-26T07:42:17Z Application master commit 7fbc387634bb1acaf7807b02348b40364c81d282 Author: Paul Rogers Date: 2017-10-26T07:44:33Z App Master web UI commit 21fb93792290625b719899a4742573b8c3d4a7ce Author: Paul Rogers Date: 2017-10-26T07:45:44Z Distribution and project files commit 567d36787b9ada60dd2141077e629158c53fc0c4 Author: Paul Rogers Date: 2017-10-26T07:47:54Z Test files ---
[jira] [Created] (DRILL-5910) ClassNotFoundException message enhancement
Volodymyr Tkach created DRILL-5910: -- Summary: ClassNotFoundException message enhancement Key: DRILL-5910 URL: https://issues.apache.org/jira/browse/DRILL-5910 Project: Apache Drill Issue Type: Bug Affects Versions: 1.11.0 Reporter: Volodymyr Tkach Assignee: Volodymyr Tkach -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] drill issue #904: DRILL-5717: change some date time test cases with specific...
Github user weijietong commented on the issue: https://github.com/apache/drill/pull/904 @vvysotskyi thanks for your hard work, I have rewritten the corresponding method. But I have not rewritten the other Locate related cases. Since mockup is a white box testing , I payed some tedious work to explore the possible invocation path, maybe the same as you. I think current Local solution is also acceptable. ---
[GitHub] drill issue #1010: DRILL-5906: java.lang.NullPointerException while quering ...
Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1010 +1, LGTM. ---
[GitHub] drill issue #998: DRILL-5887: Display process user/groups info in Drill UI
Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/998 +1, LGTM. ---