[jira] [Commented] (DRILL-6399) Use RowSets In MiniPlanUnitTestBase To Generate Test Data
[ https://issues.apache.org/jira/browse/DRILL-6399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469829#comment-16469829 ] Paul Rogers commented on DRILL-6399: Also, consider using them to validate output. Create an expected schema, use that to generate an expected result set, then verify actual output. This approach works for very close, detailed examination of columns, values and types. Clearly, it does not work quite as well for tests that require millions of rows, but such tests are integration tests, not unit tests... > Use RowSets In MiniPlanUnitTestBase To Generate Test Data > - > > Key: DRILL-6399 > URL: https://issues.apache.org/jira/browse/DRILL-6399 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline
[ https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469788#comment-16469788 ] ASF GitHub Bot commented on DRILL-6364: --- sohami commented on a change in pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart URL: https://github.com/apache/drill/pull/1241#discussion_r187217229 ## File path: exec/java-exec/src/main/resources/rest/index.ftl ## @@ -217,21 +217,34 @@
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469742#comment-16469742 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187210825 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). + +## Creating An Instance of [QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java) + +``` +UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder() + .setPart1(1L) + .setPart2(2L) + .build(); +``` + +## Creating [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) + +``` +ExecProtos.FragmentHandle fragmentHandle = ExecProtos.FragmentHandle.newBuilder() + .setQueryId(queryId) + .setMinorFragmentId(1) + .setMajorFragmentId(2) + .build(); +``` + +## Creating A [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) + +There are a few ways to create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). See the class for all of them. The simplest way is to + create a default config. + +``` +DrillConfig config = DrillConfig.create(); Review comment: Added a reference to ConfigBuilder. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469738#comment-16469738 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187210241 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). Review comment: The OperatorFixture uses the MockFragmentContext. I added this snippet to show how to retrieve the FragmentContext from the OperatorFixture. ``` FragmentContext context = operatorFixture.getFragmentContext(); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469737#comment-16469737 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187210052 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). Review comment: I just removed this detail since we are saying these libraries are deprecated and we no longer want to use them. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469735#comment-16469735 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187209902 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469734#comment-16469734 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187209858 ## File path: docs/dev/DevDocs.md ## @@ -6,4 +6,8 @@ For information about configuring your development enviornment see [Environment. ## Testing -For information about how to do Integration and Unit Testing in Drill see [Testing.md](Testing.md). \ No newline at end of file +For information about how to do Integration and Unit Testing in Drill see [Testing.md](Testing.md). Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469719#comment-16469719
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187209131
##
File path: common/src/test/java/org/apache/drill/test/DirTestWatcher.java
##
@@ -34,7 +34,7 @@
* A {@link DirTestWatcher} is added to a test by declaring it as a JUnit
{@link org.junit.Rule}. A {@link org.junit.Rule} is
* a piece of code that is run before and after every JUnit test marked with
the {@link org.junit.Test} annotation. When the
* {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher}
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test}s and optionally delete the temp directory after each
of your {@link org.junit.Test}s. The base temp directory
Review comment:
It doesn't show the base name. You are right, you have to use `{@link
org.junit.Test Test}`. Updated to do so.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469712#comment-16469712 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187208621 ## File path: README.md ## @@ -20,7 +20,7 @@ Please see the [Apache Drill Website](http://drill.apache.org/) or the [Apache D ## Join the community! -Apache Drill is an Apache Foundation project and is seeking all types of contributions. Please say hello on the Apache Drill mailing list or join our Google Hangouts for more information. (More information can be found at the Apache Drill website). +Apache Drill is an Apache Foundation project and is seeking all types of contributions. Please say hello on the Apache Drill mailing list or join our Google Hangouts for more information. (More information can be found at the Apache Drill website). Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469707#comment-16469707
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207855
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/file/JsonFileBuilder.java
##
@@ -35,6 +36,16 @@
import java.util.List;
import java.util.Map;
+/**
+ * Overview
+ *
+ * This class is used to build a json file containing the data in a {@link
RowSet}.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469704#comment-16469704 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187207643 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/ops/RootFragmentContext.java ## @@ -17,6 +17,9 @@ */ package org.apache.drill.exec.ops; +/** + * This interface is implemented by fragment roots. Review comment: Updated This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469706#comment-16469706
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207707
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java
##
@@ -27,17 +27,31 @@
import java.nio.file.Paths;
/**
+ * Overview
*
* This is a {@link DirTestWatcher} which creates all the temporary
directories required by a Drillbit and the various dfs.* storage
workspaces. It also
* provides convenience methods that do the following:
- *
*
*
* Copy project files to temp directories. This is useful for copying
the sample data into a temp directory.
* Copy resource files to temp.
* Updating parquet metadata files.
*
+ *
+ *
+ *
+ * When used the {@link BaseDirTestWatcher} creates the following
directories in the base temp directory (for a description of where the
base temp directory
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469703#comment-16469703
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207501
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -70,9 +69,26 @@
import io.netty.buffer.DrillBuf;
/**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContext
- * in code to allow tests to use test-time implementations.
+ *
+ * This is the core Context which implements all the Context interfaces:
+ *
+ *
+ * {@link FragmentContext}: A context provided to non-exchange
operators.
+ * {@link ExchangeFragmentContext}: A context provided to exchange
operators.
+ * {@link RootFragmentContext}: A context provided to fragment
roots.
+ * {@link ExecutorFragmentContext}: A context used by the
Drillbit.
+ *
+ *
+ * The interfaces above expose resources to varying degrees. They are
ordered from most restrictive ({@link FragmentContext})
+ * to least restrictive ({@link ExecutorFragmentContext}).
+ *
+ *
+ * Since {@link FragmentContextImpl} implements all of the interfaces listed
above, the facade pattern is used in order
+ * to cast a {@link FragmentContextImpl} object to the desired interface
where-ever it is needed. The facade pattern
+ * is powerful since it allows us to easily create minimal mock context
objects to be used in unit tests. Without
+ * the use of interfaces and the facade pattern we would have to create a
complete {@link FragmentContextImpl} object
+ * to unit test any part of the code that depends on a context.
+ *
Review comment:
Added this to javadoc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469701#comment-16469701
]
ASF GitHub Bot commented on DRILL-6321:
---
vrozov commented on a change in pull request #1224: DRILL-6321: Customize
Drill's conformance. Allow support to APPLY key…
URL: https://github.com/apache/drill/pull/1224#discussion_r187207266
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConformance.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlDelegatingConformance;
+
+/**
+ * Drill's SQL conformance is SqlConformanceEnum.DEFAULT except for method
isApplyAllowed().
+ * Since Drill is going to allow OUTER APPLY and CROSS APPLY to allow each row
from left child of Join
+ * to join with output of right side (sub-query or table function that will be
invoked for each row).
+ * Refer to DRILL-5999 for more information.
+ */
+public class DrillConformance extends SqlDelegatingConformance {
Review comment:
Parth, I don't see what a standalone class provides that an anonymous class
does not. I would expect a standalone class in case it is used in multiple
places and/or adds methods to an existing interface/class. `SqlConformance` is
created/used in a single place and it is not necessary to jump to a different
place/file to see the definition of Drill SQL conformance and how it is
different from the Calcite default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469700#comment-16469700
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187207066
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
##
@@ -38,16 +38,8 @@
import io.netty.buffer.DrillBuf;
/**
- * Fragment context interface: separates implementation from definition.
- * Allows unit testing by mocking or reimplementing services with
- * test-time versions. The name is awkward, chosen to avoid renaming
- * the implementation class which is used in many places in legacy code.
- * New code should use this interface, and the names should eventually
- * be swapped with {@link FragmentContextImpl} becoming
- * FragmentContextImpl and this interface becoming
- * {@link FragmentContextImpl}.
+ * This interface provides the resources required by a non-exchange operator
to execute.
Review comment:
fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469699#comment-16469699
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206328
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount`
option. By default this
+
+Ex. Run 4 test processes in parallel
+
+```
+mvn clean install -DforkCount=4
+```
+
+**Note:** The `-DforkCount` option interacts with `-T`. When use together each
build thread (`-T`) gets
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>
[jira] [Created] (DRILL-6400) Hash-Aggr: Avoid recreating common Hash-Table setups for every partition
Boaz Ben-Zvi created DRILL-6400: --- Summary: Hash-Aggr: Avoid recreating common Hash-Table setups for every partition Key: DRILL-6400 URL: https://issues.apache.org/jira/browse/DRILL-6400 Project: Apache Drill Issue Type: Improvement Components: Execution - Relational Operators Affects Versions: 1.13.0 Reporter: Boaz Ben-Zvi Assignee: Boaz Ben-Zvi Fix For: 1.14.0 The current Hash-Aggr code (and soon the Hash-Join code) creates multiple partitions to hold the incoming data; each partition with its own HashTable. The current code invokes the HashTable method _createAndSetupHashTable()_ for *each* partition. But most of the setups done by this method are identical for all the partitions (e.g., code generation). Calling this method has a performance cost (some local tests measured between 3 - 30 milliseconds, depends on the key columns). Suggested performance improvement: Extract the common settings to be called *once*, and use the results later by all the partitions. When running with the default 32 partitions, this can have a measurable improvement (and if spilling, this method is used again). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469695#comment-16469695
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206176
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469697#comment-16469697
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206270
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount`
option. By default this
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469694#comment-16469694
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187206049
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469693#comment-16469693
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187205951
##
File path: docs/dev/TestDataSets.md
##
@@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+ - **Sample Data:** These are parquet files in the
[sample-data](../sample-data) folder.
+ - **Resource Data:** These are data files in the
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+ - **TPCH Data:** These are trimmed down versions of the tpch data sets. They
are retrieved and bundled
+ in the [contrib/data](../contrib/data) maven submodule. They are also
accessible on [Apache Drill's S3
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+ When unit tests are running all of the files in these data set are available
from the classpath storage plugin. The tpch
+ files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+
+### Using Sample Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage
plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+### Using Resource Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make data from
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources)
accessible from the ```dfs``` storage plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at
dfs.root.`join/empty_part` from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at
dfs.root.`join/empty_part` from my test
+ }
+}
+```
+
+### Using TPCH Data in Unit Tests
+
+TPCH data is accessible via the classpath storage plugin
+
+```
+cp.`tpch/customer.parquet`
+```
+
+## Generating Data Sets
Review comment:
I created a separate Jira to do this
https://issues.apache.org/jira/browse/DRILL-6399
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels:
[jira] [Created] (DRILL-6399) Use RowSets In MiniPlanUnitTestBase To Generate Test Data
Timothy Farkas created DRILL-6399: - Summary: Use RowSets In MiniPlanUnitTestBase To Generate Test Data Key: DRILL-6399 URL: https://issues.apache.org/jira/browse/DRILL-6399 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469689#comment-16469689 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187205150 ## File path: docs/dev/TestDataSets.md ## @@ -0,0 +1,146 @@ +# Data Sets + +Drill includes several data sets for testing, and also provides some tools for generating test data sets. + +## Bundled Data Sets + +There are three primary data sets bundled with drill for testing: + + - **Sample Data:** These are parquet files in the [sample-data](../sample-data) folder. + - **Resource Data:** These are data files in the [exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold. Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469688#comment-16469688
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187205087
##
File path: docs/dev/TestDataSets.md
##
@@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+ - **Sample Data:** These are parquet files in the
[sample-data](../sample-data) folder.
+ - **Resource Data:** These are data files in the
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+ - **TPCH Data:** These are trimmed down versions of the tpch data sets. They
are retrieved and bundled
+ in the [contrib/data](../contrib/data) maven submodule. They are also
accessible on [Apache Drill's S3
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+ When unit tests are running all of the files in these data set are available
from the classpath storage plugin. The tpch
+ files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+
+### Using Sample Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage
plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
Review comment:
Added examples for this in ExampleTest and referenced them in this doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469643#comment-16469643 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187199672 ## File path: docs/dev/TempDirectories.md ## @@ -0,0 +1,17 @@ +# Temp Directory Utilities + +The two basic temp directory classes are: + + - [DirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/DirTestWatcher.java) + - [BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java) + +These classes are used to create temp directories for each of your unit tests. The advantage to using +these temp directory classes are: + + - All files are deleted after a unit test completes. This prevents a build machine being polluted with a + bunch of unit test files. + - Each unit test outputs its files to a unique well defined location. This makes it easy to find files + for debugging. Also since each temp directory is unique, it prevents multiple unit test runs from interferring + with one another on a build machine. + +For examples on how to use these classes, please read the javadoc for each class. Review comment: Thanks for the info. I will add some examples to ExampleTest and will document this in TestDataSets.md . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469595#comment-16469595
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187192496
##
File path: docs/dev/RowSetFramework.md
##
@@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ *
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ *
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ *
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+
+It also allows the comparison of data container in
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
through the use
+of the
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
and
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
class can be used
+to create an instance
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
An example
+of how to to this can be found the **secondTest()** method of
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
populated with data can be created with the
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
and add
+records to it. Then build a
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+RowSet rowSet = new RowSetBuilder(allocator, schema)
+ .addRow(110, "green", new float[]{5.5f, 2.3f}, new String[]{"1a", "1b"})
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469547#comment-16469547 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187183549 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework + +The RowSet Framework allows you to create custom instances of: + + * [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + * [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + * [RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java) Review comment: Added documentation for TupleMetadata. Highlighted the two other points in the doc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6395) Value Window Function - LEAD and LAG on VarChar result in "No applicable constructor/method found" error
[
https://issues.apache.org/jira/browse/DRILL-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6395:
--
Labels: window_function (was: )
> Value Window Function - LEAD and LAG on VarChar result in "No applicable
> constructor/method found" error
> -
>
> Key: DRILL-6395
> URL: https://issues.apache.org/jira/browse/DRILL-6395
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
> Environment: windows 10, apache drill 1.13.0, 32GB Ram
>Reporter: Raymond Wong
>Priority: Major
> Labels: window_function
>
> {code:java}
> SELECT
> col2,
> LEAD(col1, 1) OVER (ORDER BY col2) AS nxtCol1
> FROM (
> SELECT 'A' AS col1, 1 AS col2
> UNION
> SELECT 'B' AS col1, 2 AS col2
> UNION
> SELECT 'C' AS col1, 3 AS col2
> ) AS A;
> {code}
> Causes error
> {code:java}
> SQL Error: SYSTEM ERROR: CompileException: Line 37, Column 40:
> No applicable constructor/method found for actual parameters "int, int, int,
> io.netty.buffer.DrillBuf";
> candidates are:
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> org.apache.drill.exec.expr.holders.VarCharHolder)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> org.apache.drill.exec.expr.holders.NullableVarCharHolder)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> byte[], int, int)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> java.nio.ByteBuffer, int, int)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
> int, int, io.netty.buffer.DrillBuf)"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6395) Value Window Function - LEAD and LAG on VarChar result in "No applicable constructor/method found" error
[
https://issues.apache.org/jira/browse/DRILL-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469541#comment-16469541
]
Khurram Faraaz commented on DRILL-6395:
---
Here is the stack trace from drillbit.log,
seen on Drill 1.14.0-SNAPSHOT , commit :
c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-09 14:21:57,085 [250c998a-5917-1cc5-9364-4d7cf441f209:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
250c998a-5917-1cc5-9364-4d7cf441f209: SELECT
col2,
LEAD(col1, 1) OVER (ORDER BY col2) AS nxtCol1
FROM (
SELECT 'A' AS col1, 1 AS col2
UNION
SELECT 'B' AS col1, 2 AS col2
UNION
SELECT 'C' AS col1, 3 AS col2
) AS A
...
2018-05-09 14:21:57,600 [250c998a-5917-1cc5-9364-4d7cf441f209:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: Line 37,
Column 40: No applicable constructor/method found for actual parameters "int,
int, int, io.netty.buffer.DrillBuf"; candidates are: "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, byte[],
int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
java.nio.ByteBuffer, int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
int, int, io.netty.buffer.DrillBuf)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.NullableVarCharHolder)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.VarCharHolder)"
Fragment 0:0
[Error Id: 71fc40fc-0b52-4c9b-b3ae-1ddad76ea120 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
CompileException: Line 37, Column 40: No applicable constructor/method found
for actual parameters "int, int, int, io.netty.buffer.DrillBuf"; candidates
are: "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, byte[],
int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
java.nio.ByteBuffer, int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
int, int, io.netty.buffer.DrillBuf)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.NullableVarCharHolder)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.VarCharHolder)"
Fragment 0:0
[Error Id: 71fc40fc-0b52-4c9b-b3ae-1ddad76ea120 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
org.apache.drill.exec.exception.SchemaChangeException: Exception when creating
the schema
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:167)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:118)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:108)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:118)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline
[ https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469534#comment-16469534 ] ASF GitHub Bot commented on DRILL-6364: --- kkhatua commented on a change in pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart URL: https://github.com/apache/drill/pull/1241#discussion_r187181236 ## File path: exec/java-exec/src/main/resources/rest/index.ftl ## @@ -217,21 +217,33 @@
[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline
[ https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469532#comment-16469532 ] ASF GitHub Bot commented on DRILL-6364: --- kkhatua commented on issue #1241: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart URL: https://github.com/apache/drill/pull/1241#issuecomment-387880124 Yes. Cross-Origins filter in [WebServer.java](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java#L166) allows that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI does not cleanly handle shutdown and state toggling when Drillbits go > on and offline > -- > > Key: DRILL-6364 > URL: https://issues.apache.org/jira/browse/DRILL-6364 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > When the webpage is loaded the first time, the shutdown button is enabled by > default, which might not be correct, since scenarios like HTTPS, etc does not > support this for remote bits. (i.e the user needs to navigate to that node's > UI for shutting it down). > Similarly, when a previously unseen Drillbit comes online, the node will not > be rendered until the page is refreshed by the user. > Lastly, if the node from whom the UI page was served goes down, the status > update for the rest of the cluster is not updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline
[ https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469531#comment-16469531 ] ASF GitHub Bot commented on DRILL-6364: --- sohami commented on issue #1241: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart URL: https://github.com/apache/drill/pull/1241#issuecomment-387879455 1 minor comment related to unused function. Also how come metrics are available from remote Drillbits when auth is enabled without https ? Is Cross-Origin filter helping in that ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI does not cleanly handle shutdown and state toggling when Drillbits go > on and offline > -- > > Key: DRILL-6364 > URL: https://issues.apache.org/jira/browse/DRILL-6364 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > When the webpage is loaded the first time, the shutdown button is enabled by > default, which might not be correct, since scenarios like HTTPS, etc does not > support this for remote bits. (i.e the user needs to navigate to that node's > UI for shutting it down). > Similarly, when a previously unseen Drillbit comes online, the node will not > be rendered until the page is refreshed by the user. > Lastly, if the node from whom the UI page was served goes down, the status > update for the rest of the cluster is not updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline
[ https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469527#comment-16469527 ] ASF GitHub Bot commented on DRILL-6364: --- sohami commented on a change in pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart URL: https://github.com/apache/drill/pull/1241#discussion_r187175842 ## File path: exec/java-exec/src/main/resources/rest/index.ftl ## @@ -217,21 +217,33 @@
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469526#comment-16469526
]
ASF GitHub Bot commented on DRILL-6249:
---
ilooner commented on a change in pull request #1251: DRILL-6249: Adding more
unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r187179092
##
File path: docs/dev/PhysicalOpUnitTestBase.md
##
@@ -0,0 +1,31 @@
+# Single Operator Unit Test
+
+It is possible to run an end to end test of an operator in isolation by
extending
+[PhysicalOpUnitTestBase](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/PhysicalOpUnitTestBase.java).
+
+A simple example of an operator level unit test is the following:
+
+```
+public class BasicPhysicalOpUnitTest extends PhysicalOpUnitTestBase {
+
+ @Test
+ public void testSimpleProject() {
+ Project projectConf = new Project(parseExprs("x+5", "x"), null);
+ List jsonBatches = Lists.newArrayList(
+ "[{\"x\": 5 },{\"x\": 10 }]",
+ "[{\"x\": 20 },{\"x\": 30 },{\"x\": 40 }]");
+ opTestBuilder()
+ .physicalOperator(projectConf)
+ .inputDataStreamJson(jsonBatches)
+ .baselineColumns("x")
+ .baselineValues(10l)
+ .baselineValues(15l)
+ .baselineValues(25l)
+ .baselineValues(35l)
+ .baselineValues(45l)
+ .go();
Review comment:
I created a separate jira https://issues.apache.org/jira/browse/DRILL-6397.
PhysicalOpUnitTestBase is already using OperatorFixture underneath the hood, so
we would just have to update OperatorTestBuilder to use RowSets for baselines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6397) OperatorTestBuilder, should leverage RowSets for comparing baseline values.
Timothy Farkas created DRILL-6397: - Summary: OperatorTestBuilder, should leverage RowSets for comparing baseline values. Key: DRILL-6397 URL: https://issues.apache.org/jira/browse/DRILL-6397 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6397) OperatorTestBuilder, should leverage RowSets for comparing baseline values.
[ https://issues.apache.org/jira/browse/DRILL-6397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas reassigned DRILL-6397: - Assignee: Timothy Farkas > OperatorTestBuilder, should leverage RowSets for comparing baseline values. > --- > > Key: DRILL-6397 > URL: https://issues.apache.org/jira/browse/DRILL-6397 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469517#comment-16469517 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187178022 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). + +## Creating An Instance of [QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java) + +``` +UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder() + .setPart1(1L) + .setPart2(2L) + .build(); +``` + +## Creating [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) + +``` +ExecProtos.FragmentHandle fragmentHandle = ExecProtos.FragmentHandle.newBuilder() + .setQueryId(queryId) + .setMinorFragmentId(1) + .setMajorFragmentId(2) + .build(); +``` + +## Creating A [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) + +There are a few ways to create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). See the class for all of them. The simplest way is to + create a default config. + +``` +DrillConfig config = DrillConfig.create(); +``` + +## Creating A [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java) + + 1. Create a [PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java). +``` +HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL); +``` + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). + 1. Create a [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) as described above. + 1. Create a [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java). +``` +SpillSet spillSet = new SpillSet(config, fragmentHandle, pop); +``` + +## Creating A [PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java) + +``` +LocalPersistentStoreProvider provider = new LocalPersistentStoreProvider(drillConfig); +provider.start(); +``` + +## Creating A [LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java) + +``` +LogicalPlanPersistence logicalPlanPersistence = PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig); +``` + +## Creating An Instance Of An Option Manager Review comment: Agreed. Updated doc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469515#comment-16469515 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187178022 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). + +## Creating An Instance of [QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java) + +``` +UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder() + .setPart1(1L) + .setPart2(2L) + .build(); +``` + +## Creating [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) + +``` +ExecProtos.FragmentHandle fragmentHandle = ExecProtos.FragmentHandle.newBuilder() + .setQueryId(queryId) + .setMinorFragmentId(1) + .setMajorFragmentId(2) + .build(); +``` + +## Creating A [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) + +There are a few ways to create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). See the class for all of them. The simplest way is to + create a default config. + +``` +DrillConfig config = DrillConfig.create(); +``` + +## Creating A [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java) + + 1. Create a [PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java). +``` +HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL); +``` + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). + 1. Create a [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) as described above. + 1. Create a [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java). +``` +SpillSet spillSet = new SpillSet(config, fragmentHandle, pop); +``` + +## Creating A [PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java) + +``` +LocalPersistentStoreProvider provider = new LocalPersistentStoreProvider(drillConfig); +provider.start(); +``` + +## Creating A [LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java) + +``` +LogicalPlanPersistence logicalPlanPersistence = PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig); +``` + +## Creating An Instance Of An Option Manager Review comment: Agreed. Updated dock. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469487#comment-16469487 ] ASF GitHub Bot commented on DRILL-5270: --- ilooner commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-387873929 @kkhatua I think I understand the difference in our two perspectives. You wanted a cache that will always only contain the **N** most recently created profiles. If you happen to access the **N + 1**th youngest profile, the cache will not contain it and will never contain it, the cache will only hold the **N** most recently created profiles. I still prefer the approach with the Guava cache because you can still effectively achieve the same result. As new profiles are created they can be added to the cache. If you access a very old profile, one more recently created profile will be evicted from the cache and the old profile will be added to the cache since a user just requested it. I would argue this behavior is not only easier to implement since we are leveraging a library, but actually more desirable since it caches a profile based on when it is used, not when it was created. If you still disagree with using the Guava cache. I agree with your proposal of moving your cache into a separate class. I think you should also add some unit tests for the cache to verify that it works as expected. The unit tests will also make maintaining and enhancing the class easier for future developers. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469431#comment-16469431 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187165138 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors Review comment: Agreed. Modified the description so this is clearly marked as a deprecated method, and explained why it is deprecated and should not be used. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469429#comment-16469429 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187164790 ## File path: docs/dev/LicenseHeaders.md ## @@ -0,0 +1,34 @@ +# License Headers + +Drill uses two license header checkers: + +* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) +* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/) + +## Why Two? + +[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used because it is the standard license header +checker for Apache projects. + +[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs stricter license checks and disallows +putting license headers in java docs. + +## Doing License Checks + +The license checks are disabled locally by default and are enabled on Travis. If you'd like to perform +license checks locally you can do the following: + +``` + mvn license:check -Dlicense.skip=false +``` + +## Auto Formatting Headers + +If the license checks fail and you can't figure out what's wrong with your headers, you can auto-format +your license headers with the following command: + +``` +mvn license:format -Dlicense.skip=false +``` + +This command will also add license headers to files without them. Review comment: I've already tested this. Running license:format is safe and will only impact new files that you are adding in your commit, all pre-existing files in the project are correctly formatted and will not be impacted. I've also already added all the appropriate exclusions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469427#comment-16469427 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187164186 ## File path: docs/dev/LicenseHeaders.md ## @@ -0,0 +1,34 @@ +# License Headers + +Drill uses two license header checkers: + +* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) +* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/) + +## Why Two? + +[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used because it is the standard license header +checker for Apache projects. + +[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs stricter license checks and disallows +putting license headers in java docs. Review comment: Added description. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469418#comment-16469418 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187163429 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). + 1. Create a [SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java) as shown in + [MockingComponents.md](MockingComponents.md). + 1. Create an instance of [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java). +``` +new CodeCompiler(drillConfig, optionManager) +``` + +## Debugging Generated Code + +It is possible to set break points in generated code. + +### Instructions For IntelliJ + + 1. File→Project structure…→Modules→distribution→Sources → Add content root + 1. Chose /tmp/drill/codegen + 1. Mark it as Sources directory. + 1. Set saveCodeForDebugging(true) for the code generator of interest + 1. Run the unit test of interest + 1. Now some generated classes should appear in Intellij under the distribution module + 1. Set a break point in a generated class and run the unit test in debug mode Review comment: Added your instructions for eclipse. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469404#comment-16469404 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187162074 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). + 1. Create a [SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java) as shown in + [MockingComponents.md](MockingComponents.md). Review comment: I agree that mocking libraries are not a good way to test. I meant this to be a cheat sheet about how to instantiate classes, not a description of how to use a mocking library. I have renamed it to InstantiatingComponents.md to avoid this confusion. I am also updating the document with references to OperatorFixture and ClusterFixture where appropriate. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469402#comment-16469402 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187161268 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). + 1. Create a [SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java) as shown in Review comment: Updated to reference OperatorFixture. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469397#comment-16469397 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187160309 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). Review comment: Updated to reference ClusterFixture and Operator Fixture. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6396) Remove unused getTempDir Method in BaseFixture
Timothy Farkas created DRILL-6396: - Summary: Remove unused getTempDir Method in BaseFixture Key: DRILL-6396 URL: https://issues.apache.org/jira/browse/DRILL-6396 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas This tempDirectory method is no longer used. The DirTestWatcher and BaseDirTestWatcher classes are used instead for testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469373#comment-16469373 ] ASF GitHub Bot commented on DRILL-6249: --- ilooner commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r187153599 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: Review comment: Fixed. Thanks for catching. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469297#comment-16469297
]
ASF GitHub Bot commented on DRILL-6321:
---
parthchandra commented on a change in pull request #1224: DRILL-6321: Customize
Drill's conformance. Allow support to APPLY key…
URL: https://github.com/apache/drill/pull/1224#discussion_r187141418
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConformance.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlDelegatingConformance;
+
+/**
+ * Drill's SQL conformance is SqlConformanceEnum.DEFAULT except for method
isApplyAllowed().
+ * Since Drill is going to allow OUTER APPLY and CROSS APPLY to allow each row
from left child of Join
+ * to join with output of right side (sub-query or table function that will be
invoked for each row).
+ * Refer to DRILL-5999 for more information.
+ */
+public class DrillConformance extends SqlDelegatingConformance {
Review comment:
Vlad, you make a good point. From what I understand of Chunhui's
explanation, SqlConformance is a class that logically stands on its own. That
also mirrors what Calcite does. So I would recommend we let this stand.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469293#comment-16469293 ] ASF GitHub Bot commented on DRILL-5270: --- kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-387839266 The way the cache is constructed is by first listing all the profile files and sorting them (the profile ID is generated in a monotonically decreasing value to ensure sortedness in stores like HBase), This customized TreeSet is used to inject profiles (since the FileSystem is not guaranteed to return the list in order), so the TreeSet provides the ordering. We retain only the first N (which are, implicitly, the latest profiles). If we were to add more profiles than the max capacity, the TreeSet is pruned at the rightmost end. With Guava, the eviction policy provides the option of limiting the size, but the basis on which it would evict a profile would not work with the least-recently used/accessed profile. Also, this is currently not a true cache, because the moment we detect changes in the underlying store, we reconstruct this 'cache'. Ideally, we'd want to identify the newest profiles returned from the FileSystem (using filename filters), but the Hadoop API performance is the same (irrespective of the filter). We, primarily, save the time in fetching file list from the FS and in deserializing. I can move the implementation of the TreeSet to a separate class to clean up the code. That would make debugging simpler too. With Guava, I don't see the value add beyond a lower risk of bugs, which should be minimal with the TreeSet too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469232#comment-16469232 ] ASF GitHub Bot commented on DRILL-6386: --- ilooner commented on a change in pull request #1252: DRILL-6386: Disallowed unused imports and removed them. URL: https://github.com/apache/drill/pull/1252#discussion_r187129341 ## File path: src/main/resources/checkstyle-config.xml ## @@ -27,7 +27,11 @@ - + + + Review comment: Thanks for catching. Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469181#comment-16469181
]
Paul Rogers commented on DRILL-6390:
I'll work up a concrete example later today. For now, here is the idea. Suppose
you have the following JSON:
{noformat}
{ a: {b: 10}}
{noformat}
Then, mention b, say by casting:
{noformat}
SELECT CAST('a'.'b' AS DOUBLE) FROM ...
{noformat}
The result of this is that what was {{a.b}} is now projected to the top-level
column {{EXPR$1}}. As far as I know, one cannot mention a column (for casting
or whatever) and place it back into the map.
I'll update this later with a worked example.
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {code:java}
> SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {code}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {code:java}
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {code}
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6395) Value Window Function - LEAD and LAG on VarChar result in "No applicable constructor/method found" error
Raymond Wong created DRILL-6395:
---
Summary: Value Window Function - LEAD and LAG on VarChar result in
"No applicable constructor/method found" error
Key: DRILL-6395
URL: https://issues.apache.org/jira/browse/DRILL-6395
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.13.0
Environment: windows 10, apache drill 1.13.0, 32GB Ram
Reporter: Raymond Wong
{code:java}
SELECT
col2,
LEAD(col1, 1) OVER (ORDER BY col2) AS nxtCol1
FROM (
SELECT 'A' AS col1, 1 AS col2
UNION
SELECT 'B' AS col1, 2 AS col2
UNION
SELECT 'C' AS col1, 3 AS col2
) AS A;
{code}
Causes error
{code:java}
SQL Error: SYSTEM ERROR: CompileException: Line 37, Column 40:
No applicable constructor/method found for actual parameters "int, int, int,
io.netty.buffer.DrillBuf";
candidates are:
"public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.VarCharHolder)",
"public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.NullableVarCharHolder)",
"public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, byte[],
int, int)",
"public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
java.nio.ByteBuffer, int, int)",
"public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
int, int, io.netty.buffer.DrillBuf)"
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469115#comment-16469115
]
Trung Ta commented on DRILL-6390:
-
Hi Paul,
thank you for your comments and support. I've already enabled
json.all_text_mode as default in order to make my (other) queries to work. That
doesn't help in this case.
Could you please elaborate how to re-project a column to the top-level? That
would require me to provide Drill another dataset to work with, wouldn't it?
Regards!
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {code:java}
> SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {code}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {code:java}
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {code}
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6390) UnsupportedOperationException: Unable to get holder type for minor type [LATE] and mode [OPTIONAL]
[
https://issues.apache.org/jira/browse/DRILL-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469088#comment-16469088
]
Paul Rogers commented on DRILL-6390:
Another thing to try, try enabling "all text mode." This works for top-level
nulls (you get null VarChar). It does force all your fields to VarChar, which
you can then undo (for top level fields only) via a cast.
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL]
> --
>
> Key: DRILL-6390
> URL: https://issues.apache.org/jira/browse/DRILL-6390
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Trung Ta
>Priority: Major
>
> I tried this query on some of our JSON files that are stored in a S3 Bucket,
> in which I tried to convert JSON objects (application, application_fee) into
> varchar:
> {code:java}
> SELECT
> CONCAT(
> '"', CONVERT_TO(_stg.data.application, 'JSON'), '"'
> , ',', '"', CONVERT_TO(_stg.data.application_fee, 'JSON'), '"'
> ) as JSONs
> FROM (
> SELECT
> flatten(js.data) AS data
> FROM
> s3_bucket.`` AS js
> ) _stg
> {code}
> On some of the datasets the query failed and I got this error message
> {quote}org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnsupportedOperationException: Unable to get holder type for minor type
> [LATE] and mode [OPTIONAL] Fragment 0:0 [Error Id:
> 5e74e125-de22-46fa-8d31-233d0fc40140
> {quote}
> The dataset, on which the query failed looks like following:
> {code:java}
> {
> "data":
> [
> {
> "application": null,
> "application_fee": null,
> "amount": 1
> },
> {
> "application":
> {
> "id": "some_id",
> "name": "Some App",
> "object": "application"
> },
> "application_fee": null,
> "amount": 5
> }
> ],
> "has_more": false,
> "object": "list",
> "url": "/v1/charges"
> }
> {code}
> I ran some further tests on the query and found out that the query only fails
> at object "application_fee", which is null in both arrays of object "data".
> Which might explain why I never ran into this error in the other datasets (no
> object arrays there are null).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469081#comment-16469081
]
ASF GitHub Bot commented on DRILL-6348:
---
sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code
so that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r187105067
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
##
@@ -116,4 +120,16 @@ public void close() throws Exception {
AutoCloseables.close(buffers);
}
+ /** {@inheritDoc} */
+ @Override
+ public BufferAllocator getAllocator() {
+return this.ownerAllocator;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void setAllocator(BufferAllocator allocator) {
+this.ownerAllocator = allocator;
Review comment:
sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469075#comment-16469075
]
ASF GitHub Bot commented on DRILL-6348:
---
vrozov commented on a change in pull request #1237: DRILL-6348: Fixed code so
that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r187104111
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
##
@@ -116,4 +120,16 @@ public void close() throws Exception {
AutoCloseables.close(buffers);
}
+ /** {@inheritDoc} */
+ @Override
+ public BufferAllocator getAllocator() {
+return this.ownerAllocator;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void setAllocator(BufferAllocator allocator) {
+this.ownerAllocator = allocator;
Review comment:
I'd recommend `Preconditions` over `assert`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[
https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6095:
---
Priority: Minor (was: Trivial)
> Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per
> standard practice
> ---
>
> Key: DRILL-6095
> URL: https://issues.apache.org/jira/browse/DRILL-6095
> Project: Apache Drill
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.12.0
> Environment: MapR 5.2
>
>Reporter: Hari Sekhon
>Assignee: Bridget Bevens
>Priority: Minor
>
> Drill's documentation on PStore configuration to HBase shows sample config
> with 4 ZooKeeper ip addresses in the {{hbase.zookeeper.quorum field}}, this
> should be fixed to 3 or 5 as per ZooKeeper best practices.
> [https://drill.apache.org/docs/persistent-configuration-storage/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage
[
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468999#comment-16468999
]
ASF GitHub Bot commented on DRILL-6348:
---
sachouche commented on a change in pull request #1237: DRILL-6348: Fixed code
so that Unordered Receiver reports its memory …
URL: https://github.com/apache/drill/pull/1237#discussion_r187091322
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java
##
@@ -116,4 +120,16 @@ public void close() throws Exception {
AutoCloseables.close(buffers);
}
+ /** {@inheritDoc} */
+ @Override
+ public BufferAllocator getAllocator() {
+return this.ownerAllocator;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void setAllocator(BufferAllocator allocator) {
+this.ownerAllocator = allocator;
Review comment:
I wondered about the same; the only reasons for the caller to pass a null,
is either a bug or allocator unregistration. I guess for now, I'll add an
assert since we do not really need an unregistration step (this will also help
us catch erroneous setting of the allocator).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
> Issue Type: Task
> Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the
> Unordered Receiver operator. This is problematic when analyzing OOM
> conditions since we cannot account for all of a query memory usage. This Jira
> is to fix memory reporting for the Unordered Receiver operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6394) Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object '/tmp/file.dat' not found within 'dfs'
Hari Sekhon created DRILL-6394:
--
Summary: Error: VALIDATION ERROR: From line 1, column 15 to line
1, column 17: Object '/tmp/file.dat' not found within 'dfs'
Key: DRILL-6394
URL: https://issues.apache.org/jira/browse/DRILL-6394
Project: Apache Drill
Issue Type: Improvement
Components: Server, Execution - Codegen, Metadata, Query Planning
Optimization, SQL Parser, Storage - Text CSV
Affects Versions: 1.13.0
Environment: MapR 6.0
Reporter: Hari Sekhon
Improvement request for the following error to be made more specific to mention
that this is caused by the file extension (.dat) not being one of the expected
ones, even though it was a CSV file (renaming it to .csv worked):
{code:java}
0: jdbc:drill:drillbit=> select * from dfs.`/tmp/file.dat`;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
'/tmp/file.dat' not found within 'dfs'
SQL Query null
[Error Id: e7c2863e-0feb-4b80-82b7-008056f0fcef on :31010] (state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468428#comment-16468428 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186932798 ## File path: docs/dev/TempDirectories.md ## @@ -0,0 +1,17 @@ +# Temp Directory Utilities + +The two basic temp directory classes are: + + - [DirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/DirTestWatcher.java) + - [BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java) + +These classes are used to create temp directories for each of your unit tests. The advantage to using +these temp directory classes are: + + - All files are deleted after a unit test completes. This prevents a build machine being polluted with a + bunch of unit test files. + - Each unit test outputs its files to a unique well defined location. This makes it easy to find files + for debugging. Also since each temp directory is unique, it prevents multiple unit test runs from interferring + with one another on a build machine. + +For examples on how to use these classes, please read the javadoc for each class. Review comment: Using the `ClusterFixture`, there is an easy way to define a schema that points to the test directory so you can use those directories to hold query input files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6392) System option exec.max_hash_table_size is completely unused. We should remove it.
[ https://issues.apache.org/jira/browse/DRILL-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas reassigned DRILL-6392: - Assignee: Timothy Farkas > System option exec.max_hash_table_size is completely unused. We should remove > it. > - > > Key: DRILL-6392 > URL: https://issues.apache.org/jira/browse/DRILL-6392 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468416#comment-16468416 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186884344 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). Review comment: There is a far easier way using the `ClusterFixture` (server-level tests) or `OperatorFixture` (lower-level tests). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468439#comment-16468439
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186932667
##
File path: docs/dev/RowSetFramework.md
##
@@ -0,0 +1,62 @@
+# RowSet Framework
+
+The RowSet Framework allows you to create custom instances of:
+
+ *
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+ *
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+ *
[RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java)
+
+It also allows the comparison of data container in
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
through the use
+of the
[RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java)
and
+[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java).
+
+## Creating A
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java)
+
+The
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java)
class can be used
+to create an instance
[BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java).
An example
+of how to to this can be found the **secondTest()** method of
[ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java).
+
+## Creating Test
[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)
+
+[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s
populated with data can be created with the
+[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java).
In order to use it do the following:
+
+ 1. Create an allocator
+```
+RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+```
+ 1. Create the desired BatchSchema using the
[SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java).
+```
+BatchSchema schema = new SchemaBuilder()
+.add(...)
+.add(...)
+.build();
+```
+ 1. Create a
[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java)
and add
+records to it. Then build a
[RowSet](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSet.java).
+```
+RowSet rowSet = new RowSetBuilder(allocator, schema)
+ .addRow(110, "green", new float[]{5.5f, 2.3f}, new String[]{"1a", "1b"})
Review comment:
Remember that improvement we discussed many months back? Those methods are
now available.
```
static import ...RowSetUtilities.strArray;
static import ...RowSetUtilities.floatArray; // Needed, only doubleArray at
present
.addRow(110, "green", floatArray(5.5f, 2.3f), strArray(1a", "1b"))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468437#comment-16468437
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933533
##
File path: docs/dev/TestDataSets.md
##
@@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+ - **Sample Data:** These are parquet files in the
[sample-data](../sample-data) folder.
+ - **Resource Data:** These are data files in the
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+ - **TPCH Data:** These are trimmed down versions of the tpch data sets. They
are retrieved and bundled
+ in the [contrib/data](../contrib/data) maven submodule. They are also
accessible on [Apache Drill's S3
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+ When unit tests are running all of the files in these data set are available
from the classpath storage plugin. The tpch
+ files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+
+### Using Sample Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage
plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+### Using Resource Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make data from
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources)
accessible from the ```dfs``` storage plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at
dfs.root.`join/empty_part` from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join", "empty_part"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // src/test/resources/join/empty_part is acessible at
dfs.root.`join/empty_part` from my test
+ }
+}
+```
+
+### Using TPCH Data in Unit Tests
+
+TPCH data is accessible via the classpath storage plugin
+
+```
+cp.`tpch/customer.parquet`
+```
+
+## Generating Data Sets
Review comment:
Two other possibilities.
To test a specific operator, there is the `MiniPlanUnitTestBase` and
associated cases that Jinfeng created to test operators one by one. (The code
sets up a plan, then uses that, along with a reader, to generate data that
exercises the operator.) There is an opportunity here to generate data from a
RowSet or consume the data as a RowSet for validation.
The other form, for "operator" and "sub-operator" tests is the RowSet
mechanism that can build up the exact set of vectors required. This is often
the only way to get the more obscure types that no readers will produce.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468415#comment-16468415 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186884623 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). + 1. Create a [SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java) as shown in Review comment: This is done automagically using the `OperatorFixture`. That class sets up everything needed to run a test without starting a server: configs, option manager, memory allocator, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468421#comment-16468421 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186886431 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework Review comment: Thanks much for writing this up! Suggestion: build on this to discuss the related ClusterFixture and OperatorFixture that let you run queries or operator-level code using the row sets. See the docs in my Wiki and in ExampleTest. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468456#comment-16468456
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933908
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
Review comment:
Sorry, can't parse that last sentence. Is effectively... what?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468432#comment-16468432
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933819
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
Review comment:
Model --> model
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468448#comment-16468448 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186935208 ## File path: docs/dev/DevDocs.md ## @@ -6,4 +6,8 @@ For information about configuring your development enviornment see [Environment. ## Testing -For information about how to do Integration and Unit Testing in Drill see [Testing.md](Testing.md). \ No newline at end of file +For information about how to do Integration and Unit Testing in Drill see [Testing.md](Testing.md). Review comment: Integration, Unit Testing --> no capitalization This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468452#comment-16468452 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186937205 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework + +The RowSet Framework allows you to create custom instances of: + + * [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + * [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + * [RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java) + +It also allows the comparison of data container in [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s through the use +of the [RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java) and +[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java). + +## Creating A [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + +The [SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java) class can be used +to create an instance [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java). An example +of how to to this can be found the **secondTest()** method of [ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java). + +## Creating Test [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + +[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s populated with data can be created with the +[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java). In order to use it do the following: + + 1. Create an allocator +``` +RootAllocator allocator = new RootAllocator(Long.MAX_VALUE); Review comment: Better: use the `OperatorFixture`. See `RowSetTest`. (That test exercises many ways to do things; it might be confusing as a result. Other tests show doing things the easiest possible way.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468429#comment-16468429 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186885946 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). + +## Creating An Instance of [QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java) + +``` +UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder() + .setPart1(1L) + .setPart2(2L) + .build(); +``` + +## Creating [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) + +``` +ExecProtos.FragmentHandle fragmentHandle = ExecProtos.FragmentHandle.newBuilder() + .setQueryId(queryId) + .setMinorFragmentId(1) + .setMajorFragmentId(2) + .build(); +``` + +## Creating A [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) + +There are a few ways to create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). See the class for all of them. The simplest way is to + create a default config. + +``` +DrillConfig config = DrillConfig.create(); +``` + +## Creating A [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java) + + 1. Create a [PhysicalOperator](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperator.java). +``` +HashJoinPOP pop = new HashJoinPOP(null, null, null, JoinRelType.FULL); +``` + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). + 1. Create a [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) as described above. + 1. Create a [SpillSet](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java). +``` +SpillSet spillSet = new SpillSet(config, fragmentHandle, pop); +``` + +## Creating A [PersistentStoreProvider](../../exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java) + +``` +LocalPersistentStoreProvider provider = new LocalPersistentStoreProvider(drillConfig); +provider.start(); +``` + +## Creating A [LogicalPlanPersistence](../../logical/src/main/java/org/apache/drill/common/config/LogicalPlanPersistence.java) + +``` +LogicalPlanPersistence logicalPlanPersistence = PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(drillConfig); +``` + +## Creating An Instance Of An Option Manager Review comment: Use the `OperatorFixture` or `ClusterFixture` which sets up all of this in a standard, repeatable way. Using those fixtures centralizes this stuff so we don't have to change it all if we again make changes to the option manager (as we did over the last year.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill >
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468454#comment-16468454 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186937319 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework + +The RowSet Framework allows you to create custom instances of: + + * [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + * [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + * [RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java) + +It also allows the comparison of data container in [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s through the use +of the [RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java) and +[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java). + +## Creating A [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + +The [SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java) class can be used +to create an instance [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java). An example +of how to to this can be found the **secondTest()** method of [ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java). + +## Creating Test [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + +[VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s populated with data can be created with the +[RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java). In order to use it do the following: + + 1. Create an allocator +``` +RootAllocator allocator = new RootAllocator(Long.MAX_VALUE); +``` + 1. Create the desired BatchSchema using the [SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java). +``` +BatchSchema schema = new SchemaBuilder() +.add(...) +.add(...) +.build(); +``` + 1. Create a [RowSetBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java) and add Review comment: As noted elsewhere, use the `TupleSchema` form to get a schema that can build complex types: `BatchSchema` is pretty limited beyond flat rows with scalar types. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468444#comment-16468444
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934976
##
File path: common/src/test/java/org/apache/drill/test/DirTestWatcher.java
##
@@ -34,7 +34,7 @@
* A {@link DirTestWatcher} is added to a test by declaring it as a JUnit
{@link org.junit.Rule}. A {@link org.junit.Rule} is
* a piece of code that is run before and after every JUnit test marked with
the {@link org.junit.Test} annotation. When the
* {@link DirTestWatcher} is added to a test class the {@link DirTestWatcher}
will create a temp directory before each of your
- * {@link org.junit.Test}s and optionally delete the temp directory after each
of your {@link org.junit.Test}s. The temp directory
+ * {@link org.junit.Test}s and optionally delete the temp directory after each
of your {@link org.junit.Test}s. The base temp directory
Review comment:
I forget, does the above form show just the base name or do you need to add
that: `{@link org.junit.Test Test}`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468443#comment-16468443 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186936042 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). Review comment: Generally, mocking these is needed when doing lower-level tests. But, that is exactly the problem that `OperatorFixture` attempts to solve. The one possible exception is when testing the root operators such as network exchanges: those will need more functionality then the `OperatorFixture` provides. Maybe we need to invent a `NetworkFixture` class to add in however we want to setup networking, etc. for tests. Sounds like a great project for when someone starts mucking about with the root operators. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468427#comment-16468427 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186934424 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/ops/RootFragmentContext.java ## @@ -17,6 +17,9 @@ */ package org.apache.drill.exec.ops; +/** + * This interface is implemented by fragment roots. Review comment: Provides services needed by the fragment executor. (If that is accurate; I'm guessing that's what a "fragment root" is...) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468433#comment-16468433
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933958
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount`
option. By default this
Review comment:
By default this.. what?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468425#comment-16468425 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186932197 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework + +The RowSet Framework allows you to create custom instances of: + + * [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + * [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + * [RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java) Review comment: True, but these are low-level constructs. BatchSchema is quite limited in its ability to model complex types. Instead, to use these "advanced" types, use the TupleMetadata class, which the schema builder also builds. RowSet wraps a VectorContainer. RecordBatch is badly misnamed, it is actually an operator. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468423#comment-16468423
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186886168
##
File path: docs/dev/PhysicalOpUnitTestBase.md
##
@@ -0,0 +1,31 @@
+# Single Operator Unit Test
+
+It is possible to run an end to end test of an operator in isolation by
extending
+[PhysicalOpUnitTestBase](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/PhysicalOpUnitTestBase.java).
+
+A simple example of an operator level unit test is the following:
+
+```
+public class BasicPhysicalOpUnitTest extends PhysicalOpUnitTestBase {
+
+ @Test
+ public void testSimpleProject() {
+ Project projectConf = new Project(parseExprs("x+5", "x"), null);
+ List jsonBatches = Lists.newArrayList(
+ "[{\"x\": 5 },{\"x\": 10 }]",
+ "[{\"x\": 20 },{\"x\": 30 },{\"x\": 40 }]");
+ opTestBuilder()
+ .physicalOperator(projectConf)
+ .inputDataStreamJson(jsonBatches)
+ .baselineColumns("x")
+ .baselineValues(10l)
+ .baselineValues(15l)
+ .baselineValues(25l)
+ .baselineValues(35l)
+ .baselineValues(45l)
+ .go();
Review comment:
Would be great to extend/modify/replace this with something based on, or
similar to the `OperatorFixture`, using the `RowSet` stuff to allow much better
comparisons than the above. The above, for example, is not very strict about
values and (as I recall) cannot differentiate between nullable and non-nullable
types.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468436#comment-16468436
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934484
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java
##
@@ -27,17 +27,31 @@
import java.nio.file.Paths;
/**
+ * Overview
*
* This is a {@link DirTestWatcher} which creates all the temporary
directories required by a Drillbit and the various dfs.* storage
workspaces. It also
* provides convenience methods that do the following:
- *
*
*
* Copy project files to temp directories. This is useful for copying
the sample data into a temp directory.
* Copy resource files to temp.
* Updating parquet metadata files.
*
+ *
+ *
+ *
+ * When used the {@link BaseDirTestWatcher} creates the following
directories in the base temp directory (for a description of where the
base temp directory
Review comment:
When used the --> The
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468441#comment-16468441
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934529
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/file/JsonFileBuilder.java
##
@@ -35,6 +36,16 @@
import java.util.List;
import java.util.Map;
+/**
+ * Overview
+ *
+ * This class is used to build a json file containing the data in a {@link
RowSet}.
Review comment:
This class is used to build --> Builds
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468422#comment-16468422 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186885712 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors Review comment: General advice: don't mock! Use the operator fixture instead. Design code to avoid mocking. Mocking is often a strong signal that the code has unwanted dependencies that should be refactored. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468455#comment-16468455 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186934852 ## File path: README.md ## @@ -20,7 +20,7 @@ Please see the [Apache Drill Website](http://drill.apache.org/) or the [Apache D ## Join the community! -Apache Drill is an Apache Foundation project and is seeking all types of contributions. Please say hello on the Apache Drill mailing list or join our Google Hangouts for more information. (More information can be found at the Apache Drill website). +Apache Drill is an Apache Foundation project and is seeking all types of contributions. Please say hello on the Apache Drill mailing list or join our Google Hangouts for more information. (More information can be found at the Apache Drill website). Review comment: Very nice. Maybe split into shorter lines for easier editing commenting. On the end "mailing list, join our Google Hangouts, or see the Apache Drill website for more information." Then, enclose "Apache Drill website" in a link to that website. Maybe have a link for the mail list and hangouts to the corresponding pages in the Community part of the site. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468451#comment-16468451 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186937028 ## File path: docs/dev/RowSetFramework.md ## @@ -0,0 +1,62 @@ +# RowSet Framework + +The RowSet Framework allows you to create custom instances of: + + * [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + * [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) + * [RecordBatch](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatch.java) + +It also allows the comparison of data container in [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java)s through the use +of the [RowSetComparison](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetComparison.java) and +[RowSetUtilities](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetUtilities.java). + +## Creating A [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java) + +The [SchemaBuilder](../../exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java) class can be used +to create an instance [BatchSchema](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/BatchSchema.java). An example +of how to to this can be found the **secondTest()** method of [ExampleTest](../../exec/java-exec/src/test/java/org/apache/drill/test/ExampleTest.java). + +## Creating Test [VectorContainer](../../exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java) Review comment: In general, as noted elsewhere, create a `TupleSchema`, then use that to generate a `RowSet` that can be filled with data. From that you can get the `VectorContainer` if you need it. You can also wrap an existing `VectorContainer` so you can compare it with a `RowSet` with expected results. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468450#comment-16468450
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934045
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java
##
@@ -38,16 +38,8 @@
import io.netty.buffer.DrillBuf;
/**
- * Fragment context interface: separates implementation from definition.
- * Allows unit testing by mocking or reimplementing services with
- * test-time versions. The name is awkward, chosen to avoid renaming
- * the implementation class which is used in many places in legacy code.
- * New code should use this interface, and the names should eventually
- * be swapped with {@link FragmentContextImpl} becoming
- * FragmentContextImpl and this interface becoming
- * {@link FragmentContextImpl}.
+ * This interface provides the resources required by a non-exchange operator
to execute.
Review comment:
This interface provides --> Provides
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468435#comment-16468435
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933018
##
File path: docs/dev/TestDataSets.md
##
@@ -0,0 +1,146 @@
+# Data Sets
+
+Drill includes several data sets for testing, and also provides some tools for
generating test data sets.
+
+## Bundled Data Sets
+
+There are three primary data sets bundled with drill for testing:
+
+ - **Sample Data:** These are parquet files in the
[sample-data](../sample-data) folder.
+ - **Resource Data:** These are data files in the
[exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold.
+ - **TPCH Data:** These are trimmed down versions of the tpch data sets. They
are retrieved and bundled
+ in the [contrib/data](../contrib/data) maven submodule. They are also
accessible on [Apache Drill's S3
bucket](http://apache-drill.s3.amazonaws.com/files/sf-0.01_tpc-h_parquet.tgz).
+ When unit tests are running all of the files in these data set are available
from the classpath storage plugin. The tpch
+ files include:
+- **customer.parquet**
+- **lineitem.parquet**
+- **nation.parquet**
+- **orders.parquet**
+- **part.parquet**
+- **partsup.parquet**
+- **region.parquet**
+- **supplier.parquet**
+
+### Using Sample Data in Unit Tests
+
+When using the
[BaseDirTestWatcher](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
you
+can make [sample-data](../sample-data) accessible from the ```dfs``` storage
plugin by doing the following:
+
+```
+public class TestMyClass {
+ @ClassRule
+ public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
+
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
+```
+
+Or if you are extending
[BaseTestQuery](../exec/java-exec/src/test/java/org/apache/drill/test/BaseDirTestWatcher.java)
+
+```
+public class TestMyClass extends BaseTestQuery {
+ @BeforeClass
+ public static void setupFiles() {
+dirTestWatcher.copyFileToRoot(Paths.get("sample-data", "region.parquet"));
+ }
+
+ @Test
+ public void simpleTest() {
+ // dfs.root.`sample-data/region.parquet` will be accessible from my test
+ }
+}
Review comment:
This can be done very simply with `ClusterFixture`. See
`ClusterFixture.defineWorkspace(...)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468453#comment-16468453 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186935750 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). Review comment: eclipse --> Eclipse Note that this may no longer be true if we have 1) Moved to JDK 8, and 2) updated JMockit to the latest version. Upgrading, if not done, will be a pain because the author decided to change many things and many tests need refactoring as a result. Probably another reason to avoid JMockit... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468447#comment-16468447 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186938309 ## File path: docs/dev/Testing.md ## @@ -2,27 +2,108 @@ Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other libraries for testing. This page provides pointers to the information you need to work with Drill tests. We don't repeat that information; you will want to follow the links and read the original material to get a complete understanding of the libraries that Drill uses. -Caveat: information here about Drill is "reverse engineered" from the code; this page has not yet had the benefit of insight from the developers who created Drill's test structure. - -# Topics - -"Classic" Drill testing techniques +# Writing Tests Review comment: I've not read the full, original material. This stuff is really quite good. The one suggestion I'd make is to provide some overall design hints. For example, when should someone use `ClusterFixture` to create an integration test? When should they use `OperatorFixture` to do a lower-level test? The answer I followed was that any detailed testing of an operator or other internal mechanism can only be done at the unit level. (All the `RowSet`, `ExternalSort` and `ResultSetLoader` stuff are examples.) This may mean designing the code so it can be tested by controlling dependencies. Use the `RowSet` mechanism to set up a complete set of input data sets: types, cardinalities, nesting levels: whatever is relevant. As shown in the sort tests, clever coding can automate much of this stuff. There is even a function to generate data for every data type given an int so that can populate types with data that is easy to compare. Then, once the detailed internal tests are done, use integration tests to ensure the whole system works. At present, for most readers, integration tests are really the only approach because of the dependencies that Scan has. The in-flight work in the "batch sizing" stuff solves this problem, but only for newer readers designed to use that framework. The general rule is: test as close to your code as possible, refactoring and breaking dependencies where needed to accomplish this. On the other hand, if the test is more of a SQL or planner-level concept, then testing at the query level might be fine. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468424#comment-16468424 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186885108 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: + + * [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + * [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + +## Creating A [FunctionLookupContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionLookupContext.java) + +``` +new FunctionImplementationRegistry(drillConfig) +``` + +## Creating A [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java) + + 1. Create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) as shown in [MockingComponents.md](MockingComponents.md). + 1. Create a [SystemOptionManager](../../exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java) as shown in + [MockingComponents.md](MockingComponents.md). + 1. Create an instance of [CodeCompiler](../../exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java). +``` +new CodeCompiler(drillConfig, optionManager) +``` + +## Debugging Generated Code + +It is possible to set break points in generated code. + +### Instructions For IntelliJ + + 1. File→Project structure…→Modules→distribution→Sources → Add content root + 1. Chose /tmp/drill/codegen + 1. Mark it as Sources directory. + 1. Set saveCodeForDebugging(true) for the code generator of interest + 1. Run the unit test of interest + 1. Now some generated classes should appear in Intellij under the distribution module + 1. Set a break point in a generated class and run the unit test in debug mode Review comment: To step into the generated code, set a breakpoint just before we call into the setup method. Step into that method which will step into doSetup. This opens the generated code file (at least in Eclipse.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468431#comment-16468431 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186933636 ## File path: docs/dev/Testing.md ## @@ -2,27 +2,108 @@ Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other libraries for testing. This page provides pointers to the information you need to work with Drill tests. We don't repeat that information; you will want to follow the links and read the original material to get a complete understanding of the libraries that Drill uses. -Caveat: information here about Drill is "reverse engineered" from the code; this page has not yet had the benefit of insight from the developers who created Drill's test structure. - -# Topics - -"Classic" Drill testing techniques +# Writing Tests +* [Test Data Sets](TestDataSets.md) +* [Temp Directory Utilities](TempDirectories.md) * [Testing with JUnit](JUnit.md) * [Test Logging](TestLogging.md) + +## Deprecated Drill Testing Techniques + +This is a list of old Drill testing machinery that we have cleaner machinery for now. + +* [BaseTestQuery](BaseTestQuery.md): Deprecated, use [ClusterTest](ClusterTest.md) instead. Review comment: Thanks! Should go without saying, but if there are any features in the old class that are missing from the new one, we should go ahead and move them across. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468442#comment-16468442 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186933813 ## File path: docs/dev/Testing.md ## @@ -2,27 +2,108 @@ Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other libraries for testing. This page provides pointers to the information you need to work with Drill tests. We don't repeat that information; you will want to follow the links and read the original material to get a complete understanding of the libraries that Drill uses. -Caveat: information here about Drill is "reverse engineered" from the code; this page has not yet had the benefit of insight from the developers who created Drill's test structure. - -# Topics - -"Classic" Drill testing techniques +# Writing Tests +* [Test Data Sets](TestDataSets.md) +* [Temp Directory Utilities](TempDirectories.md) * [Testing with JUnit](JUnit.md) * [Test Logging](TestLogging.md) + +## Deprecated Drill Testing Techniques + +This is a list of old Drill testing machinery that we have cleaner machinery for now. + +* [BaseTestQuery](BaseTestQuery.md): Deprecated, use [ClusterTest](ClusterTest.md) instead. + +## Legacy Drill Testing Techniques + +This is a list of old Drill testing machinery for which there is no other alternative at the moment. + * [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md) -"Updated" Drill testing techniques +## Latest Drill Testing Techniques +* [RowSet Framework](RowSetFramework.md) * [Cluster Fixture Framework](ClusterFixture.md) * [Operator Fixture Framework](OperatorFixture.md) -* [The Mock Record Reader](MockRecordReader.md) +* [ClusterTest](ClusterTest.md) +* [Single Operator Unit Test](PhysicalOpUnitTestBase.md) +* [Mocking Components](MockingComponents.md) +* [Generated Code](GeneratedCode.md) Review comment: Maybe add Jinfeng's mini plan framework. I don't think that has gotten the attention it deserves. In the figure, would be great to combine that with the OperatorFixture framework to avoid many of the ad-hoc bits. Also, `OperatorFixture` to test below the server level. Many of the row set and result set sets used this to great effect. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468446#comment-16468446
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186933973
##
File path: docs/dev/Testing.md
##
@@ -2,27 +2,108 @@
Drill makes extensive use of [JUnit](http://junit.org/junit4/) and other
libraries for testing. This page provides pointers to the information you need
to work with Drill tests. We don't repeat that information; you will want to
follow the links and read the original material to get a complete understanding
of the libraries that Drill uses.
-Caveat: information here about Drill is "reverse engineered" from the code;
this page has not yet had the benefit of insight from the developers who
created Drill's test structure.
-
-# Topics
-
-"Classic" Drill testing techniques
+# Writing Tests
+* [Test Data Sets](TestDataSets.md)
+* [Temp Directory Utilities](TempDirectories.md)
* [Testing with JUnit](JUnit.md)
* [Test Logging](TestLogging.md)
+
+## Deprecated Drill Testing Techniques
+
+This is a list of old Drill testing machinery that we have cleaner machinery
for now.
+
+* [BaseTestQuery](BaseTestQuery.md): Deprecated, use
[ClusterTest](ClusterTest.md) instead.
+
+## Legacy Drill Testing Techniques
+
+This is a list of old Drill testing machinery for which there is no other
alternative at the moment.
+
* [Testing with Physical Plans and Mock Data](LegacyTestingFrameworks.md)
-"Updated" Drill testing techniques
+## Latest Drill Testing Techniques
+* [RowSet Framework](RowSetFramework.md)
* [Cluster Fixture Framework](ClusterFixture.md)
* [Operator Fixture Framework](OperatorFixture.md)
-* [The Mock Record Reader](MockRecordReader.md)
+* [ClusterTest](ClusterTest.md)
+* [Single Operator Unit Test](PhysicalOpUnitTestBase.md)
+* [Mocking Components](MockingComponents.md)
+* [Generated Code](GeneratedCode.md)
+
+## Categories
+
+Currently Drill uses Travis to run smoke tests for every PR and commit. All of
Drill's unit tests cannot be run on Travis because Drill's tests take longer to
run than the
+maximum allowed container time for the free tier of Travis. In order to decide
which tests are run on Travis and which tests are not, tests are categorized
using JUnit's
+`@Category` annotation. Currently the following categories are excluded from
Travis:
+
+ - **SlowTest:** Tests that are slow.
+ - **UnlikelyTest:** Tests that cover parts of the code that are rarely or
never touched.
+ - **SecurityTest:** Corner case tests for security features.
+
+To mark a test with a category you can do the following:
+
+```
+@Category(SlowTest.class)
+public class MyTest {
+ // Testing code
+}
+```
+
+To mark a test with multiple categories you can do the following:
+
+```
+@Category({SlowTest.class, SecurityTest.class})
+public class MyTest {
+ // Testing code
+}
+```
+
+# Running Tests
+
+Drill tests run in parallel. The Model for parallel execution is to divide
test classes between multiple
+forked test processes. Each test process then runs the test classes assigned
to it sequentially.
+
+## Speeding Up Test Runs
+
+There are a couple knobs you can turn to make tests run faster on your machine.
+
+ * **Maven Build Threads**: `-T `
+ * **Sure Fire Fork Count**: `-DforkCount=`
+ * **Test Categories**
+
+### -T
+
+Maven allows you to use multiple threads to compile sub modules. Also when
running tests each build
+thread forks its own surefire process, so the tests for different submodules
are run in parallel. In order
+to leverage this use the `-T` flag. By default this option is effectively.
+
+Ex. In order to run the build using two maven threads use the following
command.
+
+```
+mvn -T 2 clean install
+```
+
+### -DforkCount
+
+To run tests within a submodule in parallel you can use the `-DforkCount`
option. By default this
+
+Ex. Run 4 test processes in parallel
+
+```
+mvn clean install -DforkCount=4
+```
+
+**Note:** The `-DforkCount` option interacts with `-T`. When use together each
build thread (`-T`) gets
Review comment:
use --> used
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468426#comment-16468426 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186933202 ## File path: docs/dev/TestDataSets.md ## @@ -0,0 +1,146 @@ +# Data Sets + +Drill includes several data sets for testing, and also provides some tools for generating test data sets. + +## Bundled Data Sets + +There are three primary data sets bundled with drill for testing: + + - **Sample Data:** These are parquet files in the [sample-data](../sample-data) folder. + - **Resource Data:** These are data files in the [exec/java-exec/src/test/resources](../exec/java-exec/src/test/resources) fold. Review comment: fold --> folder These files are often crafted to show specific edge cases needed for some specific test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468445#comment-16468445 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186935543 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* Review comment: Extra asterisk This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468419#comment-16468419 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186885551 ## File path: docs/dev/LicenseHeaders.md ## @@ -0,0 +1,34 @@ +# License Headers + +Drill uses two license header checkers: + +* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) +* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/) + +## Why Two? + +[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used because it is the standard license header +checker for Apache projects. + +[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs stricter license checks and disallows +putting license headers in java docs. + +## Doing License Checks + +The license checks are disabled locally by default and are enabled on Travis. If you'd like to perform +license checks locally you can do the following: + +``` + mvn license:check -Dlicense.skip=false +``` + +## Auto Formatting Headers + +If the license checks fail and you can't figure out what's wrong with your headers, you can auto-format +your license headers with the following command: + +``` +mvn license:format -Dlicense.skip=false +``` + +This command will also add license headers to files without them. Review comment: This seems a bit of a risk... Will it only do this for Java files? RAT will check all files, even data files. If the license plugin adds headers to these files, the files can end up being corrupted. Maybe just add a note to check all changed files prior to commit to identify any issues and customize (what) to exclude unwanted files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468420#comment-16468420 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186885267 ## File path: docs/dev/LicenseHeaders.md ## @@ -0,0 +1,34 @@ +# License Headers + +Drill uses two license header checkers: + +* [Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) +* [License Maven Plugin](http://code.mycila.com/license-maven-plugin/) + +## Why Two? + +[Apache RAT Plugin](http://creadur.apache.org/rat/apache-rat-plugin/) is used because it is the standard license header +checker for Apache projects. + +[License Maven Plugin](http://code.mycila.com/license-maven-plugin/) performs stricter license checks and disallows +putting license headers in java docs. Review comment: Explain what this means: it simply means to start a license header with /* and not /** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[
https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468438#comment-16468438
]
ASF GitHub Bot commented on DRILL-6249:
---
paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding
more unit testing documentation.
URL: https://github.com/apache/drill/pull/1251#discussion_r186934308
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java
##
@@ -70,9 +69,26 @@
import io.netty.buffer.DrillBuf;
/**
- * Contextual objects required for execution of a particular fragment.
- * This is the implementation; use FragmentContext
- * in code to allow tests to use test-time implementations.
+ *
+ * This is the core Context which implements all the Context interfaces:
+ *
+ *
+ * {@link FragmentContext}: A context provided to non-exchange
operators.
+ * {@link ExchangeFragmentContext}: A context provided to exchange
operators.
+ * {@link RootFragmentContext}: A context provided to fragment
roots.
+ * {@link ExecutorFragmentContext}: A context used by the
Drillbit.
+ *
+ *
+ * The interfaces above expose resources to varying degrees. They are
ordered from most restrictive ({@link FragmentContext})
+ * to least restrictive ({@link ExecutorFragmentContext}).
+ *
+ *
+ * Since {@link FragmentContextImpl} implements all of the interfaces listed
above, the facade pattern is used in order
+ * to cast a {@link FragmentContextImpl} object to the desired interface
where-ever it is needed. The facade pattern
+ * is powerful since it allows us to easily create minimal mock context
objects to be used in unit tests. Without
+ * the use of interfaces and the facade pattern we would have to create a
complete {@link FragmentContextImpl} object
+ * to unit test any part of the code that depends on a context.
+ *
Review comment:
General guideline: use the most narrow interface for the task. For example,
"internal" operators don't need visibility to the networking functionality.
Using the narrow interface allows unit testing without mocking.
Often, the surrounding structure already has exposed the most narrow
interface. If there are opportunities clean up older code, we can do so as
needed to make testing easier.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Markdown Docs for Unit Testing and Link to it in README.md
> --
>
> Key: DRILL-6249
> URL: https://issues.apache.org/jira/browse/DRILL-6249
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> I am working on a presentation about how to use the unit testing utilities in
> Drill. Instead of writing the doc and having it be lost in Google Drive
> somewhere I am going to add a Markdown doc to the drill repo and link to it
> in the README.md. This is appropriate since these docs will only be used by
> developers, and the way we unit test will change as the code changes. So the
> unit testing docs should be kept in the same repo as the code so it can be
> updated and kept in sync with the rest of Drill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468440#comment-16468440 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186935126 ## File path: docs/dev/ClusterTest.md ## @@ -0,0 +1,4 @@ +# ClusterTest Review comment: And `OperatorTest` for non-server (operator-level) tests. Sets up all the bits and pieces needed to work with Drill internals without an enclosing server. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468414#comment-16468414 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186884221 ## File path: docs/dev/GeneratedCode.md ## @@ -0,0 +1,42 @@ +# Testing Generated Code + +## Writing Unit Tests For Generated Code + +An example of unit testing generated code without running all of Drill is the **priorityQueueOrderingTest()** test in +[TopNBatchTest](../../exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TopN/TopNBatchTest.java). That test tests the +[PriorityQueueTemplate](../../exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/TopN/PriorityQueueTemplate.java) class separately from the rest of Drill. + +The main accomplished by creating instances of the following classes: Review comment: Awkward... The main (what?) accomplished... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468449#comment-16468449 ] ASF GitHub Bot commented on DRILL-6249: --- paul-rogers commented on a change in pull request #1251: DRILL-6249: Adding more unit testing documentation. URL: https://github.com/apache/drill/pull/1251#discussion_r186936197 ## File path: docs/dev/MockingComponents.md ## @@ -0,0 +1,96 @@ +# Mocking Components + +There are a few techniques for mocking classes in unit tests: + +* Use a mocking library +* Provide a simple impementation of an interface +* Build a real instance of class using the class's builders / constructors +* + +## Mocking Libraries + +Drill uses two mocking libraries in order to mock classes. + +* [Mockito](http://site.mockito.org) +* [JMockit](http://jmockit.github.io/tutorial.html) (Deprecated) + +While some tests use [JMockit](http://jmockit.github.io/tutorial.html), its usage has been deprecated since it doesn't work well with +eclipse. Specifically any test that uses JMockit will fail when run with eclipse. So if you +need to use a mocking library use [Mockito](http://site.mockito.org). + +## Mocking Contexts + +There are several contexts used throughout Drill, for a complete description of each and how +they are used please see [FragmentContextImpl](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContextImpl.java). + +When doing tests you can use the following mock contexts: + + * [MockFragmentContext](../../exec/java-exec/src/test/java/org/apache/drill/test/OperatorFixture.java) is a simple mock implementation of + the [FragmentContext](../../exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java). + +## Creating An Instance of [QueryId](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryId.java) + +``` +UserBitShared.QueryId queryId = UserBitShared.QueryId.newBuilder() + .setPart1(1L) + .setPart2(2L) + .build(); +``` + +## Creating [FragmentHandle](../../protocol/src/main/java/org/apache/drill/exec/proto/beans/FragmentHandle.java) + +``` +ExecProtos.FragmentHandle fragmentHandle = ExecProtos.FragmentHandle.newBuilder() + .setQueryId(queryId) + .setMinorFragmentId(1) + .setMajorFragmentId(2) + .build(); +``` + +## Creating A [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java) + +There are a few ways to create a [DrillConfig](../../common/src/main/java/org/apache/drill/common/config/DrillConfig.java). See the class for all of them. The simplest way is to + create a default config. + +``` +DrillConfig config = DrillConfig.create(); Review comment: See the `ConfigBuilder`. It is the guts behind the `ClusterFixture` and `OperatorFixture` for configs. It allows the fluent-style construction used in these two contexts. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
