[jira] [Created] (DRILL-7351) WebUI is Vulnerable to CSRF
Don Perial created DRILL-7351: - Summary: WebUI is Vulnerable to CSRF Key: DRILL-7351 URL: https://issues.apache.org/jira/browse/DRILL-7351 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.16.0 Reporter: Don Perial Attachments: drill-csrf.html There is no way to protect the WebUI from CSRF and the fact that the value for the access-control-allow-origin header is '*' appears to confound this issue as well. The attached file demonstrates the vulnerability. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908362#comment-16908362 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on issue #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#issuecomment-521708575 @vvysotskyi, thanks for making this change. It will allow certain other tasks to be much easier. The first observation is that it would be best not to put the RowSet and ResultSetLoader stuff in the same package: they represent two independent systems and our package naming should make that clear. Perhaps 1) Rename the existing physical.rowSet to physical.resultSet. 2) Move the RowSet classes to a new physical.rowSet. The difference, so we're clear, is that RowSet works on a single batch, while the ResultSet works across a stream of batches. If operators, say, want to use the row set stuff to read batches, then we need a ResultSetReader that automagically handles things like schema changes across batches, etc. To be very clear: "row set" -- A collection of rows. Typically called a "batch" in Drill. "result set" -- A collection of rows for an entire query. Typically consists of multiple (related) batches. I used the term "row set" only because the classes would otherwise get lost with all the existing "Batch" classes. And, we don't have a good name for the result set. I borrowed this name from SQL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908300#comment-16908300 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on issue #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#issuecomment-521715916 Two general comments. First, not sure why we needed to convert compact single-line methods (a long-time Drill standard) into four-line methods. Perhaps this is the result of an overly-helpful code formatter? Second, looks like a bunch of nested helper classes were moved elsewhere. Not sure this is helpful as it clutters the code with small files used in exactly one place. The nested classes (a perfectly normal Java practice) makes clear that these are, in fact, helpers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908293#comment-16908293
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314398293
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ * Example of the output:
+ *
+ * #: year, month, day
+ * 0: 2017, 12, 17
+ * 1: 2017, 12, 18
+ * 2: 2017, 12, 19
+ *
+ */
+public class RowSetStringBuilder {
Review comment:
The problem with this approach is that row sets can be large: creating a
string to print causes memory pressure.
Consider creating a row set builder that takes a stream (or Writer). The
Writer can be a String writer. That way, when used with a large data set, the
writer can be to stdout, or whatever.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908289#comment-16908289
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314395723
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractRowSet.java
##
@@ -41,13 +38,19 @@ public AbstractRowSet(VectorContainer container,
TupleMetadata schema) {
}
@Override
- public VectorAccessible vectorAccessible() { return container(); }
+ public VectorAccessible vectorAccessible() {
Review comment:
Why change formatting? Drill has long used single-line methods for
conciseness. Is this an artifact of some code formatter? I didn't see
discussion on the dev list about changing our coding standards...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908297#comment-16908297
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314400808
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/package-info.java
##
@@ -117,12 +134,14 @@
* are added while loading, their index is always at the end of the existing
* columns.
* Writing Data to the Batch
- * Each batch is delimited by a call to {@link #startBatch()} and a call to
- * {@link #harvestWithLookAhead()} to obtain the completed batch. Note that
readers do not
+ * Each batch is delimited by a call to {@link
org.apache.drill.exec.physical.rowSet.ResultSetLoader#startBatch()}
+ * and a call to {@link
org.apache.drill.exec.physical.rowSet.impl.VectorState#harvestWithLookAhead()}
+ * to obtain the completed batch. Note that readers do not
* call these methods; the scan operator does this work.
*
- * Each row is delimited by a call to {@link #startValue()} and a call to
- * {@link #saveRow()}. startRow() performs initialization necessary
+ * Each row is delimited by a call to {@link
org.apache.drill.exec.vector.accessor.writer.WriterEvents#startRow()}
Review comment:
WriterEvents is meant to be entirely internal; it should not be part of the
API. WriterEvents is the internal implementation of the public API described
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908292#comment-16908292
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314396758
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractSingleRowSet.java
##
@@ -66,4 +57,14 @@ public long size() {
protected RowSetReader buildReader(ReaderIndex rowIndex) {
return new RowSetReaderBuilder().buildReader(this, rowIndex);
}
+
+ public static class RowSetReaderBuilder extends BaseReaderBuilder {
Review comment:
See note above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908296#comment-16908296
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314397262
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java
##
@@ -141,5 +136,22 @@ public SingleRowSet toIndirect(Set skipIndices) {
}
@Override
- public SelectionVector2 getSv2() { return null; }
+ public SelectionVector2 getSv2() {
+return null;
+ }
+
+ public static class RowSetWriterBuilder extends BaseWriterBuilder {
Review comment:
See note above about cluttering the project with many small classes used in
only one location.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908288#comment-16908288 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314395350 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractRowSet.java ## @@ -15,23 +15,20 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; import org.apache.drill.exec.memory.BufferAllocator; import org.apache.drill.exec.record.BatchSchema; import org.apache.drill.exec.record.VectorAccessible; import org.apache.drill.exec.record.VectorContainer; import org.apache.drill.exec.record.metadata.TupleMetadata; -import org.apache.drill.exec.vector.SchemaChangeCallBack; /** * Basic implementation of a row set for both the single and multiple - * (hyper) varieties, both the fixed and extendible varieties. + * (hyper) varieties, both the fixed and extendable varieties. Review comment: extensible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908299#comment-16908299
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314401892
##
File path:
exec/java-exec/src/test/java/org/apache/drill/test/QueryRowSetIterator.java
##
@@ -99,13 +99,6 @@ public DirectRowSet next() {
}
}
- public void printAll() {
Review comment:
Please either leave this, or move it to a utility class. This turns out to
be a very handy way to validate or debug a unit test: add a call while working
on the test to visualize the results, remove the call when done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908291#comment-16908291
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314397062
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java
##
@@ -45,21 +45,10 @@
public class DirectRowSet extends AbstractSingleRowSet implements
ExtendableRowSet {
- public static class RowSetWriterBuilder extends BaseWriterBuilder {
-
-public RowSetWriterBuilder(ColumnConversionFactory conversionFactory) {
- super(conversionFactory);
-}
-
-public RowSetWriter buildWriter(DirectRowSet rowSet) {
- WriterIndexImpl index = new WriterIndexImpl();
- TupleMetadata schema = rowSet.schema();
- RowSetWriterImpl writer = new RowSetWriterImpl(rowSet, schema, index,
- buildContainerChildren(rowSet.container(),
- new MetadataRetrieval(schema)));
- return writer;
-}
- }
+ /**
+ * Initial row count, used for preliminary memory allocation.
+ */
+ public static final int INITIAL_ROW_COUNT = 10;
Review comment:
See note above about removal of nested class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908290#comment-16908290
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314396624
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractSingleRowSet.java
##
@@ -15,37 +15,27 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.drill.test.rowSet;
+package org.apache.drill.exec.physical.rowSet;
import org.apache.drill.exec.record.RecordBatchSizer;
import org.apache.drill.exec.physical.rowSet.model.ReaderIndex;
import
org.apache.drill.exec.physical.rowSet.model.MetadataProvider.MetadataRetrieval;
import org.apache.drill.exec.physical.rowSet.model.single.BaseReaderBuilder;
import org.apache.drill.exec.record.VectorContainer;
import org.apache.drill.exec.record.metadata.TupleMetadata;
-import org.apache.drill.test.rowSet.RowSet.SingleRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet;
/**
* Base class for row sets backed by a single record batch.
*/
public abstract class AbstractSingleRowSet extends AbstractRowSet implements
SingleRowSet {
- public static class RowSetReaderBuilder extends BaseReaderBuilder {
-
-public RowSetReader buildReader(AbstractSingleRowSet rowSet, ReaderIndex
rowIndex) {
- TupleMetadata schema = rowSet.schema();
- return new RowSetReaderImpl(schema, rowIndex,
- buildContainerChildren(rowSet.container(),
- new MetadataRetrieval(schema)));
-}
- }
-
- public AbstractSingleRowSet(AbstractSingleRowSet rowSet) {
+ protected AbstractSingleRowSet(AbstractSingleRowSet rowSet) {
Review comment:
Why remove nested classes? These are perfectly normal part of Java that
allows small, single-purpose classes to be associated with the one and only
class that uses them. Again, if we want to change the Drill style guidelines,
let's have the discussion on dev list first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908294#comment-16908294
]
ASF GitHub Bot commented on DRILL-7350:
---
paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314398737
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetWriterImpl.java
##
@@ -129,7 +129,7 @@ public RowSetWriter addRow(Object...values) {
@Override
public RowSetWriter addSingleCol(Object value) {
-return addRow(new Object[] {value});
+return addRow(value);
Review comment:
Won't work. Must be an array. The idea is that addRow() takes an array of
columns. But, if the row has a single column, and that one column is repeated,
the Java code is ambiguous. The original (correct) implementation handles this
case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908298#comment-16908298 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314400385 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/package-info.java ## @@ -32,6 +32,22 @@ * batch" or "record batch." (But, in Drill, the term "record batch" also * usually means an operator on that set of records. Here, a row set is * just the rows separate from operations on that data. + * SingleRowSet (abstract) Review comment: See general comment. Please do not combine the RowSet and ResultSetLoader classes into a single package: they present very different models used for different purposes. Yes, it is confusing because there is no ResultSetReader at present, but there should be to handle schema changes, etc. across batches. RowSet handles only a single row set such as in tests. It was deliberately designed to NOT be ResultSetReader solution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908295#comment-16908295 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314399600 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/package-info.java ## @@ -32,6 +32,22 @@ * batch" or "record batch." (But, in Drill, the term "record batch" also * usually means an operator on that set of records. Here, a row set is * just the rows separate from operations on that data. + * SingleRowSet (abstract) + * Represents a row set that contains a single record batch (the typical + * case. + * DirectRowSet + * A read-only single row set without a selection vector. + * IndirectRowSet + * A read-only, single row set with an SV2. Note that the SV2 itself is + * writable (such as for sorting.) + * ExtendableRowSet + * A write-only, single row set used to create a new row set. Because of + * the way Drill sets row counts, an extendable row set cannot be read; instead + * at the completion of the write the extendable row set becomes a direct or + * indirect row set. + * HyperRowSet + * A read-only row set made up of a collection of record batches, indexed via an + * SV4. As with the SV2, the SV4 itself is writable. Review comment: The SV is not writable within a RowSet. Changing the SV will mess up indexing. The way to use that is to have a DirectRowSet on the original batch. Create the SV. Then, use the DirectRowSet and the SV to create an indirect row set. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908269#comment-16908269 ] ASF GitHub Bot commented on DRILL-7350: --- paul-rogers commented on issue #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#issuecomment-521708575 @vvysotskyi, thanks for making this change. It will allow certain other tasks to be much easier. The first observation is that it would be best not to put the RowSet and ResultSetLoader stuff in the same package: they represent two independent systems and our package naming should make that clear. Perhaps 1) Rename the existing physical.rowSet to physical.resultSet. 2) Move the RowSet classes to a new physical.rowSet. The difference, so we're clear, is that RowSet works on a single batch, while the ResultSet works across a stream of batches. If operators, say, want to use the row set stuff to read batches, then we need a ResultSetReader that automagically handles things like schema changes across batches, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908255#comment-16908255
]
ASF GitHub Bot commented on DRILL-7350:
---
vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314387694
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
Review comment:
Made private and non-static.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908256#comment-16908256
]
ASF GitHub Bot commented on DRILL-7350:
---
vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314388393
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
+for (int i = 0; i < schema.size(); i++) {
+ if (i > 0) {
+stringBuilder.append(", ");
+ }
+ ColumnMetadata colSchema = schema.metadata(i);
+ stringBuilder.append(colSchema.name());
+ if (colSchema.isMap()) {
+stringBuilder.append("{");
+appendTupleSchema(stringBuilder, colSchema.mapSchema());
+stringBuilder.append("}");
+ }
+}
+ }
+
+ @Override
+ public String toString() {
+StringBuilder result = new StringBuilder();
+appendTo(result);
+return result.toString();
+ }
+
+ public void appendTo(StringBuilder stringBuilder) {
Review comment:
Thanks, done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908253#comment-16908253
]
ASF GitHub Bot commented on DRILL-7350:
---
vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314388642
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
+for (int i = 0; i < schema.size(); i++) {
+ if (i > 0) {
+stringBuilder.append(", ");
+ }
+ ColumnMetadata colSchema = schema.metadata(i);
+ stringBuilder.append(colSchema.name());
+ if (colSchema.isMap()) {
+stringBuilder.append("{");
+appendTupleSchema(stringBuilder, colSchema.mapSchema());
+stringBuilder.append("}");
+ }
+}
+ }
+
+ @Override
+ public String toString() {
+StringBuilder result = new StringBuilder();
+appendTo(result);
+return result.toString();
+ }
+
+ public void appendTo(StringBuilder stringBuilder) {
+SelectionVectorMode selectionMode = rowSet.indirectionType();
+RowSetReader reader = rowSet.reader();
+int colCount = reader.tupleSchema().size();
+appendSchema(stringBuilder, selectionMode, reader);
+while (reader.next()) {
+ appendHeader(stringBuilder, reader, selectionMode);
+ for (int i = 0; i < colCount; i++) {
+if (i > 0) {
+ stringBuilder.append(", ");
+}
+stringBuilder.append(reader.column(i).getAsString());
+ }
+ stringBuilder.append("\n");
+}
+ }
+
+ private void appendSchema(StringBuilder stringBuilder, SelectionVectorMode
selectionMode, RowSetReader reader) {
+stringBuilder.append("#");
+switch (selectionMode) {
+ case FOUR_BYTE:
+stringBuilder.append(" (batch #, row #)");
+break;
+ case TWO_BYTE:
+stringBuilder.append(" (row #)");
+break;
+ default:
+break;
+}
+stringBuilder.append(": ");
+TupleMetadata schema = reader.tupleSchema();
+appendTupleSchema(stringBuilder, schema);
+stringBuilder.append("\n");
+ }
+
+ private void appendHeader(StringBuilder stringBuilder, RowSetReader reader,
SelectionVectorMode selectionMode) {
+stringBuilder.append(reader.logicalIndex());
+switch (selectionMode) {
+ case FOUR_BYTE:
+stringBuilder.append(" (");
+stringBuilder.append(reader.hyperVectorIndex());
+stringBuilder.append(", ");
+stringBuilder.append(reader.offset());
+stringBuilder.append(")");
+break;
+ case TWO_BYTE:
+stringBuilder.append(" (");
+stringBuilder.append(reader.offset());
+stringBuilder.append(")");
+break;
+ default:
Review comment:
No, it is not required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908257#comment-16908257 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314388748 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java ## @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.physical.rowSet; + +import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode; +import org.apache.drill.exec.record.metadata.ColumnMetadata; +import org.apache.drill.exec.record.metadata.TupleMetadata; + +/** + * Helper class to obtain string representation of RowSet. Review comment: Agree, added an example. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908254#comment-16908254
]
ASF GitHub Bot commented on DRILL-7350:
---
vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314387494
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
Review comment:
Thanks, done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908235#comment-16908235
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314378996
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
Review comment:
final?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908238#comment-16908238 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314382345 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java ## @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.physical.rowSet; + +import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode; +import org.apache.drill.exec.record.metadata.ColumnMetadata; +import org.apache.drill.exec.record.metadata.TupleMetadata; + +/** + * Helper class to obtain string representation of RowSet. Review comment: It would be nice to add an example of the output. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908236#comment-16908236
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314382220
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
+for (int i = 0; i < schema.size(); i++) {
+ if (i > 0) {
+stringBuilder.append(", ");
+ }
+ ColumnMetadata colSchema = schema.metadata(i);
+ stringBuilder.append(colSchema.name());
+ if (colSchema.isMap()) {
+stringBuilder.append("{");
+appendTupleSchema(stringBuilder, colSchema.mapSchema());
+stringBuilder.append("}");
+ }
+}
+ }
+
+ @Override
+ public String toString() {
+StringBuilder result = new StringBuilder();
+appendTo(result);
+return result.toString();
+ }
+
+ public void appendTo(StringBuilder stringBuilder) {
Review comment:
Same here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908239#comment-16908239
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314378880
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
+for (int i = 0; i < schema.size(); i++) {
+ if (i > 0) {
+stringBuilder.append(", ");
+ }
+ ColumnMetadata colSchema = schema.metadata(i);
+ stringBuilder.append(colSchema.name());
+ if (colSchema.isMap()) {
+stringBuilder.append("{");
+appendTupleSchema(stringBuilder, colSchema.mapSchema());
+stringBuilder.append("}");
+ }
+}
+ }
+
+ @Override
+ public String toString() {
+StringBuilder result = new StringBuilder();
+appendTo(result);
+return result.toString();
+ }
+
+ public void appendTo(StringBuilder stringBuilder) {
+SelectionVectorMode selectionMode = rowSet.indirectionType();
+RowSetReader reader = rowSet.reader();
+int colCount = reader.tupleSchema().size();
+appendSchema(stringBuilder, selectionMode, reader);
+while (reader.next()) {
+ appendHeader(stringBuilder, reader, selectionMode);
+ for (int i = 0; i < colCount; i++) {
+if (i > 0) {
+ stringBuilder.append(", ");
+}
+stringBuilder.append(reader.column(i).getAsString());
+ }
+ stringBuilder.append("\n");
+}
+ }
+
+ private void appendSchema(StringBuilder stringBuilder, SelectionVectorMode
selectionMode, RowSetReader reader) {
+stringBuilder.append("#");
+switch (selectionMode) {
+ case FOUR_BYTE:
+stringBuilder.append(" (batch #, row #)");
+break;
+ case TWO_BYTE:
+stringBuilder.append(" (row #)");
+break;
+ default:
+break;
+}
+stringBuilder.append(": ");
+TupleMetadata schema = reader.tupleSchema();
+appendTupleSchema(stringBuilder, schema);
+stringBuilder.append("\n");
+ }
+
+ private void appendHeader(StringBuilder stringBuilder, RowSetReader reader,
SelectionVectorMode selectionMode) {
+stringBuilder.append(reader.logicalIndex());
+switch (selectionMode) {
+ case FOUR_BYTE:
+stringBuilder.append(" (");
+stringBuilder.append(reader.hyperVectorIndex());
+stringBuilder.append(", ");
+stringBuilder.append(reader.offset());
+stringBuilder.append(")");
+break;
+ case TWO_BYTE:
+stringBuilder.append(" (");
+stringBuilder.append(reader.offset());
+stringBuilder.append(")");
+break;
+ default:
Review comment:
Do we need default here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908237#comment-16908237
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314382035
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetStringBuilder.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+
+/**
+ * Helper class to obtain string representation of RowSet.
+ */
+public class RowSetStringBuilder {
+ private RowSet rowSet;
+
+ public RowSetStringBuilder(RowSet rowSet) {
+this.rowSet = rowSet;
+ }
+
+ public static void appendTupleSchema(StringBuilder stringBuilder,
TupleMetadata schema) {
Review comment:
Why this is public and static?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908216#comment-16908216 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314362960 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSet.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Thanks, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908219#comment-16908219 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314367564 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetReader.java ## @@ -46,8 +46,7 @@ * The index of the underlying row which may be indexed by an * SV2 or SV4. * - * @return + * @return index of the underlying row */ - int offset(); } Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908214#comment-16908214 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314362662 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/IndirectRowSet.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Thanks, replaced with `Collections.emptySet()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908212#comment-16908212 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314359422 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractRowSet.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Thanks, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908221#comment-16908221 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314369366 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java ## @@ -20,12 +20,15 @@ import java.util.Set; import org.apache.drill.exec.memory.BufferAllocator; +import org.apache.drill.exec.physical.rowSet.DirectRowSet; +import org.apache.drill.exec.physical.rowSet.RowSet; +import org.apache.drill.exec.physical.rowSet.RowSetWriter; import org.apache.drill.exec.record.BatchSchema; import org.apache.drill.exec.record.metadata.MetadataUtils; import org.apache.drill.exec.record.metadata.TupleMetadata; import org.apache.drill.exec.vector.accessor.convert.ColumnConversionFactory; import org.apache.drill.shaded.guava.com.google.common.collect.Sets; -import org.apache.drill.test.rowSet.RowSet.SingleRowSet; +import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet; Review comment: Thanks, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908218#comment-16908218 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314367486 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetReader.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Thanks, renamed to `setPosition()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908217#comment-16908217
]
ASF GitHub Bot commented on DRILL-7350:
---
vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related
classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314361974
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java
##
@@ -132,7 +131,7 @@ public RowSetReader reader() {
@Override
public SingleRowSet toIndirect() {
-return new IndirectRowSet(this, Sets.newHashSet());
+return new IndirectRowSet(this, new HashSet<>());
Review comment:
Agree, replaced with `Collections.emptySet()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908213#comment-16908213 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314360423 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java ## @@ -33,9 +32,10 @@ import org.apache.drill.exec.record.VectorContainer; import org.apache.drill.exec.record.selection.SelectionVector2; import org.apache.drill.exec.vector.accessor.convert.ColumnConversionFactory; -import org.apache.drill.test.rowSet.RowSet.ExtendableRowSet; -import org.apache.drill.test.rowSet.RowSetWriterImpl.WriterIndexImpl; +import org.apache.drill.exec.physical.rowSet.RowSet.ExtendableRowSet; +import org.apache.drill.exec.physical.rowSet.RowSetWriterImpl.WriterIndexImpl; +import java.util.HashSet; Review comment: Thanks, done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908220#comment-16908220 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314365904 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetPrinter.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Thanks, replaced `PrintStream` usage with `StringBuilder`, renamed class to `RowSetStringBuilder` and renamed its methods. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908215#comment-16908215 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314359926 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractSingleRowSet.java ## @@ -15,15 +15,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908182#comment-16908182
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314356974
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
Review comment:
It's more common to start each chaining method call from new line:
```
testBuilder()
.sqlQuery(query)
.unOrdered()
.baselineColumns(...)
.baselineValues(...)
.go();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add User-Agent UDFs to Drill
>
>
> Key: DRILL-7343
> URL: https://issues.apache.org/jira/browse/DRILL-7343
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> This collection of UDFs adds the ability to parse user agent strings which is
> useful for security data analysis.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908179#comment-16908179
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314358047
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908180#comment-16908180
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314356394
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
+ .baselineColumns("DeviceClass", "DeviceName",
"DeviceBrand","DeviceCpuBits","OperatingSystemClass",
"OperatingSystemName","OperatingSystemVersion",
+
"OperatingSystemVersionMajor","OperatingSystemNameVersion","OperatingSystemNameVersionMajor","LayoutEngineClass","LayoutEngineName","LayoutEngineVersion",
+
"LayoutEngineVersionMajor","LayoutEngineNameVersion","LayoutEngineBuild","AgentClass","AgentName","AgentVersion","AgentVersionMajor","AgentNameVersionMajor",
+"AgentLanguage","AgentLanguageCode","AgentSecurity")
+ .baselineValues("Desktop","Desktop", "Unknown","32", "Desktop", "Windows
NT", "XP", "XP", "Windows XP", "Windows XP", "Browser", "Gecko", "1.8.1.11",
"1", "Gecko 1.8.1.11",
+"20071127", "Browser", "Firefox", "2.0.0.11", "2", "Firefox 2",
"English (United States)", "en-us","Strong security")
+ .go();
+ }
+
+ @Test
+ public void testGetHostName() throws Exception {
+String query = "SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11', 'AgentSecurity')
AS agent FROM " +
+ "(values" +
+ "(1))";
+
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908173#comment-16908173 ] ASF GitHub Bot commented on DRILL-7343: --- arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#discussion_r314354091 ## File path: contrib/udfs/README.md ## @@ -0,0 +1,56 @@ +# Drill User Defined Functions + +This `README` documents functions which users have submitted to Apaceh Drill. + +## User Agent Functions +Drill UDF for parsing User Agent Strings. +This function is based on Niels Basjes Java library for parsing user agent strings which is available here: https://github.com/nielsbasjes/yauaa. + +### Usage +Using this function is fairly simple. The function `parse_user_agent()` takes a user agent string as an argument and returns a map of the available fields. Note that not every field will be present in every user agent string. +``` +SELECT parse_user_agent( columns[0] ) as ua +FROM dfs.`/Users/cgivre/drill-httpd/ua.csv`; Review comment: Please use some generic source rather yours: `/tmp/data` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908178#comment-16908178
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314354355
##
File path: contrib/udfs/README.md
##
@@ -0,0 +1,56 @@
+# Drill User Defined Functions
+
+This `README` documents functions which users have submitted to Apaceh Drill.
+
+## User Agent Functions
+Drill UDF for parsing User Agent Strings.
+This function is based on Niels Basjes Java library for parsing user agent
strings which is available here: https://github.com/nielsbasjes/yauaa.
+
+### Usage
+Using this function is fairly simple. The function `parse_user_agent()` takes
a user agent string as an argument and returns a map of the available fields.
Note that not every field will be present in every user agent string.
+```
+SELECT parse_user_agent( columns[0] ) as ua
+FROM dfs.`/Users/cgivre/drill-httpd/ua.csv`;
+```
+The query above returns:
+```
+{
+ "DeviceClass":"Desktop",
+ "DeviceName":"Macintosh",
+ "DeviceBrand":"Apple",
+ "OperatingSystemClass":"Desktop",
+ "OperatingSystemName":"Mac OS X",
+ "OperatingSystemVersion":"10.10.1",
+ "OperatingSystemNameVersion":"Mac OS X 10.10.1",
+ "LayoutEngineClass":"Browser",
+ "LayoutEngineName":"Blink",
+ "LayoutEngineVersion":"39.0",
+ "LayoutEngineVersionMajor":"39",
+ "LayoutEngineNameVersion":"Blink 39.0",
+ "LayoutEngineNameVersionMajor":"Blink 39",
+ "AgentClass":"Browser",
+ "AgentName":"Chrome",
+ "AgentVersion":"39.0.2171.99",
+ "AgentVersionMajor":"39",
+ "AgentNameVersion":"Chrome 39.0.2171.99",
+ "AgentNameVersionMajor":"Chrome 39",
+ "DeviceCpu":"Intel"
+}
+```
+The function returns a Drill map, so you can access any of the fields using
Drill's table.map.key notation. For example, the query below illustrates how
to extract a field from this map and summarize it:
+
+```
+SELECT uadata.ua.AgentNameVersion AS Browser,
+COUNT( * ) AS BrowserCount
+FROM (
+ SELECT parse_user_agent( columns[0] ) AS ua
+ FROM dfs.drillworkshop.`user-agents.csv`
+) AS uadata
+GROUP BY uadata.ua.AgentNameVersion
+ORDER BY BrowserCount DESC
+```
+The function can also be called with an optional field as an argument. IE:
Review comment:
```suggestion
The function can also be called with an optional field as an argument. IE:
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add User-Agent UDFs to Drill
>
>
> Key: DRILL-7343
> URL: https://issues.apache.org/jira/browse/DRILL-7343
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> This collection of UDFs adds the ability to parse user agent strings which is
> useful for security data analysis.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908175#comment-16908175
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314356145
##
File path:
contrib/udfs/src/main/java/org/apache/drill/exec/udfs/UserAgentFunctions.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+import org.apache.drill.exec.vector.complex.writer.BaseWriter;
+
+import javax.inject.Inject;
+
+public class UserAgentFunctions {
+
+ @FunctionTemplate(name = "parse_user_agent", scope =
FunctionTemplate.FunctionScope.SIMPLE)
+
+ public static class UserAgentFunction implements DrillSimpleFunc {
+@Param
+VarCharHolder input;
+
+@Output
+BaseWriter.ComplexWriter outWriter;
+
+@Inject
+DrillBuf outBuffer;
+
+@Workspace
+nl.basjes.parse.useragent.UserAgentAnalyzerDirect uaa;
+
+public void setup() {
+ uaa =
nl.basjes.parse.useragent.UserAgentAnalyzerDirect.newBuilder().dropTests().hideMatcherLoadStats().build();
+ uaa.getAllPossibleFieldNamesSorted();
+}
+
+public void eval() {
+ org.apache.drill.exec.vector.complex.writer.BaseWriter.MapWriter
queryMapWriter = outWriter.rootAsMap();
+
+ String userAgentString =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(input);
+
+ if (userAgentString.isEmpty() || userAgentString.equals("null")) {
+userAgentString = "";
+ }
+
+ nl.basjes.parse.useragent.UserAgent agent = uaa.parse(userAgentString);
+
+ for (String fieldName : agent.getAvailableFieldNamesSorted()) {
+
+org.apache.drill.exec.expr.holders.VarCharHolder rowHolder = new
org.apache.drill.exec.expr.holders.VarCharHolder();
+String field = agent.getValue(fieldName);
+
+byte[] rowStringBytes = field.getBytes();
+outBuffer.reallocIfNeeded(rowStringBytes.length);
+outBuffer.setBytes(0, rowStringBytes);
+
+rowHolder.start = 0;
+rowHolder.end = rowStringBytes.length;
+rowHolder.buffer = outBuffer;
+
+queryMapWriter.varChar(fieldName).write(rowHolder);
+ }
+}
+ }
+
+ @FunctionTemplate(name = "parse_user_agent", scope =
FunctionTemplate.FunctionScope.SIMPLE)
+
+ public static class UserAgentFieldFunction implements DrillSimpleFunc {
+@Param
+VarCharHolder input;
+
+@Param
+VarCharHolder desiredField;
+
+@Output
+VarCharHolder out;
+
+@Inject
+DrillBuf outBuffer;
+
+@Workspace
+nl.basjes.parse.useragent.UserAgentAnalyzerDirect uaa;
+
+public void setup() {
+ uaa =
nl.basjes.parse.useragent.UserAgentAnalyzerDirect.newBuilder().dropTests().hideMatcherLoadStats().build();
+ uaa.getAllPossibleFieldNamesSorted();
+}
+
+public void eval() {
+ String userAgentString =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(input);
+ String requestedField =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(desiredField);
+
+ if (userAgentString.isEmpty() || userAgentString.equals("null")) {
+userAgentString = "";
+ }
+
+ nl.basjes.parse.useragent.UserAgent agent = uaa.parse(userAgentString);
+ String field = agent.getValue(requestedField);
Review comment:
What behavior if requested field is absent? It returns `Unknown`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908171#comment-16908171 ] ASF GitHub Bot commented on DRILL-7343: --- arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#discussion_r314353528 ## File path: contrib/udfs/README.md ## @@ -0,0 +1,56 @@ +# Drill User Defined Functions + +This `README` documents functions which users have submitted to Apaceh Drill. + +## User Agent Functions +Drill UDF for parsing User Agent Strings. +This function is based on Niels Basjes Java library for parsing user agent strings which is available here: https://github.com/nielsbasjes/yauaa. + +### Usage +Using this function is fairly simple. The function `parse_user_agent()` takes a user agent string as an argument and returns a map of the available fields. Note that not every field will be present in every user agent string. Review comment: ```suggestion The function `parse_user_agent()` takes a user agent string as an argument and returns a map of the available fields. Note that not every field will be present in every user agent string. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908177#comment-16908177
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314356974
##
File path:
contrib/udfs/src/test/java/org/apache/drill/exec/udfs/TestUserAgentFunctions.java
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import org.apache.drill.categories.SqlFunctionTest;
+import org.apache.drill.categories.UnlikelyTest;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({UnlikelyTest.class, SqlFunctionTest.class})
+public class TestUserAgentFunctions extends ClusterTest {
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+startCluster(builder);
+ }
+
+ @Test
+ public void testParseUserAgentString() throws Exception {
+String query = "SELECT t1.ua.DeviceClass AS DeviceClass,\n" +
+ "t1.ua.DeviceName AS DeviceName,\n" +
+ "t1.ua.DeviceBrand AS DeviceBrand,\n" +
+ "t1.ua.DeviceCpuBits AS DeviceCpuBits,\n" +
+ "t1.ua.OperatingSystemClass AS OperatingSystemClass,\n" +
+ "t1.ua.OperatingSystemName AS OperatingSystemName,\n" +
+ "t1.ua.OperatingSystemVersion AS OperatingSystemVersion,\n" +
+ "t1.ua.OperatingSystemVersionMajor AS OperatingSystemVersionMajor,\n" +
+ "t1.ua.OperatingSystemNameVersion AS OperatingSystemNameVersion,\n" +
+ "t1.ua.OperatingSystemNameVersionMajor AS
OperatingSystemNameVersionMajor,\n" +
+ "t1.ua.LayoutEngineClass AS LayoutEngineClass,\n" +
+ "t1.ua.LayoutEngineName AS LayoutEngineName,\n" +
+ "t1.ua.LayoutEngineVersion AS LayoutEngineVersion,\n" +
+ "t1.ua.LayoutEngineVersionMajor AS LayoutEngineVersionMajor,\n" +
+ "t1.ua.LayoutEngineNameVersion AS LayoutEngineNameVersion,\n" +
+ "t1.ua.LayoutEngineBuild AS LayoutEngineBuild,\n" +
+ "t1.ua.AgentClass AS AgentClass,\n" +
+ "t1.ua.AgentName AS AgentName,\n" +
+ "t1.ua.AgentVersion AS AgentVersion,\n" +
+ "t1.ua.AgentVersionMajor AS AgentVersionMajor,\n" +
+ "t1.ua.AgentNameVersionMajor AS AgentNameVersionMajor,\n" +
+ "t1.ua.AgentLanguage AS AgentLanguage,\n" +
+ "t1.ua.AgentLanguageCode AS AgentLanguageCode,\n" +
+ "t1.ua.AgentSecurity AS AgentSecurity\n" +
+ "FROM (SELECT parse_user_agent('Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') AS ua FROM (values(1)))
AS t1";
+
+testBuilder().sqlQuery(query).unOrdered()
Review comment:
It's more common to start each chasing call from new line:
```
testBuilder()
.sqlQuery(query)
.unOrdered()
.baselineColumns(...)
.baselineValues(...)
.go();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add User-Agent UDFs to Drill
>
>
> Key: DRILL-7343
> URL: https://issues.apache.org/jira/browse/DRILL-7343
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> This collection of UDFs adds the ability to parse user agent strings which is
> useful for security data analysis.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908170#comment-16908170 ] ASF GitHub Bot commented on DRILL-7343: --- arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#discussion_r314352894 ## File path: contrib/udfs/README.md ## @@ -0,0 +1,56 @@ +# Drill User Defined Functions + +This `README` documents functions which users have submitted to Apaceh Drill. Review comment: ```suggestion This `README` documents functions which users have submitted to Apache Drill. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908172#comment-16908172
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314354293
##
File path: contrib/udfs/README.md
##
@@ -0,0 +1,56 @@
+# Drill User Defined Functions
+
+This `README` documents functions which users have submitted to Apaceh Drill.
+
+## User Agent Functions
+Drill UDF for parsing User Agent Strings.
+This function is based on Niels Basjes Java library for parsing user agent
strings which is available here: https://github.com/nielsbasjes/yauaa.
+
+### Usage
+Using this function is fairly simple. The function `parse_user_agent()` takes
a user agent string as an argument and returns a map of the available fields.
Note that not every field will be present in every user agent string.
+```
+SELECT parse_user_agent( columns[0] ) as ua
+FROM dfs.`/Users/cgivre/drill-httpd/ua.csv`;
+```
+The query above returns:
+```
+{
+ "DeviceClass":"Desktop",
+ "DeviceName":"Macintosh",
+ "DeviceBrand":"Apple",
+ "OperatingSystemClass":"Desktop",
+ "OperatingSystemName":"Mac OS X",
+ "OperatingSystemVersion":"10.10.1",
+ "OperatingSystemNameVersion":"Mac OS X 10.10.1",
+ "LayoutEngineClass":"Browser",
+ "LayoutEngineName":"Blink",
+ "LayoutEngineVersion":"39.0",
+ "LayoutEngineVersionMajor":"39",
+ "LayoutEngineNameVersion":"Blink 39.0",
+ "LayoutEngineNameVersionMajor":"Blink 39",
+ "AgentClass":"Browser",
+ "AgentName":"Chrome",
+ "AgentVersion":"39.0.2171.99",
+ "AgentVersionMajor":"39",
+ "AgentNameVersion":"Chrome 39.0.2171.99",
+ "AgentNameVersionMajor":"Chrome 39",
+ "DeviceCpu":"Intel"
+}
+```
+The function returns a Drill map, so you can access any of the fields using
Drill's table.map.key notation. For example, the query below illustrates how
to extract a field from this map and summarize it:
Review comment:
```suggestion
The function returns a Drill map, so you can access any of the fields using
Drill's table.map.key notation. For example, the query below illustrates how to
extract a field from this map and summarize it:
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add User-Agent UDFs to Drill
>
>
> Key: DRILL-7343
> URL: https://issues.apache.org/jira/browse/DRILL-7343
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> This collection of UDFs adds the ability to parse user agent strings which is
> useful for security data analysis.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[
https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908176#comment-16908176
]
ASF GitHub Bot commented on DRILL-7343:
---
arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent
UDFs to Drill
URL: https://github.com/apache/drill/pull/1840#discussion_r314355115
##
File path:
contrib/udfs/src/main/java/org/apache/drill/exec/udfs/UserAgentFunctions.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.udfs;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+import org.apache.drill.exec.vector.complex.writer.BaseWriter;
+
+import javax.inject.Inject;
+
+public class UserAgentFunctions {
+
+ @FunctionTemplate(name = "parse_user_agent", scope =
FunctionTemplate.FunctionScope.SIMPLE)
+
Review comment:
Remove new line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add User-Agent UDFs to Drill
>
>
> Key: DRILL-7343
> URL: https://issues.apache.org/jira/browse/DRILL-7343
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.17.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.17.0
>
>
> This collection of UDFs adds the ability to parse user agent strings which is
> useful for security data analysis.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7343) Add User-Agent UDFs to Drill
[ https://issues.apache.org/jira/browse/DRILL-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908174#comment-16908174 ] ASF GitHub Bot commented on DRILL-7343: --- arina-ielchiieva commented on pull request #1840: DRILL-7343: Add User-Agent UDFs to Drill URL: https://github.com/apache/drill/pull/1840#discussion_r314353595 ## File path: contrib/udfs/README.md ## @@ -0,0 +1,56 @@ +# Drill User Defined Functions + +This `README` documents functions which users have submitted to Apaceh Drill. + +## User Agent Functions +Drill UDF for parsing User Agent Strings. +This function is based on Niels Basjes Java library for parsing user agent strings which is available here: https://github.com/nielsbasjes/yauaa. Review comment: ```suggestion This function is based on Niels Basjes Java library for parsing user agent strings which is available here: https://github.com/nielsbasjes/yauaa. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add User-Agent UDFs to Drill > > > Key: DRILL-7343 > URL: https://issues.apache.org/jira/browse/DRILL-7343 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.17.0 > > > This collection of UDFs adds the ability to parse user agent strings which is > useful for security data analysis. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908135#comment-16908135
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314321216
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractRowSet.java
##
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.drill.test.rowSet;
+package org.apache.drill.exec.physical.rowSet;
Review comment:
1. Remove `protected SchemaChangeCallBack callBack = new
SchemaChangeCallBack();`
2. Better error message: `public long size() {
throw new UnsupportedOperationException("getSize");
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908121#comment-16908121 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314319626 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetReader.java ## @@ -46,8 +46,7 @@ * The index of the underlying row which may be indexed by an * SV2 or SV4. * - * @return + * @return index of the underlying row */ - int offset(); } Review comment: New line. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908128#comment-16908128 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314323618 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/IndirectRowSet.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Replace usage of `Sets.newHashSet()` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908123#comment-16908123
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314321216
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractRowSet.java
##
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.drill.test.rowSet;
+package org.apache.drill.exec.physical.rowSet;
Review comment:
1. Remove `protected SchemaChangeCallBack callBack = new
SchemaChangeCallBack();`
2.Better error message: `public long size() {
throw new UnsupportedOperationException("getSize");
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908126#comment-16908126 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314320356 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetReader.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Please rename method: `void setPosn(int index);` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908120#comment-16908120 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314319854 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetPrinter.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Make sure Printer returns strings instead of sout This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908129#comment-16908129 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314324678 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/rowSet/RowSetBuilder.java ## @@ -20,12 +20,15 @@ import java.util.Set; import org.apache.drill.exec.memory.BufferAllocator; +import org.apache.drill.exec.physical.rowSet.DirectRowSet; +import org.apache.drill.exec.physical.rowSet.RowSet; +import org.apache.drill.exec.physical.rowSet.RowSetWriter; import org.apache.drill.exec.record.BatchSchema; import org.apache.drill.exec.record.metadata.MetadataUtils; import org.apache.drill.exec.record.metadata.TupleMetadata; import org.apache.drill.exec.vector.accessor.convert.ColumnConversionFactory; import org.apache.drill.shaded.guava.com.google.common.collect.Sets; -import org.apache.drill.test.rowSet.RowSet.SingleRowSet; +import org.apache.drill.exec.physical.rowSet.RowSet.SingleRowSet; Review comment: 1. Capacity here can use constant we introduce for `10`. 2. Also can be considered to be moved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908125#comment-16908125
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314322007
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java
##
@@ -33,9 +32,10 @@
import org.apache.drill.exec.record.VectorContainer;
import org.apache.drill.exec.record.selection.SelectionVector2;
import org.apache.drill.exec.vector.accessor.convert.ColumnConversionFactory;
-import org.apache.drill.test.rowSet.RowSet.ExtendableRowSet;
-import org.apache.drill.test.rowSet.RowSetWriterImpl.WriterIndexImpl;
+import org.apache.drill.exec.physical.rowSet.RowSet.ExtendableRowSet;
+import org.apache.drill.exec.physical.rowSet.RowSetWriterImpl.WriterIndexImpl;
+import java.util.HashSet;
Review comment:
`public RowSetWriter writer() {
return writer(10);
}`
Create a constant instead with Java-doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908122#comment-16908122
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314324081
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSet.java
##
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.drill.test.rowSet;
+package org.apache.drill.exec.physical.rowSet;
Review comment:
1. Remove public: `public interface SingleRowSet extends RowSet {`
2. Please try to fix all java doc errors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908132#comment-16908132 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314319854 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/RowSetPrinter.java ## @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Make sure Printer returns strings instead of sout Maybe rename it to other name since now it won't print. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[
https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908127#comment-16908127
]
ASF GitHub Bot commented on DRILL-7350:
---
arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet
related classes from test folder
URL: https://github.com/apache/drill/pull/1843#discussion_r314323421
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/DirectRowSet.java
##
@@ -132,7 +131,7 @@ public RowSetReader reader() {
@Override
public SingleRowSet toIndirect() {
-return new IndirectRowSet(this, Sets.newHashSet());
+return new IndirectRowSet(this, new HashSet<>());
Review comment:
Should be immutable?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move RowSet related classes from test folder
>
>
> Key: DRILL-7350
> URL: https://issues.apache.org/jira/browse/DRILL-7350
> Project: Apache Drill
> Issue Type: Task
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Move RowSet related classes from test folder to main to be able to use them
> for Metastore.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908124#comment-16908124 ] ASF GitHub Bot commented on DRILL-7350: --- arina-ielchiieva commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843#discussion_r314321490 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/AbstractSingleRowSet.java ## @@ -15,15 +15,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.drill.test.rowSet; +package org.apache.drill.exec.physical.rowSet; Review comment: Constructors can be protected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908095#comment-16908095 ] ASF GitHub Bot commented on DRILL-7350: --- vvysotskyi commented on pull request #1843: DRILL-7350: Move RowSet related classes from test folder URL: https://github.com/apache/drill/pull/1843 - Moved row-set related classes from test folder into main and changed package from `org.apache.drill.test.rowSet` to `org.apache.drill.exec.physical.rowSet` where other row-set related classes were placed. - Updated package info for both packages. For details please see [DRILL-7350](https://issues.apache.org/jira/browse/DRILL-7350). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7350) Move RowSet related classes from test folder
[ https://issues.apache.org/jira/browse/DRILL-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7350: Reviewer: Arina Ielchiieva > Move RowSet related classes from test folder > > > Key: DRILL-7350 > URL: https://issues.apache.org/jira/browse/DRILL-7350 > Project: Apache Drill > Issue Type: Task >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > Move RowSet related classes from test folder to main to be able to use them > for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DRILL-7350) Move RowSet related classes from test folder
Volodymyr Vysotskyi created DRILL-7350: -- Summary: Move RowSet related classes from test folder Key: DRILL-7350 URL: https://issues.apache.org/jira/browse/DRILL-7350 Project: Apache Drill Issue Type: Task Reporter: Volodymyr Vysotskyi Assignee: Volodymyr Vysotskyi Fix For: 1.17.0 Move RowSet related classes from test folder to main to be able to use them for Metastore. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7156) Empty Parquet is not getting created if 0 records in result
[ https://issues.apache.org/jira/browse/DRILL-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7156: Labels: ready-to-commit (was: ) > Empty Parquet is not getting created if 0 records in result > --- > > Key: DRILL-7156 > URL: https://issues.apache.org/jira/browse/DRILL-7156 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.16.0 >Reporter: Sayalee Bhanavase >Assignee: Oleg Zinoviev >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > I am creating parquet tables out of joins. If there is no record in join, it > does not create empty. table and when I reused the table my further script > fails. > Has anyone faced this issue? Any suggestion or workaround? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7156) Empty Parquet is not getting created if 0 records in result
[ https://issues.apache.org/jira/browse/DRILL-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7156: Issue Type: Improvement (was: Bug) > Empty Parquet is not getting created if 0 records in result > --- > > Key: DRILL-7156 > URL: https://issues.apache.org/jira/browse/DRILL-7156 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Affects Versions: 1.16.0 >Reporter: Sayalee Bhanavase >Assignee: Oleg Zinoviev >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > I am creating parquet tables out of joins. If there is no record in join, it > does not create empty. table and when I reused the table my further script > fails. > Has anyone faced this issue? Any suggestion or workaround? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7156) Empty Parquet is not getting created if 0 records in result
[ https://issues.apache.org/jira/browse/DRILL-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908018#comment-16908018 ] ASF GitHub Bot commented on DRILL-7156: --- arina-ielchiieva commented on issue #1836: DRILL-7156: Support empty Parquet files creation URL: https://github.com/apache/drill/pull/1836#issuecomment-521618404 Ran all tests on the test cluster, all passed. LGTM, +1 @oleg-zinovev thanks for making the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Empty Parquet is not getting created if 0 records in result > --- > > Key: DRILL-7156 > URL: https://issues.apache.org/jira/browse/DRILL-7156 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.16.0 >Reporter: Sayalee Bhanavase >Assignee: Oleg Zinoviev >Priority: Major > Fix For: 1.17.0 > > > I am creating parquet tables out of joins. If there is no record in join, it > does not create empty. table and when I reused the table my further script > fails. > Has anyone faced this issue? Any suggestion or workaround? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7348) Aggregate on Subquery with Select Distinct or UNION fails to Group By
[
https://issues.apache.org/jira/browse/DRILL-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907999#comment-16907999
]
Arina Ielchiieva commented on DRILL-7348:
-
[~snapdoodle] please provide dataset and actual and expected results.
> Aggregate on Subquery with Select Distinct or UNION fails to Group By
> -
>
> Key: DRILL-7348
> URL: https://issues.apache.org/jira/browse/DRILL-7348
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
>Reporter: Keith G Yu
>Priority: Major
>
> The following query fails to group properly.
> {code:java}
> SELECT date, COUNT(1)
> FROM (
> SELECT DISTINCT
> id,
> date,
> status
> FROM table(dfs.`path`(type => 'text', fieldDelimiter => ',',
> extractHeader => TRUE))
> )
> GROUP BY 1{code}
> This also fails to group properly.
> {code:java}
> SELECT date, COUNT(1)
> FROM (
> SELECT
> id,
> date,
> status
> FROM table(dfs.`path1`(type => 'text', fieldDelimiter => ',',
> extractHeader => TRUE))
> UNION
> SELECT
> id,
> date,
> status
> FROM table(dfs.`path2`(type => 'text', fieldDelimiter => ',',
> extractHeader => TRUE))
> )
> GROUP BY 1
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7349) ClusterTest dfs.tmp storage inconsistence with cluster size > 1
[
https://issues.apache.org/jira/browse/DRILL-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7349:
Affects Version/s: (was: 1.17.0)
1.16.0
> ClusterTest dfs.tmp storage inconsistence with cluster size > 1
> ---
>
> Key: DRILL-7349
> URL: https://issues.apache.org/jira/browse/DRILL-7349
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build Test
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Priority: Minor
>
> ClusterTest#dirTestWatcher works inconsistent if cluster size is greater when
> 1.
> For example, test
> {code:java}
> public class TestClusterTest extends ClusterTest {
> @BeforeClass
> public static void setUp() throws Exception {
> startCluster(ClusterFixture.builder(dirTestWatcher).clusterSize(2));
> }
> @Test
> public void testWriteFile() throws Exception {
> final String outputFileName = "some_file_name";
> final File outputFile =
> FileUtils.getFile(dirTestWatcher.getDfsTestTmpDir(), outputFileName);
> queryBuilder().sql("CREATE TABLE dfs.tmp.%s AS SELECT * FROM
> cp.`employee.json` limit 1", outputFileName).run();
> assertTrue(outputFile.exists());
> }
> }
> {code}
> may fail on assertion.
> Reason: several directories for the storage `dfs.tmp` (possibly 1 for each
> node) was created, and dirTestWatcher.getDfsTestTmpDir () may return the
> wrong one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7349) ClusterTest dfs.tmp storage inconsistence with cluster size > 1
[
https://issues.apache.org/jira/browse/DRILL-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zinoviev updated DRILL-7349:
-
Description:
ClusterTest#dirTestWatcher works inconsistent if cluster size is greater when 1.
For example, test
{code:java}
public class TestClusterTest extends ClusterTest {
@BeforeClass
public static void setUp() throws Exception {
startCluster(ClusterFixture.builder(dirTestWatcher).clusterSize(2));
}
@Test
public void testWriteFile() throws Exception {
final String outputFileName = "some_file_name";
final File outputFile =
FileUtils.getFile(dirTestWatcher.getDfsTestTmpDir(), outputFileName);
queryBuilder().sql("CREATE TABLE dfs.tmp.%s AS SELECT * FROM
cp.`employee.json` limit 1", outputFileName).run();
assertTrue(outputFile.exists());
}
}
{code}
may fail on assertion.
Reason: several directories for the storage `dfs.tmp` (possibly 1 for each
node) was created, and dirTestWatcher.getDfsTestTmpDir () may return the wrong
one.
was:
ClusterTest#dirTestWatcher works inconsistent if cluster size is greater when 1.
For example, test
{code:java}
public class TestClusterTest extends ClusterTest {
@BeforeClass
public static void setUp() throws Exception {
startCluster(ClusterFixture.builder(dirTestWatcher).clusterSize(2));
}
@Test
public void testWriteFile() throws Exception {
final String outputFileName =
"testparquetwriteremptyfiles_testwriteemptyfile";
final File outputFile =
FileUtils.getFile(dirTestWatcher.getDfsTestTmpDir(), outputFileName);
queryBuilder().sql("CREATE TABLE dfs.tmp.%s AS SELECT * FROM
cp.`employee.json` limit 1", outputFileName).run();
assertTrue(outputFile.exists());
}
}
{code}
may fail on assertion.
Reason: several directories for the storage `dfs.tmp` (possibly 1 for each
node) was created, and dirTestWatcher.getDfsTestTmpDir () may return the wrong
one.
> ClusterTest dfs.tmp storage inconsistence with cluster size > 1
> ---
>
> Key: DRILL-7349
> URL: https://issues.apache.org/jira/browse/DRILL-7349
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build Test
>Affects Versions: 1.17.0
>Reporter: Oleg Zinoviev
>Priority: Minor
>
> ClusterTest#dirTestWatcher works inconsistent if cluster size is greater when
> 1.
> For example, test
> {code:java}
> public class TestClusterTest extends ClusterTest {
> @BeforeClass
> public static void setUp() throws Exception {
> startCluster(ClusterFixture.builder(dirTestWatcher).clusterSize(2));
> }
> @Test
> public void testWriteFile() throws Exception {
> final String outputFileName = "some_file_name";
> final File outputFile =
> FileUtils.getFile(dirTestWatcher.getDfsTestTmpDir(), outputFileName);
> queryBuilder().sql("CREATE TABLE dfs.tmp.%s AS SELECT * FROM
> cp.`employee.json` limit 1", outputFileName).run();
> assertTrue(outputFile.exists());
> }
> }
> {code}
> may fail on assertion.
> Reason: several directories for the storage `dfs.tmp` (possibly 1 for each
> node) was created, and dirTestWatcher.getDfsTestTmpDir () may return the
> wrong one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DRILL-7349) ClusterTest dfs.tmp storage inconsistence with cluster size > 1
Oleg Zinoviev created DRILL-7349:
Summary: ClusterTest dfs.tmp storage inconsistence with cluster
size > 1
Key: DRILL-7349
URL: https://issues.apache.org/jira/browse/DRILL-7349
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build Test
Affects Versions: 1.17.0
Reporter: Oleg Zinoviev
ClusterTest#dirTestWatcher works inconsistent if cluster size is greater when 1.
For example, test
{code:java}
public class TestClusterTest extends ClusterTest {
@BeforeClass
public static void setUp() throws Exception {
startCluster(ClusterFixture.builder(dirTestWatcher).clusterSize(2));
}
@Test
public void testWriteFile() throws Exception {
final String outputFileName =
"testparquetwriteremptyfiles_testwriteemptyfile";
final File outputFile =
FileUtils.getFile(dirTestWatcher.getDfsTestTmpDir(), outputFileName);
queryBuilder().sql("CREATE TABLE dfs.tmp.%s AS SELECT * FROM
cp.`employee.json` limit 1", outputFileName).run();
assertTrue(outputFile.exists());
}
}
{code}
may fail on assertion.
Reason: several directories for the storage `dfs.tmp` (possibly 1 for each
node) was created, and dirTestWatcher.getDfsTestTmpDir () may return the wrong
one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7156) Empty Parquet is not getting created if 0 records in result
[ https://issues.apache.org/jira/browse/DRILL-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907960#comment-16907960 ] ASF GitHub Bot commented on DRILL-7156: --- oleg-zinovev commented on issue #1836: DRILL-7156: Support empty Parquet files creation URL: https://github.com/apache/drill/pull/1836#issuecomment-521581932 @arina-ielchiieva , thanks for review. Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Empty Parquet is not getting created if 0 records in result > --- > > Key: DRILL-7156 > URL: https://issues.apache.org/jira/browse/DRILL-7156 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.16.0 >Reporter: Sayalee Bhanavase >Assignee: Oleg Zinoviev >Priority: Major > Fix For: 1.17.0 > > > I am creating parquet tables out of joins. If there is no record in join, it > does not create empty. table and when I reused the table my further script > fails. > Has anyone faced this issue? Any suggestion or workaround? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DRILL-7339) Upgrade to Iceberg latest commits to fix issue with orphan files after delete in transaction
[ https://issues.apache.org/jira/browse/DRILL-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7339: --- Labels: ready-to-commit (was: ) > Upgrade to Iceberg latest commits to fix issue with orphan files after delete > in transaction > > > Key: DRILL-7339 > URL: https://issues.apache.org/jira/browse/DRILL-7339 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Drill Metastore executes many operations in transaction including delete. > Currently Iceberg creates orphan files when executing delete in transaction > and these files cannot be expired and keep pilling up. Iceberg issue - > https://github.com/apache/incubator-iceberg/issues/330. When #330 is fixed, > we need to update Iceberg commit to ensure these issue is resolved in Drill > as well. > PR with the fix - https://github.com/apache/incubator-iceberg/pull/352 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7339) Upgrade to Iceberg latest commits to fix issue with orphan files after delete in transaction
[ https://issues.apache.org/jira/browse/DRILL-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907903#comment-16907903 ] ASF GitHub Bot commented on DRILL-7339: --- arina-ielchiieva commented on pull request #1842: DRILL-7339: Iceberg commit upgrade and Metastore tests categorization URL: https://github.com/apache/drill/pull/1842 1. Upgraded Iceberg commit to fix issue with deletes in transaction 2. Categorize Metastore tests Jira - [DRILL-7339](https://issues.apache.org/jira/browse/DRILL-7339). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Upgrade to Iceberg latest commits to fix issue with orphan files after delete > in transaction > > > Key: DRILL-7339 > URL: https://issues.apache.org/jira/browse/DRILL-7339 > Project: Apache Drill > Issue Type: Task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.17.0 > > > Drill Metastore executes many operations in transaction including delete. > Currently Iceberg creates orphan files when executing delete in transaction > and these files cannot be expired and keep pilling up. Iceberg issue - > https://github.com/apache/incubator-iceberg/issues/330. When #330 is fixed, > we need to update Iceberg commit to ensure these issue is resolved in Drill > as well. > PR with the fix - https://github.com/apache/incubator-iceberg/pull/352 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
