date:20170808

[jira] [Created] (DRILL-5710) drill-config.sh incorrectly exits with Java 1.7 or later is required to run Apache Dril

2017-08-08 Thread Angel Aray (JIRA)

Angel Aray created DRILL-5710:
-

 Summary: drill-config.sh incorrectly exits with Java 1.7 or later 
is required to run Apache Dril
 Key: DRILL-5710
 URL: https://issues.apache.org/jira/browse/DRILL-5710
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
 Environment: java version "1.8.0_144"
OSX
Reporter: Angel Aray


drill-config fails to recognize 1.8.0_144 as Java 1.7 or later.

The scripts validates the java version using the following code:
"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" 

this should be replaced by:

"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5709) Provide a value vector method to convert a vector to nullable

2017-08-08 Thread Paul Rogers (JIRA)

Paul Rogers created DRILL-5709:
--

 Summary: Provide a value vector method to convert a vector to 
nullable
 Key: DRILL-5709
 URL: https://issues.apache.org/jira/browse/DRILL-5709
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.12.0


The hash agg spill work has need to convert a non-null scalar vector to the 
nullable equivalent. For efficiency, the code wishes to simply transfer the 
underlying data buffer(s), and create the required "bits" vector, rather than 
generating code that does the transfer row-by-row.

The solution is to add a {{toNullable(ValueVector nullableVector)}} method to 
the {{ValueVector}} class, then implement it where needed.

Since the target code only works with scalars (that is, no arrays, no maps, no 
lists), the code only handles these cases, throwing an 
{{UnsupportedOperationException}} in other cases.

Usage:

{code}
ValueVector nonNullableVector = // your non-nullable vector
MajorType type = MajorType.newBuilder(nonNullableVector.getType)
.setMode(DataMode.OPTIONAL)
.build();
MaterializedField field = MaterializedField.create(name, type);
ValueVector nullableVector = TypeHelper.getNewVector(field, 
oContext.getAllocator());
nonNullableVector.toNullable(nullableVector);
// Data is now in nullableVector
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r130429208
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/TupleSchema.java
 ---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.physical.rowSet.impl.MaterializedSchema;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.MaterializedField;
+
+/**
+ * Defines the schema of a tuple: either the top-level row or a nested
+ * "map" (really structure). A schema is a collection of columns (backed
+ * by vectors in the loader itself.) Columns are accessible by name or
+ * index. New columns may be added at any time; the new column takes the
+ * next available index.
+ */
+
+public interface TupleSchema {
+
+  public interface TupleColumnSchema {
+MaterializedField schema();
+
+/**
+ * Report if a column is selected.
+ * @param colIndex index of the column to check
+ * @return true if the column is selected (data is collected),
+ * false if the column is unselected (data is discarded)
+ */
+
+boolean isSelected();
--- End diff --

What does it mean for a  column to be selected? Selected in the query?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131564509
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/ResultVectorCache.java
 ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.vector.ValueVector;
+
+/**
+ * Manages an inventory of value vectors used across row batch readers.
+ * Drill semantics for batches is complex. Each operator logically returns
+ * a batch of records on each call of the Drill Volcano iterator protocol
+ * next() operation. However, the batches "returned" are not
+ * separate objects. Instead, Drill enforces the following semantics:
+ * 
+ * If a next() call returns OK then the set of 
vectors
+ * in the "returned" batch must be identical to those in the prior batch. 
Not
+ * just the same type; they must be the same ValueVector objects.
+ * (The buffers within the vectors will be different.)
+ * If the set of vectors changes in any way (add a vector, remove a
+ * vector, change the type of a vector), then the next() call
+ * must return OK_NEW_SCHEMA.
+ * 
+ * These rules create interesting constraints for the scan operator.
+ * Conceptually, each batch is distinct. But, it must share vectors. The
+ * {@link ResultSetLoader} class handles this by managing the set of 
vectors
+ * used by a single reader.
+ * 
+ * Readers are independent: each may read a distinct schema (as in JSON.)
+ * Yet, the Drill protocol requires minimizing spurious 
OK_NEW_SCHEMA
+ * events. As a result, two readers run by the same scan operator must
+ * share the same set of vectors, despite the fact that they may have
+ * different schemas and thus different ResultSetLoaders.
+ * 
+ * The purpose of this inventory is to persist vectors across readers, even
+ * when, say, reader B does not use a vector that reader A created.
+ * 
+ * The semantics supported by this class include:
+ * 
+ * Ability to "pre-declare" columns based on columns that appear in
+ * an explicit select list. This ensures that the columns are known (but
+ * not their types).
+ * Ability to reuse a vector across readers if the column retains the 
same
+ * name and type (minor type and mode.)
+ * Ability to flush unused vectors for readers with changing schemas
+ * if a schema change occurs.
+ * Support schema "hysteresis"; that is, the a "sticky" schema that
+ * minimizes spurious changes. Once a vector is declared, it can be 
included
+ * in all subsequent batches (provided the column is nullable or an 
array.)
+ * 
+ */
+public class ResultVectorCache {
+
+  /**
+   * State of a projected vector. At first all we have is a name.
+   * Later, we'll discover the type.
+   */
+
+  private static class VectorState {
+protected final String name;
+protected ValueVector vector;
+protected boolean touched;
+
+public VectorState(String name) {
+  this.name = name;
+}
+
+public boolean satisfies(MaterializedField colSchema) {
+  if (vector == null) {
+return false;
+  }
+  MaterializedField vectorSchema = vector.getField();
+  return vectorSchema.getType().equals(colSchema.getType());
+}
+  }
+
+  private final BufferAllocator allocator;
+  private final Map vectors = new HashMap<>();
+
+  public ResultVectorCache(BufferAllocator allocator) {
+this.allocator =

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131284158
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/LogicalTupleLoader.java
 ---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.exec.physical.rowSet.ColumnLoader;
+import org.apache.drill.exec.physical.rowSet.TupleLoader;
+import org.apache.drill.exec.physical.rowSet.TupleSchema;
+import org.apache.drill.exec.physical.rowSet.TupleSchema.TupleColumnSchema;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.MaterializedField;
+
+/**
+ * Shim inserted between an actual tuple loader and the client to remove 
columns
+ * that are not projected from input to output. The underlying loader 
handles only
+ * the projected columns in order to improve efficiency. This class 
presents the
+ * full table schema, but returns null for the non-projected columns. This 
allows
+ * the reader to work with the table schema as defined by the data source, 
but
+ * skip those columns which are not projected. Skipping non-projected 
columns avoids
+ * creating value vectors which are immediately discarded. It may also 
save the reader
+ * from reading unwanted data.
+ */
+public class LogicalTupleLoader implements TupleLoader {
+
+  public static final int UNMAPPED = -1;
+
+  private static class MappedColumn implements TupleColumnSchema {
+
+private final MaterializedField schema;
+private final int mapping;
+
+public MappedColumn(MaterializedField schema, int mapping) {
+  this.schema = schema;
+  this.mapping = mapping;
+}
+
+@Override
+public MaterializedField schema() { return schema; }
+
+@Override
+public boolean isSelected() { return mapping != UNMAPPED; }
+
+@Override
+public int vectorIndex() { return mapping; }
+  }
+
+  /**
+   * Implementation of the tuple schema that describes the full data source
+   * schema. The underlying loader schema is a subset of these columns. 
Note
+   * that the columns appear in the same order in both schemas, but the 
loader
+   * schema is a subset of the table schema.
+   */
+
+  private class LogicalTupleSchema implements TupleSchema {
+
+private final Set selection = new HashSet<>();
+private final TupleSchema physicalSchema;
+
+private LogicalTupleSchema(TupleSchema physicalSchema, 
Collection selection) {
+  this.physicalSchema = physicalSchema;
+  this.selection.addAll(selection);
+}
+
+@Override
+public int columnCount() { return logicalSchema.count(); }
+
+@Override
+public int columnIndex(String colName) {
+  return logicalSchema.indexOf(rsLoader.toKey(colName));
+}
+
+@Override
+public TupleColumnSchema metadata(int colIndex) { return 
logicalSchema.get(colIndex); }
+
+@Override
+public MaterializedField column(int colIndex) { return 
logicalSchema.get(colIndex).schema(); }
+
+@Override
+public TupleColumnSchema metadata(String colName) { return 
logicalSchema.get(colName); }
+
+@Override
+public MaterializedField column(String colName) { return 
logicalSchema.get(colName).schema(); }
+
+@Override
+public int addColumn(MaterializedField columnSchema) {
+  String key = rsLoader.toKey(columnSchema.getName());
+  int pIndex;
+  if (selection.contains(key)) {
--- End diff --

selection is already normalized if caseSensitive is false ?

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131216670
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/ColumnLoaderImpl.java
 ---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import org.apache.drill.exec.physical.rowSet.ColumnLoader;
+
+/**
+ * Implementation interface for a column loader. Adds to the public 
interface
+ * a number of methods needed to coordinate batch overflow.
+ */
+
+public interface ColumnLoaderImpl extends ColumnLoader {
--- End diff --

"Impl"  in an interface name sounds odd.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r130250994
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/ResultSetLoader.java
 ---
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet;
+
+import org.apache.drill.exec.record.VectorContainer;
+
+/**
+ * Builds a result set (series of zero or more row sets) based on a defined
+ * schema which may
+ * evolve (expand) over time. Automatically rolls "overflow" rows over
+ * when a batch fills.
+ * 
+ * Many of the methods in this interface are verify that the loader is
--- End diff --

"to verify"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131554894
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/ResultSetLoaderImpl.java
 ---
@@ -0,0 +1,412 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.Collection;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.physical.rowSet.ResultSetLoader;
+import org.apache.drill.exec.physical.rowSet.TupleLoader;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.vector.ValueVector;
+
+/**
+ * Implementation of the result set loader.
+ * @see {@link ResultSetLoader}
+ */
+
+public class ResultSetLoaderImpl implements ResultSetLoader, 
WriterIndexImpl.WriterIndexListener {
+
+  public static class ResultSetOptions {
+public final int vectorSizeLimit;
+public final int rowCountLimit;
+public final boolean caseSensitive;
+public final ResultVectorCache inventory;
+private final Collection selection;
+
+public ResultSetOptions() {
+  vectorSizeLimit = ValueVector.MAX_BUFFER_SIZE;
+  rowCountLimit = ValueVector.MAX_ROW_COUNT;
+  caseSensitive = false;
+  selection = null;
+  inventory = null;
+}
+
+public ResultSetOptions(OptionBuilder builder) {
+  this.vectorSizeLimit = builder.vectorSizeLimit;
+  this.rowCountLimit = builder.rowCountLimit;
+  this.caseSensitive = builder.caseSensitive;
+  this.selection = builder.selection;
+  this.inventory = builder.inventory;
+}
+  }
+
+  public static class OptionBuilder {
+private int vectorSizeLimit;
+private int rowCountLimit;
+private boolean caseSensitive;
+private Collection selection;
+private ResultVectorCache inventory;
+
+public OptionBuilder() {
+  ResultSetOptions options = new ResultSetOptions();
+  vectorSizeLimit = options.vectorSizeLimit;
+  rowCountLimit = options.rowCountLimit;
+  caseSensitive = options.caseSensitive;
+}
+
+public OptionBuilder setCaseSensitive(boolean flag) {
+  caseSensitive = flag;
+  return this;
+}
+
+public OptionBuilder setRowCountLimit(int limit) {
+  rowCountLimit = Math.min(limit, ValueVector.MAX_ROW_COUNT);
+  return this;
+}
+
+public OptionBuilder setSelection(Collection selection) {
+  this.selection = selection;
+  return this;
+}
+
+public OptionBuilder setVectorCache(ResultVectorCache inventory) {
+  this.inventory = inventory;
+  return this;
+}
+
+// TODO: No setter for vector length yet: is hard-coded
+// at present in the value vector.
+
+public ResultSetOptions build() {
+  return new ResultSetOptions(this);
+}
+  }
+
+  public static class VectorContainerBuilder {
+private final ResultSetLoaderImpl rowSetMutator;
+private int lastUpdateVersion = -1;
+private VectorContainer container;
+
+public VectorContainerBuilder(ResultSetLoaderImpl rowSetMutator) {
+  this.rowSetMutator = rowSetMutator;
+  container = new VectorContainer(rowSetMutator.allocator);
+}
+
+public void update() {
+  if (lastUpdateVersion < rowSetMutator.schemaVersion()) {
+rowSetMutator.rootTuple.buildContainer(this);
+container.buildSchema(SelectionVectorMode.NONE);
+lastUpdateVersion = rowSetMutator.schemaVersion();
+  }
+}
+
+public VectorContainer container() { return container; }
+
+public int

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131459203
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/ResultSetLoaderImpl.java
 ---
@@ -0,0 +1,412 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.Collection;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.physical.rowSet.ResultSetLoader;
+import org.apache.drill.exec.physical.rowSet.TupleLoader;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.vector.ValueVector;
+
+/**
+ * Implementation of the result set loader.
+ * @see {@link ResultSetLoader}
+ */
+
+public class ResultSetLoaderImpl implements ResultSetLoader, 
WriterIndexImpl.WriterIndexListener {
+
+  public static class ResultSetOptions {
+public final int vectorSizeLimit;
+public final int rowCountLimit;
+public final boolean caseSensitive;
+public final ResultVectorCache inventory;
--- End diff --

The name 'inventory' does not convey the intent clearly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131684173
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/TupleSetImpl.java
 ---
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.physical.rowSet.ColumnLoader;
+import org.apache.drill.exec.physical.rowSet.TupleLoader;
+import org.apache.drill.exec.physical.rowSet.TupleSchema;
+import 
org.apache.drill.exec.physical.rowSet.impl.ResultSetLoaderImpl.VectorContainerBuilder;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.VectorOverflowException;
+import org.apache.drill.exec.vector.accessor.impl.AbstractColumnWriter;
+import org.apache.drill.exec.vector.accessor.impl.ColumnAccessorFactory;
+
+/**
+ * Implementation of a column when creating a row batch.
+ * Every column resides at an index, is defined by a schema,
+ * is backed by a value vector, and and is written to by a writer.
+ * Each column also tracks the schema version in which it was added
+ * to detect schema evolution. Each column has an optional overflow
+ * vector that holds overflow record values when a batch becomes
+ * full.
+ * 
+ * Overflow vectors require special consideration. The vector class itself
+ * must remain constant as it is bound to the writer. To handle overflow,
+ * the implementation must replace the buffer in the vector with a new
+ * one, saving the full vector to return as part of the final row batch.
+ * This puts the column in one of three states:
+ * 
+ * Normal: only one vector is of concern - the vector for the active
+ * row batch.
+ * Overflow: a write to a vector caused overflow. For all columns,
+ * the data buffer is shifted to a harvested vector, and a new, empty
+ * buffer is put into the active vector.
+ * Excess: a (small) column received values for the row that will
+ * overflow due to a later column. When overflow occurs, the excess
+ * column value, from the overflow record, resides in the active
+ * vector. It must be shifted from the active vector into the new
+ * overflow buffer.
+ */
+
+public class TupleSetImpl implements TupleSchema {
+
+  public static class TupleLoaderImpl implements TupleLoader {
+
+public TupleSetImpl tupleSet;
+
+public TupleLoaderImpl(TupleSetImpl tupleSet) {
+  this.tupleSet = tupleSet;
+}
+
+@Override
+public TupleSchema schema() { return tupleSet; }
+
+@Override
+public ColumnLoader column(int colIndex) {
+  // TODO: Cache loaders here
+  return tupleSet.columnImpl(colIndex).writer;
+}
+
+@Override
+public ColumnLoader column(String colName) {
+  ColumnImpl col = tupleSet.columnImpl(colName);
+  if (col == null) {
+throw new UndefinedColumnException(colName);
+  }
+  return col.writer;
+}
+
+@Override
+public TupleLoader loadRow(Object... values) {
--- End diff --

Is there a need to verify the types of the incoming args?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

2017-08-08 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/866#discussion_r131685349
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/rowSet/impl/TupleSetImpl.java
 ---
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.rowSet.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.physical.rowSet.ColumnLoader;
+import org.apache.drill.exec.physical.rowSet.TupleLoader;
+import org.apache.drill.exec.physical.rowSet.TupleSchema;
+import 
org.apache.drill.exec.physical.rowSet.impl.ResultSetLoaderImpl.VectorContainerBuilder;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.VectorOverflowException;
+import org.apache.drill.exec.vector.accessor.impl.AbstractColumnWriter;
+import org.apache.drill.exec.vector.accessor.impl.ColumnAccessorFactory;
+
+/**
+ * Implementation of a column when creating a row batch.
+ * Every column resides at an index, is defined by a schema,
+ * is backed by a value vector, and and is written to by a writer.
+ * Each column also tracks the schema version in which it was added
+ * to detect schema evolution. Each column has an optional overflow
+ * vector that holds overflow record values when a batch becomes
+ * full.
+ * 
+ * Overflow vectors require special consideration. The vector class itself
+ * must remain constant as it is bound to the writer. To handle overflow,
+ * the implementation must replace the buffer in the vector with a new
+ * one, saving the full vector to return as part of the final row batch.
+ * This puts the column in one of three states:
+ * 
+ * Normal: only one vector is of concern - the vector for the active
+ * row batch.
+ * Overflow: a write to a vector caused overflow. For all columns,
+ * the data buffer is shifted to a harvested vector, and a new, empty
+ * buffer is put into the active vector.
+ * Excess: a (small) column received values for the row that will
--- End diff --

'Excess' is the LOOK_AHEAD state, correct? I think it would be better if 
the comments use the same terminology as in the code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: First Drill Ticket

2017-08-08 Thread Timothy Farkas

Hi Chunhui,

My github id is ilooner-mapr
My issues.apache.org id is timothyfarkas

Thanks,
Tim


From: Chunhui Shi 
Sent: Tuesday, August 8, 2017 3:38:53 PM
To: dev@drill.apache.org
Subject: Re: First Drill Ticket

What is your github account?

From: Timothy Farkas 
Sent: Tuesday, August 8, 2017 3:31:13 PM
To: dev@drill.apache.org
Subject: First Drill Ticket

Hello All,

I'm getting started on my first newbie drill ticket 
https://issues.apache.org/jira/browse/DRILL-4211 . If I should be looking at a 
different ticket to do instead please let me know! Also how can I assign the 
ticket to myself? I seem to be blocked from doing so even though I already have 
an account on issues.apache.org.

Thanks,
Tim

Re: First Drill Ticket

2017-08-08 Thread Chunhui Shi

What is your github account?

From: Timothy Farkas 
Sent: Tuesday, August 8, 2017 3:31:13 PM
To: dev@drill.apache.org
Subject: First Drill Ticket

Hello All,

I'm getting started on my first newbie drill ticket 
https://issues.apache.org/jira/browse/DRILL-4211 . If I should be looking at a 
different ticket to do instead please let me know! Also how can I assign the 
ticket to myself? I seem to be blocked from doing so even though I already have 
an account on issues.apache.org.

Thanks,
Tim

First Drill Ticket

2017-08-08 Thread Timothy Farkas

Hello All,

I'm getting started on my first newbie drill ticket 
https://issues.apache.org/jira/browse/DRILL-4211 . If I should be looking at a 
different ticket to do instead please let me know! Also how can I assign the 
ticket to myself? I seem to be blocked from doing so even though I already have 
an account on issues.apache.org.

Thanks,
Tim

[GitHub] drill issue #814: Add clickable localhost URL

2017-08-08 Thread bbevens

Github user bbevens commented on the issue:

https://github.com/apache/drill/pull/814
  
Thank you for the feedback. I've updated the page based on the request in 
this PR. 
~Bridget



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #776: DRILL-5165: limitRel to return correct rows for lim...

2017-08-08 Thread chunhui-shi

Github user chunhui-shi closed the pull request at:

https://github.com/apache/drill/pull/776


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IntelliJ code format

2017-08-08 Thread Padma Penumarthy

You are right. It is configured as 4. We should fix that. 

Thanks,
Padma


> On Aug 8, 2017, at 8:12 AM, weijie tong  wrote:
> 
> the download  site url :
> https://drill.apache.org/docs/apache-drill-contribution-guidelines/
> 
> On Tue, Aug 8, 2017 at 10:59 PM, weijie tong 
> wrote:
> 
>> The IntelliJ code format downloaded from the Drill web site seems to have
>> 4 indent .The eclipse one is 2 indent. Wonder ti's my personal env problem
>> or the provided Intellij code format error?
>>

Re: IntelliJ code format

2017-08-08 Thread weijie tong

the download  site url :
https://drill.apache.org/docs/apache-drill-contribution-guidelines/

On Tue, Aug 8, 2017 at 10:59 PM, weijie tong 
wrote:

> The IntelliJ code format downloaded from the Drill web site seems to have
> 4 indent .The eclipse one is 2 indent. Wonder ti's my personal env problem
> or the provided Intellij code format error?
>

[GitHub] drill issue #889: DRILL-5691: enhance scalar sub queries checking for the ca...

2017-08-08 Thread weijietong

Github user weijietong commented on the issue:

https://github.com/apache/drill/pull/889
  
@arina-ielchiieva  your test case can not reproduce the error . You can 
search the dev email to find the origin error description with the keyword 
"Drill query planning error".  Your query already satisfy the 
NestedLoopJoinPrule. My case is that I add another rule to change the 
Aggregate-->Aggregate-->Scan to Scan as the transformed Scan relnode already 
holding the count(distinct ) value. When this transformation occurs, the 
NestedLoopJoinPrule's checkPreconditions method will invoke 
JoinUtils.hasScalarSubqueryInput. Then it will fail, as the transformed relnode 
has no aggregate node which does not satisfy the current scalar rule. 

I think it's hard to reproduce this error without a specific rule like what 
I do. the precondition is:
1. a nested loop join
2. no (aggregate--> aggregate) count distinct relation nodes in the plan
3. the row number of one child of the nested loop join is 1 .

I wonder if the enhanced code does not break the current unit test ,it will 
be ok.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IntelliJ code format

2017-08-08 Thread weijie tong

The IntelliJ code format downloaded from the Drill web site seems to have 4
indent .The eclipse one is 2 indent. Wonder ti's my personal env problem or
the provided Intellij code format error?

Re: compiling drill 1.11.0 with cdh profile

2017-08-08 Thread yuliya Feldman

Feels like you can't access: 
https://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom
as no other repo contain that pom.

  From: Dor Ben Dov 
 To: "dev@drill.apache.org"  
 Sent: Tuesday, August 8, 2017 1:12 AM
 Subject: RE: compiling drill 1.11.0 with cdh profile

 Can one help me with this ? 

Dor

-Original Message-
From: Dor Ben Dov 
Sent: יום ב 07 אוגוסט 2017 13:44
To: dev@drill.apache.org
Subject: compiling drill 1.11.0 with cdh profile

Hi all,

Tried to compile source code of branch 1.11.0 with profile cdh for cloudera - 
getting this exception, anyone ? 

 [dor@dor-fedora64 drill]$ mvn -U -DskipTests clean install -Pcdh [INFO] 
Scanning for projects...
Downloading: 
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom
Downloading: http://conjars.org/repo/org/apache/apache/14/apache-14.pom
Downloading: http://repository.mapr.com/maven/org/apache/apache/14/apache-14.pom
Downloading: http://repo.dremio.com/release/org/apache/apache/14/apache-14.pom
Downloading: 
http://repository.mapr.com/nexus/content/repositories/drill/org/apache/apache/14/apache-14.pom
Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: Could 
not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11  @ [ERROR] The build could not 
read 1 project -> [Help 1]
[ERROR]  
[ERROR]  The project org.apache.drill:drill-root:1.11.0 
(/home/dor/Downloads/drill/pom.xml) has 1 error
[ERROR]    Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: 
Could not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11: Unknown host 
repository.cloudera.com: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[dor@dor-fedora64 drill]$

** I am using fedora 26 **

Regards,
Dor
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer

[GitHub] drill issue #877: DRILL-5660: Drill 1.10 queries fail due to Parquet Metadat...

2017-08-08 Thread vdiravka

Github user vdiravka commented on the issue:

https://github.com/apache/drill/pull/877
  
@arina-ielchiieva Small fix was made to resolve some regression tests 
failings. The branch is rebased to the master version.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #889: DRILL-5691: enhance scalar sub queries checking for the ca...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/889
  
@weijietong regarding the unit test, I have tried to reproduce the problem 
and written the following unit test:
```
  @Test
  public void test() throws Exception {
FileSystem fs = null;
try {
  fs = FileSystem.get(new Configuration());

  // create table with partition pruning
  test("use dfs_test.tmp");
  String tableName = "table_with_pruning";
  Path dataFile = new 
Path(TestTools.getWorkingPath(),"src/test/resources/parquet/alltypes_required.parquet");
  test("create table %s partition by (col_int) as select * from 
dfs.`%s`", tableName, dataFile);

  // generate metadata
  test("refresh table metadata `%s`", tableName);
  
  // execute query
  String query = String.format("select count(distinct col_int), 
count(distinct col_chr) from `%s` where col_int = 45436", tableName);
  test(query);

} finally {
  if (fs != null) {
fs.close();
  }
}
  }
```
`AbstractGroupScan.getScanStats` method returns one row but it does not 
fail. Can you please take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131871360
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -231,6 +236,12 @@ public static boolean isScalarSubquery(RelNode root) {
 return true;
   }
 }
+if(!hasMoreInputs && currentRel!=null){
+  double rowSize = 
RelMetadataQuery.instance().getMaxRowCount(currentRel);
+  if(rowSize==1){
--- End diff --

 Please add spaces: `if (rowSize == 1) {`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131871292
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -231,6 +236,12 @@ public static boolean isScalarSubquery(RelNode root) {
 return true;
   }
 }
+if(!hasMoreInputs && currentRel!=null){
--- End diff --

Please add spaces: `if (!hasMoreInputs && currentRel != null) {`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131871049
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -204,24 +205,28 @@ public static void 
addLeastRestrictiveCasts(LogicalExpression[] leftExpressions,
 
   /**
* Utility method to check if a subquery (represented by its root 
RelNode) is provably scalar. Currently
-   * only aggregates with no group-by are considered scalar. In the 
future, this method should be generalized
-   * to include more cases and reconciled with Calcite's notion of scalar.
+   * only aggregates with no group-by and sub input rel with one row are 
considered scalar. In the future,
+   * this method should be generalized to include more cases and 
reconciled with Calcite's notion of scalar.
* @param root The root RelNode to be examined
* @return True if the root rel or its descendant is scalar, False 
otherwise
*/
   public static boolean isScalarSubquery(RelNode root) {
 DrillAggregateRel agg = null;
-RelNode currentrel = root;
-while (agg == null && currentrel != null) {
-  if (currentrel instanceof DrillAggregateRel) {
-agg = (DrillAggregateRel)currentrel;
-  } else if (currentrel instanceof RelSubset) {
-currentrel = ((RelSubset)currentrel).getBest() ;
-  } else if (currentrel.getInputs().size() == 1) {
+RelNode currentRel = root;
+boolean hasMoreInputs = false;
+while (agg == null && currentRel != null) {
+  if (currentRel instanceof DrillAggregateRel) {
+agg = (DrillAggregateRel)currentRel;
+  } else if (currentRel instanceof RelSubset) {
+currentRel = ((RelSubset)currentRel).getBest() ;
+  } else if (currentRel.getInputs().size() == 1) {
 // If the rel is not an aggregate or RelSubset, but is a 
single-input rel (could be Project,
 // Filter, Sort etc.), check its input
-currentrel = currentrel.getInput(0);
+currentRel = currentRel.getInput(0);
   } else {
+if(currentRel.getInputs().size()>1){
--- End diff --

Please add spaces: `if (currentRel.getInputs().size() > 1) {`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131870863
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -204,24 +205,28 @@ public static void 
addLeastRestrictiveCasts(LogicalExpression[] leftExpressions,
 
   /**
* Utility method to check if a subquery (represented by its root 
RelNode) is provably scalar. Currently
-   * only aggregates with no group-by are considered scalar. In the 
future, this method should be generalized
-   * to include more cases and reconciled with Calcite's notion of scalar.
+   * only aggregates with no group-by and sub input rel with one row are 
considered scalar. In the future,
+   * this method should be generalized to include more cases and 
reconciled with Calcite's notion of scalar.
* @param root The root RelNode to be examined
* @return True if the root rel or its descendant is scalar, False 
otherwise
*/
   public static boolean isScalarSubquery(RelNode root) {
 DrillAggregateRel agg = null;
-RelNode currentrel = root;
-while (agg == null && currentrel != null) {
-  if (currentrel instanceof DrillAggregateRel) {
-agg = (DrillAggregateRel)currentrel;
-  } else if (currentrel instanceof RelSubset) {
-currentrel = ((RelSubset)currentrel).getBest() ;
-  } else if (currentrel.getInputs().size() == 1) {
+RelNode currentRel = root;
+boolean hasMoreInputs = false;
+while (agg == null && currentRel != null) {
+  if (currentRel instanceof DrillAggregateRel) {
+agg = (DrillAggregateRel)currentRel;
--- End diff --

` agg = (DrillAggregateRel) currentRel;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131871224
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -204,24 +205,28 @@ public static void 
addLeastRestrictiveCasts(LogicalExpression[] leftExpressions,
 
   /**
* Utility method to check if a subquery (represented by its root 
RelNode) is provably scalar. Currently
-   * only aggregates with no group-by are considered scalar. In the 
future, this method should be generalized
-   * to include more cases and reconciled with Calcite's notion of scalar.
+   * only aggregates with no group-by and sub input rel with one row are 
considered scalar. In the future,
+   * this method should be generalized to include more cases and 
reconciled with Calcite's notion of scalar.
* @param root The root RelNode to be examined
* @return True if the root rel or its descendant is scalar, False 
otherwise
*/
   public static boolean isScalarSubquery(RelNode root) {
 DrillAggregateRel agg = null;
-RelNode currentrel = root;
-while (agg == null && currentrel != null) {
-  if (currentrel instanceof DrillAggregateRel) {
-agg = (DrillAggregateRel)currentrel;
-  } else if (currentrel instanceof RelSubset) {
-currentrel = ((RelSubset)currentrel).getBest() ;
-  } else if (currentrel.getInputs().size() == 1) {
+RelNode currentRel = root;
+boolean hasMoreInputs = false;
+while (agg == null && currentRel != null) {
+  if (currentRel instanceof DrillAggregateRel) {
+agg = (DrillAggregateRel)currentRel;
+  } else if (currentRel instanceof RelSubset) {
+currentRel = ((RelSubset)currentRel).getBest() ;
+  } else if (currentRel.getInputs().size() == 1) {
 // If the rel is not an aggregate or RelSubset, but is a 
single-input rel (could be Project,
 // Filter, Sort etc.), check its input
-currentrel = currentrel.getInput(0);
+currentRel = currentRel.getInput(0);
   } else {
+if(currentRel.getInputs().size()>1){
+  hasMoreInputs=true;
--- End diff --

Please add spaces: `hasMoreInputs = true;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131871578
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestNestedLoopJoin.java
 ---
@@ -325,4 +325,5 @@ public void testNlJoinWithLargeRightInputSuccess() 
throws Exception {
   test(RESET_JOIN_OPTIMIZATION);
 }
   }
+
--- End diff --

Please revert changes in this file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/889#discussion_r131870970
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinUtils.java
 ---
@@ -204,24 +205,28 @@ public static void 
addLeastRestrictiveCasts(LogicalExpression[] leftExpressions,
 
   /**
* Utility method to check if a subquery (represented by its root 
RelNode) is provably scalar. Currently
-   * only aggregates with no group-by are considered scalar. In the 
future, this method should be generalized
-   * to include more cases and reconciled with Calcite's notion of scalar.
+   * only aggregates with no group-by and sub input rel with one row are 
considered scalar. In the future,
+   * this method should be generalized to include more cases and 
reconciled with Calcite's notion of scalar.
* @param root The root RelNode to be examined
* @return True if the root rel or its descendant is scalar, False 
otherwise
*/
   public static boolean isScalarSubquery(RelNode root) {
 DrillAggregateRel agg = null;
-RelNode currentrel = root;
-while (agg == null && currentrel != null) {
-  if (currentrel instanceof DrillAggregateRel) {
-agg = (DrillAggregateRel)currentrel;
-  } else if (currentrel instanceof RelSubset) {
-currentrel = ((RelSubset)currentrel).getBest() ;
-  } else if (currentrel.getInputs().size() == 1) {
+RelNode currentRel = root;
+boolean hasMoreInputs = false;
+while (agg == null && currentRel != null) {
+  if (currentRel instanceof DrillAggregateRel) {
+agg = (DrillAggregateRel)currentRel;
+  } else if (currentRel instanceof RelSubset) {
+currentRel = ((RelSubset)currentRel).getBest() ;
--- End diff --

`currentRel = ((RelSubset) currentRel).getBest() ;`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #891: DRILL-5699: Drill Web UI Page Source Has Links To External...

2017-08-08 Thread arina-ielchiieva

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/891
  
@parthchandra could you please do final review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: compiling drill 1.11.0 with cdh profile

2017-08-08 Thread Dor Ben Dov

 Can one help me with this ? 

Dor

-Original Message-
From: Dor Ben Dov 
Sent: יום ב 07 אוגוסט 2017 13:44
To: dev@drill.apache.org
Subject: compiling drill 1.11.0 with cdh profile

Hi all,

Tried to compile source code of branch 1.11.0 with profile cdh for cloudera - 
getting this exception, anyone ? 

 [dor@dor-fedora64 drill]$ mvn -U -DskipTests clean install -Pcdh [INFO] 
Scanning for projects...
Downloading: 
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom
Downloading: http://conjars.org/repo/org/apache/apache/14/apache-14.pom
Downloading: http://repository.mapr.com/maven/org/apache/apache/14/apache-14.pom
Downloading: http://repo.dremio.com/release/org/apache/apache/14/apache-14.pom
Downloading: 
http://repository.mapr.com/nexus/content/repositories/drill/org/apache/apache/14/apache-14.pom
Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: Could 
not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11  @ [ERROR] The build could not 
read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.drill:drill-root:1.11.0 
(/home/dor/Downloads/drill/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: 
Could not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11: Unknown host 
repository.cloudera.com: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[dor@dor-fedora64 drill]$

** I am using fedora 26 **

Regards,
Dor
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer

[jira] [Created] (DRILL-5708) Add DNS decode function for PCAP storage

2017-08-08 Thread Takeo Ogawara (JIRA)

Takeo Ogawara created DRILL-5708:


 Summary: Add DNS decode function for PCAP storage
 Key: DRILL-5708
 URL: https://issues.apache.org/jira/browse/DRILL-5708
 Project: Apache Drill
  Issue Type: Improvement
  Components: Storage - Other
Reporter: Takeo Ogawara
Priority: Minor


As described in DRILL-5432, it is very useful to analyze packet contents and 
application layer protocols. To improve the PCAP analysis function, it's better 
to add a function to decode DNS queries and responses. This enables to classify 
packets by FQDN and display user access trends.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5710) drill-config.sh incorrectly exits with Java 1.7 or later is required to run Apache Dril

[jira] [Created] (DRILL-5709) Provide a value vector method to convert a vector to nullable

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

[GitHub] drill pull request #866: DRILL-5657: Implement size-aware result set loader

Re: First Drill Ticket

Re: First Drill Ticket

First Drill Ticket

[GitHub] drill issue #814: Add clickable localhost URL

[GitHub] drill pull request #776: DRILL-5165: limitRel to return correct rows for lim...

Re: IntelliJ code format

Re: IntelliJ code format

[GitHub] drill issue #889: DRILL-5691: enhance scalar sub queries checking for the ca...

IntelliJ code format

Re: compiling drill 1.11.0 with cdh profile

[GitHub] drill issue #877: DRILL-5660: Drill 1.10 queries fail due to Parquet Metadat...

[GitHub] drill issue #889: DRILL-5691: enhance scalar sub queries checking for the ca...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill pull request #889: DRILL-5691: enhance scalar sub queries checking for...

[GitHub] drill issue #891: DRILL-5699: Drill Web UI Page Source Has Links To External...

RE: compiling drill 1.11.0 with cdh profile

[jira] [Created] (DRILL-5708) Add DNS decode function for PCAP storage

33 matches

Site Navigation

Mail list logo

Footer information