[GitHub] drill issue #643: DRILL-4989: Fix TestParquetWriter.testImpalaParquetBinaryA...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/643
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86201965
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
+  private List literals;
+
+  BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+  }
+
+  public int getNumericValue() {
+return numericValue;
+  }
+
+  public List getLiterals() {
+return literals;
+  }
+
+  /** Contains all literals that are allowed to represent boolean type. */
+  private final static Map allLiterals = 
Maps.newHashMap();
--- End diff --

Use `org.apache.drill.common.map.CaseInsensitiveMap`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86202063
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
--- End diff --

final (here and wherever possible below)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86201969
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
+  private List literals;
+
+  BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+  }
+
+  public int getNumericValue() {
+return numericValue;
+  }
+
+  public List getLiterals() {
+return literals;
+  }
+
+  /** Contains all literals that are allowed to represent boolean type. */
+  private final static Map allLiterals = 
Maps.newHashMap();
+
+  static {
+for (BooleanType booleanType : BooleanType.values()) {
+  for (String literal : booleanType.getLiterals()) {
+allLiterals.put(literal, booleanType);
+  }
+}
+  }
+
+  /**
+   * Finds boolean by passed literal.
+   * Leading or trailing whitespace is ignored, and case does not matter.
+   *
+   * @param literal boolean string representation
+   * @return boolean type
+   * @throws IllegalArgumentException if boolean type is not found
+   */
+  public static BooleanType get(String literal) {
+String value = literal.toLowerCase().trim();
+BooleanType booleanType = allLiterals.get(value);
+if (booleanType == null) {
+  throw new IllegalArgumentException("Invalid value for boolean: " + 
literal);
--- End diff --

How is this exception handled?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86238839
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
+  private List literals;
+
+  BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+  }
+
+  public int getNumericValue() {
+return numericValue;
+  }
+
+  public List getLiterals() {
+return literals;
+  }
+
+  /** Contains all literals that are allowed to represent boolean type. */
+  private final static Map allLiterals = 
Maps.newHashMap();
+
+  static {
+for (BooleanType booleanType : BooleanType.values()) {
+  for (String literal : booleanType.getLiterals()) {
+allLiterals.put(literal, booleanType);
+  }
+}
+  }
+
+  /**
+   * Finds boolean by passed literal.
+   * Leading or trailing whitespace is ignored, and case does not matter.
+   *
+   * @param literal boolean string representation
+   * @return boolean type
+   * @throws IllegalArgumentException if boolean type is not found
+   */
+  public static BooleanType get(String literal) {
+String value = literal.toLowerCase().trim();
+BooleanType booleanType = allLiterals.get(value);
+if (booleanType == null) {
+  throw new IllegalArgumentException("Invalid value for boolean: " + 
literal);
--- End diff --

Exception is straight-forward, I am not sure it needs additional handling. 
It returns wrapped up in UserRemoteException.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86230956
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86231467
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/BooleanType.java ---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr;
+
+import com.google.common.collect.Maps;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals if formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html;>Postgre 
Documentation}
+ */
+public enum BooleanType {
+  TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+  FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+  private int numericValue;
+  private List literals;
+
+  BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+  }
+
+  public int getNumericValue() {
+return numericValue;
+  }
+
+  public List getLiterals() {
+return literals;
+  }
+
+  /** Contains all literals that are allowed to represent boolean type. */
+  private final static Map allLiterals = 
Maps.newHashMap();
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4986) Allow users to customize the Drill log file name

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4986.

Resolution: Fixed
  Assignee: Paul Rogers
  Reviewer: Boaz Ben-Zvi

Fixed in 
[e044f5a|https://github.com/apache/drill/commit/e044f5a9d007266b9094fb8899d0a05df0b0dd0d]

> Allow users to customize the Drill log file name
> 
>
> Key: DRILL-4986
> URL: https://issues.apache.org/jira/browse/DRILL-4986
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> From John Omernik on the user list:
> I am getting my head around the new Drill config files in 1.8, and I have
> been setting DRILL_LOG_PREFIX in previous versions. Based on what I am
> seeing in drill-config.sh though, it looks like I no longer can set this as
> instead of using a "If set use what's set otherwise default to" methodology
> like other variables, drill-config.sh just exports a static value... is
> this correct?
> My goal is to have logs all in a single directory but have prefixes based
> on the host names... "centralized" logging if you will :)
> Suggestion:
> {code}
> export DRILL_LOG_PREFIX="$DRILL_LOG_DIR/drillbit"
> {code}
> So looking at this, I think that DRILL_LOG_PREFIX is a misnomer, it should
> be DRILL_LOG_PATH = Path to a directory, DRILL_LOG_PREFIX = A prefix to
> prepend to drill log files, thus, in reality, DRILL_LOG_PREFIX should not
> be set to $DRILL_LOG_DIR/drillbit, instead the default should be "
> {code}
> export DRILL_LOG_PREFIX=${DRILL_LOG_PREFIX:-"drillbit"}
> {code}
> and then the next line that makes the drill log path should be:
> {code}
> export DRILLBIT_LOG_PATH="${DRILL_LOG_DIR}/${DRILL_LOG_PREFIX}.log"
> {code}
> That way the Prefix is just a file prefix, and then in the drill-env.sh I
> could set
> {code}
> export DRILL_LOG_PREFIX="drillbit-$HOSTNAME"
> {code}
> and have all my logs in one folder, but not be overwritten by easy other



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #643: DRILL-4989: Fix TestParquetWriter.testImpalaParquet...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/643


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #635: DRILL-4927 (part 2): Add support for Null Equality ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/635


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #551: DRILL-4792: Include session options used for a quer...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/551


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #636: DRILL-4951: Do Guava.patch earlier so we can run si...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/636


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #641: DRILL-4986: Allow customization of the Drill log fi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/641


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #642: DRILL-4987: Use ImpersonationUtil to get process user info...

[parthchandra]

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/642
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #482: DRILL-4604: Generate warning on Web UI if drillbits...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/482#discussion_r85954983
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
   @GET
   @Path("/stats.json")
   @Produces(MediaType.APPLICATION_JSON)
-  public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits", 
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit : 
work.getContext().getBits()) {
-  String initialized = bit.isInitialized() ? " initialized" : " not 
initialized";
-  stats.add(new Stat("Bit #" + number, bit.getAddress() + 
initialized));
-  ++number;
+  public Stats getStatsJSON() {
+final String version = 
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
+
+final Map props = Maps.newLinkedHashMap();
+props.put("Cluster Version", version);
+props.put("Number of Drillbits", work.getContext().getBits().size());
+CoordinationProtos.DrillbitEndpoint currentEndpoint = 
work.getContext().getEndpoint();
+final String address = currentEndpoint.getAddress();
+props.put("Data Port Address", address + ":" + 
currentEndpoint.getDataPort());
+props.put("User Port Address", address + ":" + 
currentEndpoint.getUserPort());
+props.put("Control Port Address", address + ":" + 
currentEndpoint.getControlPort());
+props.put("Maximum Direct Memory", DrillConfig.getMaxDirectMemory());
+
+return new Stats(props, collectDrillbits(version));
+  }
+
+  private Collection collectDrillbits(String version) {
+Set drillbits = Sets.newTreeSet();
+for (CoordinationProtos.DrillbitEndpoint endpoint : 
work.getContext().getBits()) {
+  boolean versionMatch = version.equals(endpoint.getVersion());
+  DrillbitInfo drillbit = new DrillbitInfo(endpoint.getAddress(), 
endpoint.isInitialized(), endpoint.getVersion(), versionMatch);
--- End diff --

1. Removed initialized.
2. Check for null is done using freemarker: 
<#if 
(drillbit.getVersion())?has_content>${drillbit.getVersion()}<#else>Undefined
 
has_content checks if version is not null and is not empty. 
http://freemarker.org/docs/ref_builtins_expert.html#ref_builtin_has_content


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #482: DRILL-4604: Generate warning on Web UI if drillbits...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/482#discussion_r85978846
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +210,21 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * If cluster version is the same as default, update option to current 
drillbit version.
+   */
+  private void checkAndUpdateClusterVersionOption(OptionManager 
optionManager, DrillbitEndpoint drillbitEndpoint) {
+OptionValue versionOption = 
optionManager.getOption(ExecConstants.CLUSTER_VERSION);
+OptionValidator validator = 
SystemOptionManager.getValidator(ExecConstants.CLUSTER_VERSION);
+if (versionOption.equals(validator.getDefault())) {
+  optionManager.setOption(OptionValue.createOption(
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #482: DRILL-4604: Generate warning on Web UI if drillbits...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/482#discussion_r85974147
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -17,6 +17,7 @@
  */
 package org.apache.drill.exec;
 
+import org.apache.drill.common.util.DrillVersionInfo;
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #482: DRILL-4604: Generate warning on Web UI if drillbits...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/482#discussion_r85974663
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +210,21 @@ private void javaPropertiesToSystemOptions() {
   }
 
   /**
+   * If cluster version is the same as default, update option to current 
drillbit version.
+   */
+  private void checkAndUpdateClusterVersionOption(OptionManager 
optionManager, DrillbitEndpoint drillbitEndpoint) {
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #482: DRILL-4604: Generate warning on Web UI if drillbits...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/482#discussion_r86107203
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
   @GET
   @Path("/stats.json")
   @Produces(MediaType.APPLICATION_JSON)
-  public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits", 
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit : 
work.getContext().getBits()) {
-  String initialized = bit.isInitialized() ? " initialized" : " not 
initialized";
-  stats.add(new Stat("Bit #" + number, bit.getAddress() + 
initialized));
-  ++number;
+  public Stats getStatsJSON() {
+final String version = 
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
+
+final Map props = Maps.newLinkedHashMap();
+props.put("Cluster Version", version);
+props.put("Number of Drillbits", work.getContext().getBits().size());
+CoordinationProtos.DrillbitEndpoint currentEndpoint = 
work.getContext().getEndpoint();
+final String address = currentEndpoint.getAddress();
+props.put("Data Port Address", address + ":" + 
currentEndpoint.getDataPort());
+props.put("User Port Address", address + ":" + 
currentEndpoint.getUserPort());
+props.put("Control Port Address", address + ":" + 
currentEndpoint.getControlPort());
+props.put("Maximum Direct Memory", DrillConfig.getMaxDirectMemory());
+
+return new Stats(props, collectDrillbits(version));
+  }
+
+  private Collection collectDrillbits(String version) {
+Set drillbits = Sets.newTreeSet();
+for (CoordinationProtos.DrillbitEndpoint endpoint : 
work.getContext().getBits()) {
+  boolean versionMatch = version.equals(endpoint.getVersion());
+  DrillbitInfo drillbit = new DrillbitInfo(endpoint.getAddress(), 
endpoint.isInitialized(), endpoint.getVersion(), versionMatch);
+  drillbits.add(drillbit);
 }
-stats.add(new Stat("Data Port Address", 
work.getContext().getEndpoint().getAddress() +
-  ":" + work.getContext().getEndpoint().getDataPort()));
-stats.add(new Stat("User Port Address", 
work.getContext().getEndpoint().getAddress() +
-  ":" + work.getContext().getEndpoint().getUserPort()));
-stats.add(new Stat("Control Port Address", 
work.getContext().getEndpoint().getAddress() +
-  ":" + work.getContext().getEndpoint().getControlPort()));
-stats.add(new Stat("Maximum Direct Memory", 
DrillConfig.getMaxDirectMemory()));
-
-return stats;
+return drillbits;
   }
 
   @XmlRootElement
-  public class Stat {
-private String name;
-private Object value;
+  public static class Stats {
--- End diff --

Renamed to ClusterInfo


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #610: DRILL-4674: Allow casting to boolean the same liter...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/610#discussion_r86103902
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/SimpleCastFunctions.java
 ---
@@ -48,14 +46,8 @@ public void setup() {
 public void eval() {
   byte[] buf = new byte[in.end - in.start];
   in.buffer.getBytes(in.start, buf, 0, in.end - in.start);
-  String input = new String(buf, 
com.google.common.base.Charsets.UTF_8).toLowerCase();
-  if ("true".equals(input)) {
-out.value = 1;
-  } else if ("false".equals(input)) {
-out.value = 0;
-  } else {
-throw new IllegalArgumentException("Invalid value for boolean: " + 
input);
-  }
+  String input = new String(buf, 
com.google.common.base.Charsets.UTF_8);
+  out.value = 
org.apache.drill.exec.expr.BooleanType.get(input).getNumericValue();
--- End diff --

@sudheeshkatkam 
I can't make static `Map`, since sometimes we need numeric 
representation and sometimes bytes. But you are right iteration through the 
list per each record is too expensive. So I have added static `Map`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #642: DRILL-4987: Use ImpersonationUtil to get process user info...

[arina-ielchiieva]

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/642
  
No, you did not. User who is starting the drillbit should check if udf 
areas are writable. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #613: DRILL-4730: Update JDBC DatabaseMetaData implementation to...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/613
  
I started looking into this. Can you please split the commit into two 
(refactoring changes, and the core changes that use new API) for quicker review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #610: DRILL-4674: Allow casting to boolean the same literals as ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/610
  
Can you please squash and rebase?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #632: DRILL-4969: Basic implementation of display size fo...

[laurentgo]

Github user laurentgo commented on a diff in the pull request:

https://github.com/apache/drill/pull/632#discussion_r86254267
  
--- Diff: common/src/main/java/org/apache/drill/common/types/Types.java ---
@@ -275,6 +275,84 @@ public static boolean isJdbcSignedType( final 
MajorType type ) {
 return isSigned;
   }
 
+  public static int getJdbcDisplaySize(MajorType type) {
+if (type.getMode() == DataMode.REPEATED || type.getMinorType() == 
MinorType.LIST) {
+  return 0;
--- End diff --

0 implies that we don't know the display size


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4990) Use new HDFS API access instead of listStatus to check if users have permissions to access workspace.

[Padma Penumarthy (JIRA)]

Padma Penumarthy created DRILL-4990:
---

 Summary: Use new HDFS API access instead of listStatus to check if 
users have permissions to access workspace.
 Key: DRILL-4990
 URL: https://issues.apache.org/jira/browse/DRILL-4990
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Affects Versions: 1.8.0
Reporter: Padma Penumarthy
Assignee: Padma Penumarthy
 Fix For: 1.9.0


For every query, we build the schema tree 
(runSQL->getPlan->getNewDefaultSchema->getRootSchema). All workspaces in all 
storage plugins are checked and are added to the schema tree if they are 
accessible by the user who initiated the query.  For file system plugin, 
listStatus API is used to check if  the workspace is accessible or not 
(WorkspaceSchemaFactory.accessible) by the user.  The idea seem to be if the 
user does not have access to file(s) in the workspace, listStatus will generate 
an exception and we return false. But, listStatus (which lists all the entries 
of a directory) is an expensive operation when there are large number of files 
in the directory. A new API is added in Hadoop 2.6 called access (HDFS-6570) 
which provides the ability to check if the user has permissions on a 
file/directory.  Use this new API instead of listStatus. For a directory with 
256k+ files, an improvement of upto 10 sec in planning time was observed when 
using the new API vs. old way of listStatus. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #632: DRILL-4969: Basic implementation of display size fo...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/632#discussion_r86253425
  
--- Diff: common/src/main/java/org/apache/drill/common/types/Types.java ---
@@ -275,6 +275,84 @@ public static boolean isJdbcSignedType( final 
MajorType type ) {
 return isSigned;
   }
 
+  public static int getJdbcDisplaySize(MajorType type) {
+if (type.getMode() == DataMode.REPEATED || type.getMinorType() == 
MinorType.LIST) {
+  return 0;
--- End diff --

Why does 0 imply?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r86267336
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -246,28 +154,80 @@ protected void handle(UserClientConnectionImpl 
connection, int rpcType, ByteBuf
   public class UserClientConnectionImpl extends RemoteConnection 
implements UserClientConnection {
 
 private UserSession session;
+private SaslServer saslServer;
+private RequestHandler currentHandler;
+private UserToBitHandshake inbound;
 
 public UserClientConnectionImpl(SocketChannel channel) {
   super(channel, "user client");
+  currentHandler = authFactory == null ? handler : new 
UserServerAuthenticationHandler(handler);
 }
 
 void disableReadTimeout() {
   getChannel().pipeline().remove(BasicServer.TIMEOUT_HANDLER);
 }
 
-void setUser(final UserToBitHandshake inbound) throws IOException {
+void setHandshake(final UserToBitHandshake inbound) throws IOException 
{
+  this.inbound = inbound;
+}
+
+void initSaslServer(final String mechanismName, final Map 
properties)
+throws IllegalStateException, SaslException {
+  if (saslServer != null) {
+throw new IllegalStateException("SASL server already 
initialized.");
+  }
+  this.saslServer = authFactory.getMechanism(mechanismName)
--- End diff --

Makes sense as an enhancement. The current properties are for SASL library.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4991) NPE when running CTAS auto partition on date and varchar columns

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4991:


 Summary: NPE when running CTAS auto partition on date and varchar 
columns
 Key: DRILL-4991
 URL: https://issues.apache.org/jira/browse/DRILL-4991
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Reporter: Rahul Challapalli


git.commit.id.abbrev=4ee1d4c

The below ctas query fails with NPE
{code}
create table item_multipart partition by (i_rec_start_date, i_size) as select * 
from item;
Error: SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 0f861e32-4d14-461e-bfc3-b75fd3ac6668 on qa-node190.qa.lab:31010] 
(state=,code=0)
{code}

There are null values for i_size column for some of the records. Still not sure 
if this is expected though. This is also happening in drill 1.8 as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r86268011
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/security/SimpleServer.java
 ---
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security;
+
+import com.google.common.primitives.Ints;
+
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.sasl.SaslException;
+import javax.security.sasl.SaslServer;
+import javax.security.sasl.SaslServerFactory;
+import java.util.Map;
+
+public class SimpleServer implements SaslServer {
+
+  private boolean completed;
+  private String authorizationId;
+  private final int total;
+  private int count = 0;
+
+  SimpleServer(final int total) {
+this.total = total;
+  }
+
+  @Override
+  public String getMechanismName() {
+return SimpleProvider.MECHANISM_NAME;
+  }
+
+  @Override
+  public byte[] evaluateResponse(byte[] response) throws SaslException {
+if (completed) {
+  throw new IllegalStateException("SimpleSasl authentication already 
completed");
+}
+if (response == null || response.length < 1) {
+  throw new SaslException("Received challenge is empty when secret 
expected");
+}
+
+if (count == 0) { // first expect authorization ID
+  //This SaslServer simply permits a client to authenticate according 
to whatever username
+  //was supplied in client's response[]
+  authorizationId = new String(response);
--- End diff --

Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r86267015
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/AuthenticationMechanismFactory.java
 ---
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security;
+
+import com.google.common.base.Function;
+import com.google.common.base.Strings;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.AutoCloseables;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.rpc.security.kerberos.KerberosMechanism;
+import org.apache.drill.exec.rpc.security.plain.PlainMechanism;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticatorFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class AuthenticationMechanismFactory implements AutoCloseable {
+  private static final org.slf4j.Logger logger =
+  
org.slf4j.LoggerFactory.getLogger(AuthenticationMechanismFactory.class);
+
+  private final Map mechanisms = 
CaseInsensitiveMap.newHashMapWithExpectedSize(5);
+
+  @SuppressWarnings("unchecked")
+  public AuthenticationMechanismFactory(final ScanResult scan, final 
DrillConfig config,
+final List 
configuredMechanisms)
+  throws DrillbitStartupException {
+logger.debug("Configuring authentication mechanisms: {}", 
configuredMechanisms);
+// transform all names to uppercase
+final Set configuredMechanismsSet = 
Sets.newHashSet(Iterators.transform(configuredMechanisms.iterator(),
+new Function() {
+  @Nullable
+  @Override
+  public String apply(@Nullable String input) {
+return input == null ? null : input.toUpperCase();
+  }
+}));
+
+// First, load Drill provided out-of-box mechanisms
+if (configuredMechanismsSet.contains(PlainMechanism.SIMPLE_NAME)) {
+  logger.trace("Plain mechanism enabled.");
+  // instantiated here, but closed in PlainMechanism#close
+  final UserAuthenticator userAuthenticator = 
UserAuthenticatorFactory.createAuthenticator(config, scan);
+  mechanisms.put(PlainMechanism.SIMPLE_NAME, new 
PlainMechanism(userAuthenticator));
+  configuredMechanismsSet.remove(PlainMechanism.SIMPLE_NAME);
+}
+
+if (configuredMechanismsSet.contains(KerberosMechanism.SIMPLE_NAME)) {
+  logger.trace("Kerberos mechanism enabled.");
+  final String servicePrincipal = 
config.getString("drill.exec.security.auth.principal");
+  final String keytab = 
config.getString("drill.exec.security.auth.keytab");
+
+  try { // Kerberos mechanism requires a service to login
+final Configuration conf = new Configuration();
+conf.set(CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION, 
"KERBEROS");
+// To parse non-typical principal name, uncomment below line
+// CommonConfigurationKeys.HADOOP_SECURITY_AUTH_TO_LOCAL, rules);
+UserGroupInformation.setConfiguration(conf);
+UserGroupInformation.loginUserFromKeytab(servicePrincipal, keytab);

[GitHub] drill issue #613: DRILL-4730: Update JDBC DatabaseMetaData implementation to...

[laurentgo]

Github user laurentgo commented on the issue:

https://github.com/apache/drill/pull/613
  
the only "refactoring" I can think of is between DrillCursor and 
DrillResultSetImpl, anything else is related to the feature. I'll try to split 
it, but hopefully in the current state, it should still be pretty readable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #610: DRILL-4674: Allow casting to boolean the same literals as ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/610
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #632: DRILL-4969: Basic implementation of display size for colum...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/632
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apache Drill Hangout Minutes - 11/1/16

[Laurent Goujon]

That's a good point that the introduction of new request types for prepared
statement and metadata introduced some incompatibility issues, and that's
why I added a client/server version exchange so that client can make smart
decisions about using these new methods or not (this is the approach used
by the ODBC driver, where the new calls will be used only if server
advertises 1.9.0 or newer). On Java side, direct users of DrillClient would
have to check the version as well, a thing the JDBC driver doesn't. That
said, the metadata change is not merged yet (and I will add the extra
logic), which is not the case regarding prepared statement support.

Also, I verified that the old server doesn't drop the connection: it's
actually dropping the request, and not responding to the client :( (best
would be to return an error message to say unsupported).

Laurent


On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam  wrote:

> Hi Laurent,
>
> That's right; this was mentioned in the design document.
>
> I am piggybacking on previous changes that break the "newer clients talking
> to older servers" compatibility. For example, as I understand, some
> resolved sub-tasks of DRILL-4714 [1] *implicitly* break this compatibility;
> say the "newer" API that was introduced is used by an application which is
> talking to an older server. The older server drops the connection, unable
> to handle the message.
>
> In DRILL-4280, there is an *explicit* break in that specific compatibility,
> and the error message is much cleaner with a version mismatch message. The
> difference is that the C++ client (unlike the Java client) checks for the
> server version as well, which make the compatibility break more visible.
>
> I am not sure about the plan of action in general about this compatibility.
> However, I could work around the issue by advertising clients' SASL
> capability to the server. What do you think?
>
> Thank you,
> Sudheesh
>
> [1] https://issues.apache.org/jira/browse/DRILL-4714
>
> On Nov 1, 2016, at 7:49 PM, Laurent Goujon  wrote:
>
> Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan
> to defer this change to a later release, or to defer bringing back
> compatibility between newer clients and older servers to a later release?
>
> Laurent
>
> On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong  wrote:
>
> Oops, mistake in my notes.  For the second item, I meant DRILL-4280, not
> DRILL-1950.
>
> On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong  wrote:
>
> Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
> Arina, Laurent, Kunal, Zelaine
>
> I had to leave the hangout at 10:30, so my notes only cover the
>
> discussion
>
> up till then.
>
> 1) Variable width decimal support - Dave O
>
> Currently Drill only supports fixed width byte array storage of decimals.
> Dave has submitted a pull request for DRILL-4834 to add support for
>
> storing
>
> decimals with variable width byte arrays.  Eventually, variable width can
> replace fixed width, but the pull request doesn't cover that.  Dave would
> like someone in the community to review his pull request.
>
> 2) 1.9 release - Sudheesh
>
> Sudheesh is collecting pull requests for the release.  Some have been
> reviewed and are waiting to be merged.  Sudheesh plans to commit a batch
> this Wed and another this Friday.  He's targeting having a release
> candidate build available next Monday.
>
> Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
> whether thought had been given to supporting newer Drill clients with
>
> older
>
> Drill servers.  Sudheesh indicated that doing this would entail a
>
> breaking
>
> change in the protocol, and the plan was to defer doing this for a later
> release where we may want to make other breaking changes like this.
>

[jira] [Created] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.7.0 server

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4994:
-

 Summary: Prepared statement stopped working between 1.8.0 client 
and < 1.7.0 server
 Key: DRILL-4994
 URL: https://issues.apache.org/jira/browse/DRILL-4994
 Project: Apache Drill
  Issue Type: Bug
Reporter: Laurent Goujon


Older servers (pre-1.8.0) don't support the prepared statement rpc method, but 
the JDBC client doesn't check if it is available or not. The end result is that 
the statement is stuck as the server is not responding back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4992) Failing query with same case statement in both select and order by clause using using hash aggregate

[Gautam Kumar Parai (JIRA)]

Gautam Kumar Parai created DRILL-4992:
-

 Summary: Failing query with same case statement in both select and 
order by clause using using hash aggregate
 Key: DRILL-4992
 URL: https://issues.apache.org/jira/browse/DRILL-4992
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.8.0, 1.7.0
Reporter: Gautam Kumar Parai


Queries that contain case statement in both the select and order by clause as a 
sub query. Here is an example of such query:


dummy
+-++
|   c_date|  c_timestamp   |
+-++
| 2015-04-23  | 2014-03-16 03:55:21.0  |
+-++

alter session set `planner.enable_streamagg` = false;
select distinct a1 from ( select SUM(case when c_timestamp is null then 0 else 
1 end) from dummy group by c_date order by SUM(case when c_timestamp is null 
then 0 else 1 end)) as dt(a1);

Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema 
changes
Note that this table is a single table. 
Below is the stacktrace from log file:
{code}
2016-10-31 15:57:45,643 [27e83395-8074-7c94-e318-a6b54176ea9d:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 27e83395-8074-7c94-e318-a6b54176ea9d:0:0: 
State to report: RUNNING
2016-10-31 15:57:45,665 [27e83395-8074-7c94-e318-a6b54176ea9d:frag:0:0] INFO  
o.a.d.e.p.i.aggregate.HashAggBatch - User Error Occurred
org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR: 
Hash aggregate does not support schema changes


[Error Id: ad36df25-3ea8-4c07-87a0-f105b1ce5ae1 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:144)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.7.0_67]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_67]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1607.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
 [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_67]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2016-10-31 15:57:45,665 [27e83395-8074-7c94-e318-a6b54176ea9d:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27e83395-8074-7c94-e318-a6b54176ea9d:0:0: 
State change requested RUNNING --> FAILED
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4993) Documentation: Wrong output displayed for convert_from() with a map

[Boaz Ben-Zvi (JIRA)]

Boaz Ben-Zvi created DRILL-4993:
---

 Summary: Documentation: Wrong output displayed for convert_from() 
with a map
 Key: DRILL-4993
 URL: https://issues.apache.org/jira/browse/DRILL-4993
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.8.0
Reporter: Boaz Ben-Zvi
Priority: Trivial
 Fix For: 1.9.0


In the Drill docs: 
 SQL REFERENCE -> SQL FUNCTIONS -> DATA TYPE CONVERSION

The output for the example shown for convert_from() with a map into JSON is 
wrong:
-  BEGIN EXCERPT 
-
This example uses a map as input to return a repeated list vector (JSON).

SELECT CONVERT_FROM('[{a : 100, b: 200}, {a:300, b: 400}]' ,'JSON') AS MYCOL1  
FROM (VALUES(1));
++
|   MYCOL1   |
++
| [[1,2],[3,4],[5]]  |
++
1 row selected (0.141 seconds)

-  END --

The correct output should be:
---
SELECT CONVERT_FROM('[{a : 100, b: 200}, {a:300, b: 400}]' ,'JSON') AS MYCOL1  
FROM (VALUES(1));
++
| MYCOL1 |
++
| [{"a":100,"b":200},{"a":300,"b":400}]  |
++
1 row selected (2.618 seconds)

The error probably resulted from copying the output of the previous example.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #619: DRILL-4946: redirect System.err so users under embe...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/619#discussion_r86069796
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/StdErrCapturer.java ---
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.util;
+
+
+import java.io.ByteArrayOutputStream;
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * The safe way of this class is to redirect System.err to nullOutputStream
+ * If
--- End diff --

comment got chopped after "If ..."



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apache Drill Hangout Minutes - 11/1/16

[Laurent Goujon]

(+dev@drill.apache.org bcc u...@drill.apache.org)
other options:
- let the user decide between plain auth and sasl auth
- let the client could send a flag to announce support for SASL auth, and
do not send the credentials in the first handshake. An old server would
then send AUTH_FAILED (and drop the connection) but the client would retry
with the plain credentials, whereas the new server would send AUTH_REQUIRED
as expected.

Laurent

Laurent

On Wed, Nov 2, 2016 at 3:30 PM, Sudheesh Katkam  wrote:

> I am going to update the pull request so that both will be "ok".
>
> This implies that username/password credentials will be sent to the server
> twice, during handshake and during SASL exchange. And sending credentials
> through handshake will be deprecated (and removed in a future release).
>
> Thank you,
> Sudheesh
>
> On Wed, Nov 2, 2016 at 2:58 PM, Jacques Nadeau  wrote:
>
> > Since I'm not that close to DRILL-4280, I wanted to clarify expectation:
> >
> >
> > <1.9 Client  <==>  1.9 Server (ok)
> >  1.9 Client  <==> <1.9 Server (fails)
> >
> > Is that correct?
> >
> >
> >
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam 
> > wrote:
> >
> > > Hi Laurent,
> > >
> > > That's right; this was mentioned in the design document.
> > >
> > > I am piggybacking on previous changes that break the "newer clients
> > talking
> > > to older servers" compatibility. For example, as I understand, some
> > > resolved sub-tasks of DRILL-4714 [1] *implicitly* break this
> > compatibility;
> > > say the "newer" API that was introduced is used by an application which
> > is
> > > talking to an older server. The older server drops the connection,
> unable
> > > to handle the message.
> > >
> > > In DRILL-4280, there is an *explicit* break in that specific
> > compatibility,
> > > and the error message is much cleaner with a version mismatch message.
> > The
> > > difference is that the C++ client (unlike the Java client) checks for
> the
> > > server version as well, which make the compatibility break more
> visible.
> > >
> > > I am not sure about the plan of action in general about this
> > compatibility.
> > > However, I could work around the issue by advertising clients' SASL
> > > capability to the server. What do you think?
> > >
> > > Thank you,
> > > Sudheesh
> > >
> > > [1] https://issues.apache.org/jira/browse/DRILL-4714
> > >
> > > On Nov 1, 2016, at 7:49 PM, Laurent Goujon  wrote:
> > >
> > > Just for clarity, DRILL-4280 is a breaking-protocol change, so is the
> > plan
> > > to defer this change to a later release, or to defer bringing back
> > > compatibility between newer clients and older servers to a later
> release?
> > >
> > > Laurent
> > >
> > > On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong 
> wrote:
> > >
> > > Oops, mistake in my notes.  For the second item, I meant DRILL-4280,
> not
> > > DRILL-1950.
> > >
> > > On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong 
> wrote:
> > >
> > > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
> > > Arina, Laurent, Kunal, Zelaine
> > >
> > > I had to leave the hangout at 10:30, so my notes only cover the
> > >
> > > discussion
> > >
> > > up till then.
> > >
> > > 1) Variable width decimal support - Dave O
> > >
> > > Currently Drill only supports fixed width byte array storage of
> decimals.
> > > Dave has submitted a pull request for DRILL-4834 to add support for
> > >
> > > storing
> > >
> > > decimals with variable width byte arrays.  Eventually, variable width
> can
> > > replace fixed width, but the pull request doesn't cover that.  Dave
> would
> > > like someone in the community to review his pull request.
> > >
> > > 2) 1.9 release - Sudheesh
> > >
> > > Sudheesh is collecting pull requests for the release.  Some have been
> > > reviewed and are waiting to be merged.  Sudheesh plans to commit a
> batch
> > > this Wed and another this Friday.  He's targeting having a release
> > > candidate build available next Monday.
> > >
> > > Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
> > > whether thought had been given to supporting newer Drill clients with
> > >
> > > older
> > >
> > > Drill servers.  Sudheesh indicated that doing this would entail a
> > >
> > > breaking
> > >
> > > change in the protocol, and the plan was to defer doing this for a
> later
> > > release where we may want to make other breaking changes like this.
> > >
> >
>

[GitHub] drill pull request #560: DRILL-4823: Fix OOM while trying to prune partition...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/560#discussion_r86104090
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
 ---
@@ -378,8 +388,14 @@ public ValueHolder 
visitDoubleConstant(ValueExpressions.DoubleExpression dExpr,
 }
 
 @Override
-public ValueHolder 
visitQuotedStringConstant(ValueExpressions.QuotedString e, Integer value) 
throws RuntimeException {
-  return 
ValueHolderHelper.getVarCharHolder(getManagedBufferIfAvailable(), e.value);
+public ValueHolder visitQuotedStringConstant(final 
ValueExpressions.QuotedString e, Integer value) throws RuntimeException {
+  return getConstantValueHolder(e.value, new Function() {
--- End diff --

My fault, you are totally right. Now cache stores holders by type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #643: DRILL-4989: Fix TestParquetWriter.testImpalaParquet...

[vdiravka]

GitHub user vdiravka opened a pull request:

https://github.com/apache/drill/pull/643

DRILL-4989: Fix 
TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange

Test `TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange` 
expects sorted output but query does not include order by clause. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vdiravka/drill DRILL-4989

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/643.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #643


commit 1c39bd3a601b0adc38ee1c1915390a1da8a590d4
Author: Vitalii Diravka 
Date:   2016-11-02T15:51:56Z

DRILL-4989: Fix 
TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4989) Fix TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-4989:
---

 Summary: Fix 
TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange
 Key: DRILL-4989
 URL: https://issues.apache.org/jira/browse/DRILL-4989
 Project: Apache Drill
  Issue Type: Bug
Reporter: Arina Ielchiieva
Assignee: Vitalii Diravka
 Fix For: 1.9.0


Test TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange expects 
sorted output but query does not include order by clause. This causes unit test 
to fail sometimes:
{noformat}
Expected Records near verification failure:
Record Number: 0 { `int96_ts` : 1970-01-01T00:00:01.000Z, }
Record Number: 1 { `int96_ts` : 1971-01-01T00:00:01.000Z, }
Record Number: 2 { `int96_ts` : 1972-01-01T00:00:01.000Z, }
Record Number: 3 { `int96_ts` : 1973-01-01T00:00:01.000Z, }
Record Number: 4 { `int96_ts` : 1974-01-01T00:00:01.000Z, }
Record Number: 5 { `int96_ts` : 2010-01-01T00:00:01.000Z, }
Record Number: 6 { `int96_ts` : 2011-01-01T00:00:01.000Z, }
Record Number: 7 { `int96_ts` : 2012-01-01T00:00:01.000Z, }
Record Number: 8 { `int96_ts` : 2013-01-01T00:00:01.000Z, }
Record Number: 9 { `int96_ts` : 2014-01-01T00:00:01.000Z, }

Actual Records near verification failure:
Record Number: 0 { `int96_ts` : 2010-01-01T00:00:01.000Z, }
Record Number: 1 { `int96_ts` : 2011-01-01T00:00:01.000Z, }
Record Number: 2 { `int96_ts` : 2012-01-01T00:00:01.000Z, }
Record Number: 3 { `int96_ts` : 2013-01-01T00:00:01.000Z, }
Record Number: 4 { `int96_ts` : 2014-01-01T00:00:01.000Z, }
Record Number: 5 { `int96_ts` : 2015-01-01T00:00:01.000Z, }
Record Number: 6 { `int96_ts` : 2016-01-01T00:00:01.000Z, }
Record Number: 7 { `int96_ts` : 1970-01-01T00:00:01.000Z, }
Record Number: 8 { `int96_ts` : 1971-01-01T00:00:01.000Z, }
Record Number: 9 { `int96_ts` : 1972-01-01T00:00:01.000Z, }
{noformat}

To fix add order by in select statement:
{noformat}
select int96_ts from dfs_test.`%s/parquet/int96_dict_change` order by int96_ts
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill issue #643: DRILL-4989: Fix TestParquetWriter.testImpalaParquetBinaryA...

[arina-ielchiieva]

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/643
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---