[jira] [Commented] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Jose Flores (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112159#comment-16112159
 ] 

Jose Flores commented on DRILL-5698:


I have the same issue and got here when I was about to create an issue.
It's can also be reproduced this way:

- Current behaviour

bash-3.2# echo "version 1.8.0_66" | grep "version" | egrep -e "1.4|1.5|1.6"
(no output)

bash-3.2# echo "version 1.8.0_144" | grep "version" | egrep -e "1.4|1.5|1.6"
version 1.8.0_144

- With the proposal fix

bash-3.2# echo "version 1.8.0_144" | grep "version" | egrep -e "1\.4|1\.5|1\.6"
(no output)

bash-3.2# echo "version 1.6" | grep "version" | egrep -e "1\.4|1\.5|1\.6"
version 1.6

Workaround (for now): install java version not containing 14, 15 16 in the 
version (revision)

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111806#comment-16111806
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r130993406
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -122,10 +122,10 @@
   String HTTP_SESSION_MEMORY_RESERVATION = 
"drill.exec.http.session.memory.reservation";
   String HTTP_SESSION_MEMORY_MAXIMUM = 
"drill.exec.http.session.memory.maximum";
   String HTTP_SESSION_MAX_IDLE_SECS = 
"drill.exec.http.session_max_idle_secs";
-  String HTTP_KEYSTORE_PATH = "javax.net.ssl.keyStore";
--- End diff --

These are predefined system property names and we should continue to use 
these. System.getProperties will get these values correctly even if the user 
does not provide them in the config file. 
Also, JSSE looks for these properties. 


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111807#comment-16111807
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r131009531
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/TestSSLConfig.java ---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.SSLConfig;
+import org.apache.drill.test.OperatorFixture;
+import org.junit.Test;
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestSSLConfig {
+
+
+  @Test
+  public void firstTest() throws Exception {
+
+OperatorFixture.OperatorFixtureBuilder builder = 
OperatorFixture.builder();
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PASSWORD, 
"root");
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PATH, "");
+try (OperatorFixture fixture = builder.build()) {
+  DrillConfig config = fixture.config();
+  try {
+SSLConfig sslv = new SSLConfig(config);
+fail();
+  } catch (Exception e) {
+assertTrue(e instanceof DrillException);
+  }
+}
+  }
+
+  @Test
+  public void secondTest() throws Exception {
+
+OperatorFixture.OperatorFixtureBuilder builder = 
OperatorFixture.builder();
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "");
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PATH, "/root");
+try (OperatorFixture fixture = builder.build()) {
+  DrillConfig config = fixture.config();
+  try {
+SSLConfig sslv = new SSLConfig(config);
+fail();
+  } catch (Exception e) {
+assertTrue(e instanceof DrillException);
+  }
+}
+  }
+
+  @Test
+  public void thirdTest() throws Exception {
+
+OperatorFixture.OperatorFixtureBuilder builder = 
OperatorFixture.builder();
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PASSWORD, 
"root");
+builder.configBuilder().put(ExecConstants.HTTP_KEYSTORE_PATH, "/root");
+try (OperatorFixture fixture = builder.build()) {
+  DrillConfig config = fixture.config();
+  SSLConfig sslv = new SSLConfig(config);
+  assertEquals("root", sslv.getkeystorePassword());
+  assertEquals("/root", sslv.getkeystorePath());
+}
+  }
+}
--- End diff --

@paul-rogers  No, truststore path and truststore password are optional. The 
patch correctly handles the creation of the SSLContext by setting these params 
only if not empty. 


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5701) drill.connections.rpc.. metric not behaving correctly

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia reassigned DRILL-5701:


Assignee: Sorabh Hamirwasia

> drill.connections.rpc.. metric not 
> behaving correctly
> ---
>
> Key: DRILL-5701
> URL: https://issues.apache.org/jira/browse/DRILL-5701
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>
> Reported by [~knguyen]
> When sqlline fails to connect to drillbit such as a handshake failure, the 
> drill.connections.rpc.user. metric subtracts 2 
> counters (one for handshake failure and one when the user exits from 
> sqlline). As a result, the metric can end up with a negative value. For 
> control connections if the handshake fails then again counters related to 
> those are updated incorrectly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5701) drill.connections.rpc.. metric not behaving correctly

[Sorabh Hamirwasia (JIRA)]

Sorabh Hamirwasia created DRILL-5701:


 Summary: 
drill.connections.rpc.. metric not 
behaving correctly
 Key: DRILL-5701
 URL: https://issues.apache.org/jira/browse/DRILL-5701
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Sorabh Hamirwasia


Reported by [~knguyen]

When sqlline fails to connect to drillbit such as a handshake failure, the 
drill.connections.rpc.user. metric subtracts 2 counters 
(one for handshake failure and one when the user exits from sqlline). As a 
result, the metric can end up with a negative value. For control connections if 
the handshake fails then again counters related to those are updated 
incorrectly.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5660) Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111712#comment-16111712
 ] 

ASF GitHub Bot commented on DRILL-5660:
---

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/877#discussion_r130992403
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetMetadataCache.java
 ---
@@ -424,32 +432,121 @@ public void testMoveCache() throws Exception {
 
   @Test
   public void testMetadataCacheAbsolutePaths() throws Exception {
+final String absolutePathsMetadata = "absolute_paths_metadata";
 try {
   test("use dfs_test.tmp");
-  final String relative_path_metadata_t1 = RELATIVE_PATHS_METADATA + 
"/t1";
-  final String relative_path_metadata_t2 = RELATIVE_PATHS_METADATA + 
"/t2";
-  test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
relative_path_metadata_t1);
-  test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
relative_path_metadata_t2);
+  // creating two inner directories to leverage 
METADATA_DIRECTORIES_FILENAME metadata file as well
+  final String absolutePathsMetadataT1 = absolutePathsMetadata + "/t1";
+  final String absolutePathsMetadataT2 = absolutePathsMetadata + "/t2";
+  test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
absolutePathsMetadataT1);
+  test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
absolutePathsMetadataT2);
   
copyMetaDataCacheToTempReplacingInternalPaths("parquet/metadata_with_absolute_path/"
 +
-  "metadata_directories_with_absolute_paths.requires_replace.txt", 
RELATIVE_PATHS_METADATA, Metadata.METADATA_DIRECTORIES_FILENAME);
+  "metadata_directories_with_absolute_paths.requires_replace.txt", 
absolutePathsMetadata, Metadata.METADATA_DIRECTORIES_FILENAME);
   
copyMetaDataCacheToTempReplacingInternalPaths("parquet/metadata_with_absolute_path/"
 +
-  "metadata_table_with_absolute_paths.requires_replace.txt", 
RELATIVE_PATHS_METADATA, Metadata.METADATA_FILENAME);
+  "metadata_table_with_absolute_paths.requires_replace.txt", 
absolutePathsMetadata, Metadata.METADATA_FILENAME);
   
copyMetaDataCacheToTempReplacingInternalPaths("parquet/metadata_with_absolute_path/"
 +
-  "metadata_table_with_absolute_paths_t1.requires_replace.txt", 
relative_path_metadata_t1, Metadata.METADATA_FILENAME);
+  "metadata_table_with_absolute_paths_t1.requires_replace.txt", 
absolutePathsMetadataT1, Metadata.METADATA_FILENAME);
   
copyMetaDataCacheToTempReplacingInternalPaths("parquet/metadata_with_absolute_path/"
 +
-  "metadata_table_with_absolute_paths_t2.requires_replace.txt", 
relative_path_metadata_t2, Metadata.METADATA_FILENAME);
+  "metadata_table_with_absolute_paths_t2.requires_replace.txt", 
absolutePathsMetadataT2, Metadata.METADATA_FILENAME);
+  String query = String.format("select * from %s", 
absolutePathsMetadata);
+  int expectedRowCount = 50;
+  int expectedNumFiles = 1; // point to selectionRoot since no pruning 
is done in this query
+  int actualRowCount = testSql(query);
+  assertEquals("An incorrect result was obtained while querying a 
table with metadata cache files",
+  expectedRowCount, actualRowCount);
+  String numFilesPattern = "numFiles=" + expectedNumFiles;
+  String usedMetaPattern = "usedMetadataFile=true";
+  String cacheFileRootPattern = String.format("cacheFileRoot=%s/%s", 
getDfsTestTmpSchemaLocation(), absolutePathsMetadata);
+  PlanTestBase.testPlanMatchingPatterns(query, new 
String[]{numFilesPattern, usedMetaPattern, cacheFileRootPattern},
+  new String[] {"Filter"});
+} finally {
+  test("drop table if exists %s", absolutePathsMetadata);
+}
+  }
 
-  int rowCount = testSql(String.format("select * from %s", 
RELATIVE_PATHS_METADATA));
-  assertEquals("An incorrect result was obtained while querying a 
table with metadata cache files", 50, rowCount);
+  @Test
+  public void testSpacesInMetadataCachePath() throws Exception {
+final String pathWithSpaces = "path with spaces";
+try {
+  // creating multilevel table to store path with spaces in both 
metadata files (METADATA and METADATA_DIRECTORIES)
+  test("create table dfs_test.tmp.`%s` as select * from 
cp.`tpch/nation.parquet`", pathWithSpaces);
+  test("create table dfs_test.tmp.`%1$s/%1$s` as select * from 
cp.`tpch/nation.parquet`", pathWithSpaces);
+  test("refresh table metadata dfs_test.tmp.`%s`", pathWithSpaces);
+  checkForMetadataFile(pathWithSpaces);
+  String query = 

[jira] [Commented] (DRILL-5660) Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111713#comment-16111713
 ] 

ASF GitHub Bot commented on DRILL-5660:
---

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/877#discussion_r130990507
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetMetadataCache.java
 ---
@@ -495,14 +495,14 @@ public void testFutureUnsupportedMetadataVersion() 
throws Exception {
 try {
   test("use dfs_test.tmp");
   test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
unsupportedMetadataVersion);
-  String lastVersion = 
Iterables.getLast(MetadataVersion.Constants.SUPPORTED_VERSIONS);
-  for (String supportedVersion : 
MetadataVersion.Constants.SUPPORTED_VERSIONS) {
-if (new MetadataVersion(lastVersion).compareTo(new 
MetadataVersion(supportedVersion)) < 0) {
+  MetadataVersion lastVersion = 
Iterables.getLast(MetadataVersion.Constants.SUPPORTED_VERSIONS);
--- End diff --

Added `ImmutableSortedSet` in the last commit. It allows do not worry about 
order of elements.


> Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
> 
>
> Key: DRILL-5660
> URL: https://issues.apache.org/jira/browse/DRILL-5660
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Vitalii Diravka
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Drill recently accepted a PR for the following bug:
> DRILL-3867: Store relative paths in metadata file
> This PR turned out to have a flaw which affects version compatibility.
> The DRILL-3867 PR changed the format of Parquet metadata files to store 
> relative paths. All Drill servers after that PR create files with relative 
> paths. But, the version number of the file is unchanged, so that older 
> Drillbits don't know that they can't understand the file.
> Instead, if an older Drill tries to read the file, queries fail left and 
> right. Drill will resolve the paths, but does so relative to the user's HDFS 
> home directory, which is wrong.
> What should have happened is that we should have bumped the parquet metadata 
> file version number so that older Drillbits can’t read the file. This ticket 
> requests that we do that.
> Now, one could argue that the lack of version number change is fine. Once a 
> user upgrades Drill, they won't use an old Drillbit. But, things are not that 
> simple:
> * A developer tests a branch based on a pre-DRILL-3867 build on a cluster in 
> which metadata files have been created by a post-DRILL-3867 build. (This has 
> already occurred multiple times in our shop.)
> * A user tries to upgrade to Drill 1.11, finds an issue, and needs to roll 
> back to Drill 1.10. Doing so will cause queries to fail due to 
> seemingly-corrupt metadata files.
> * A user tries to do a rolling upgrade: running 1.11 on some servers, 1.10 on 
> others. Once a 1.11 server is installed, the metadata is updated ("corrupted" 
> from the perspective of 1.10) and queries fail.
> Standard practice in this scenario is to:
> * Bump the file version number when the file format changes, and
> * Software refuses to read files with a version newer than the software was 
> designed for.
> Of course, it is highly desirable that newer servers read old files, but that 
> is not the issue here.
> *Main technical points of working of parquet metadata caching for now.*
> Only process of reading the parquet metadata is changed (the process of 
> writing isn't changed):
> +1. Metadata files are valid:+
> Metadata objects are created by deserialization of parquet metadata files in 
> the process of creating ParquetGroupScan physical operator. 
> All supported versions are stored in the "MetadataVersion.Constants" class 
> and in the Jackson annotations for Metadata.ParquetTableMetadataBase class.
> +2. Metadata files version isn't supported (created by newer Drill version). 
> Drill table has at least one metadata file of unsupported version:+
> JsonMappingException is obtained and swallowed without creating metadata 
> object. Error message is logged. The state is stored in MetadataContext, 
> therefore further there will be no attempt to deserialize metadata file again 
> in context of performing current query. The physical plan will be created 
> without using parquet metadata caching. Warning message is logged for every 
> further check "is metadata corrupted".
> +3. Drill table has at least one corrupted metadata file, which can't be 
> deserialized:+
> JsonParseException is obtained. Then the same behaviour as for the 
> unsupported version files.
> +4. The 

[jira] [Commented] (DRILL-5660) Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111711#comment-16111711
 ] 

ASF GitHub Bot commented on DRILL-5660:
---

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/877#discussion_r130973130
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/MetadataVersion.java
 ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.parquet;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ComparisonChain;
+import com.google.common.collect.ImmutableList;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+
+public class MetadataVersion implements Comparable {
+
+  private static final String FORMAT = "v((?!0)\\d+)\\.?((?!0)\\d+)?";
--- End diff --

Sounds good.  Implemented. 
Just one restriction - the last minor version is 9, after that increase 
major version. 


> Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
> 
>
> Key: DRILL-5660
> URL: https://issues.apache.org/jira/browse/DRILL-5660
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Vitalii Diravka
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Drill recently accepted a PR for the following bug:
> DRILL-3867: Store relative paths in metadata file
> This PR turned out to have a flaw which affects version compatibility.
> The DRILL-3867 PR changed the format of Parquet metadata files to store 
> relative paths. All Drill servers after that PR create files with relative 
> paths. But, the version number of the file is unchanged, so that older 
> Drillbits don't know that they can't understand the file.
> Instead, if an older Drill tries to read the file, queries fail left and 
> right. Drill will resolve the paths, but does so relative to the user's HDFS 
> home directory, which is wrong.
> What should have happened is that we should have bumped the parquet metadata 
> file version number so that older Drillbits can’t read the file. This ticket 
> requests that we do that.
> Now, one could argue that the lack of version number change is fine. Once a 
> user upgrades Drill, they won't use an old Drillbit. But, things are not that 
> simple:
> * A developer tests a branch based on a pre-DRILL-3867 build on a cluster in 
> which metadata files have been created by a post-DRILL-3867 build. (This has 
> already occurred multiple times in our shop.)
> * A user tries to upgrade to Drill 1.11, finds an issue, and needs to roll 
> back to Drill 1.10. Doing so will cause queries to fail due to 
> seemingly-corrupt metadata files.
> * A user tries to do a rolling upgrade: running 1.11 on some servers, 1.10 on 
> others. Once a 1.11 server is installed, the metadata is updated ("corrupted" 
> from the perspective of 1.10) and queries fail.
> Standard practice in this scenario is to:
> * Bump the file version number when the file format changes, and
> * Software refuses to read files with a version newer than the software was 
> designed for.
> Of course, it is highly desirable that newer servers read old files, but that 
> is not the issue here.
> *Main technical points of working of parquet metadata caching for now.*
> Only process of reading the parquet metadata is changed (the process of 
> writing isn't changed):
> +1. Metadata files are valid:+
> Metadata objects are created by deserialization of parquet metadata files in 
> the process of creating ParquetGroupScan physical operator. 
> All supported versions are stored in the "MetadataVersion.Constants" class 
> and in the Jackson annotations for 

[jira] [Commented] (DRILL-5660) Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111710#comment-16111710
 ] 

ASF GitHub Bot commented on DRILL-5660:
---

Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/877#discussion_r130991076
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestParquetMetadataCache.java
 ---
@@ -495,14 +495,14 @@ public void testFutureUnsupportedMetadataVersion() 
throws Exception {
 try {
   test("use dfs_test.tmp");
   test("create table `%s` as select * from cp.`tpch/nation.parquet`", 
unsupportedMetadataVersion);
-  String lastVersion = 
Iterables.getLast(MetadataVersion.Constants.SUPPORTED_VERSIONS);
-  for (String supportedVersion : 
MetadataVersion.Constants.SUPPORTED_VERSIONS) {
-if (new MetadataVersion(lastVersion).compareTo(new 
MetadataVersion(supportedVersion)) < 0) {
+  MetadataVersion lastVersion = 
Iterables.getLast(MetadataVersion.Constants.SUPPORTED_VERSIONS);
+  for (MetadataVersion supportedVersion : 
MetadataVersion.Constants.SUPPORTED_VERSIONS) {
+if (lastVersion.compareTo(supportedVersion) < 0) {
   lastVersion = supportedVersion;
 }
   }
   // Get the future version, which is absent in 
MetadataVersion.SUPPORTED_VERSIONS list
-  String futureVersion = "v" + 
(Integer.parseInt(String.valueOf(lastVersion.charAt(1))) + 1);
+  String futureVersion = "v" + 
(Integer.parseInt(String.valueOf(lastVersion.toString().charAt(1))) + 1);
--- End diff --

This code was replaced since float metadata versions are introduced.


> Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
> 
>
> Key: DRILL-5660
> URL: https://issues.apache.org/jira/browse/DRILL-5660
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Vitalii Diravka
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Drill recently accepted a PR for the following bug:
> DRILL-3867: Store relative paths in metadata file
> This PR turned out to have a flaw which affects version compatibility.
> The DRILL-3867 PR changed the format of Parquet metadata files to store 
> relative paths. All Drill servers after that PR create files with relative 
> paths. But, the version number of the file is unchanged, so that older 
> Drillbits don't know that they can't understand the file.
> Instead, if an older Drill tries to read the file, queries fail left and 
> right. Drill will resolve the paths, but does so relative to the user's HDFS 
> home directory, which is wrong.
> What should have happened is that we should have bumped the parquet metadata 
> file version number so that older Drillbits can’t read the file. This ticket 
> requests that we do that.
> Now, one could argue that the lack of version number change is fine. Once a 
> user upgrades Drill, they won't use an old Drillbit. But, things are not that 
> simple:
> * A developer tests a branch based on a pre-DRILL-3867 build on a cluster in 
> which metadata files have been created by a post-DRILL-3867 build. (This has 
> already occurred multiple times in our shop.)
> * A user tries to upgrade to Drill 1.11, finds an issue, and needs to roll 
> back to Drill 1.10. Doing so will cause queries to fail due to 
> seemingly-corrupt metadata files.
> * A user tries to do a rolling upgrade: running 1.11 on some servers, 1.10 on 
> others. Once a 1.11 server is installed, the metadata is updated ("corrupted" 
> from the perspective of 1.10) and queries fail.
> Standard practice in this scenario is to:
> * Bump the file version number when the file format changes, and
> * Software refuses to read files with a version newer than the software was 
> designed for.
> Of course, it is highly desirable that newer servers read old files, but that 
> is not the issue here.
> *Main technical points of working of parquet metadata caching for now.*
> Only process of reading the parquet metadata is changed (the process of 
> writing isn't changed):
> +1. Metadata files are valid:+
> Metadata objects are created by deserialization of parquet metadata files in 
> the process of creating ParquetGroupScan physical operator. 
> All supported versions are stored in the "MetadataVersion.Constants" class 
> and in the Jackson annotations for Metadata.ParquetTableMetadataBase class.
> +2. Metadata files version isn't supported (created by newer Drill version). 
> Drill table has at least one metadata file of unsupported version:+
> JsonMappingException is obtained and swallowed without creating metadata 
> object. Error message is 

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111346#comment-16111346
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user sindhurirayavaram commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r130939205
  
--- Diff: distribution/src/resources/drill-override-example.conf ---
@@ -214,7 +214,7 @@ drill.exec: {
 }
 
 # Below SSL parameters need to be set for custom transport layer settings.
-javax.net.ssl {
+ssl{
   keyStore: "/keystore.file",
--- End diff --

Can you look into this paul? This an example I got. 
SSL certificate keystore location.
Default: conf/default.keystore.
ssl.keystore.path can be configured in 2 ways:
By using the relative path, for example, 
ssl.keystore.path=conf/default.keystore, where the root directory is \server\.
By using the absolute path, for example, 
ssl.keystore.path=/opt/wl/default.keystore for UNIX, or 
C:\\wl\\default.keystore for Windows.


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5645) negation of expression causes null pointer exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111257#comment-16111257
 ] 

ASF GitHub Bot commented on DRILL-5645:
---

GitHub user josiahyan opened a pull request:

https://github.com/apache/drill/pull/892

DRILL-5645: negation of expression causes null pointer exception

Fixes DRILL-5645, and another test case that got caught up in the change 
due to type issues.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/josiahyan/drill DRILL-5645

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #892


commit 8ef634e2204a48fe7dd35d212cb0073fdf41aa35
Author: Josiah Yan 
Date:   2017-08-02T14:50:54Z

DRILL-5645: negation of expression causes null pointer exception




> negation of expression causes null pointer exception
> 
>
> Key: DRILL-5645
> URL: https://issues.apache.org/jira/browse/DRILL-5645
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.10.0
> Environment: Drill 1.10
>Reporter: N Campbell
>
> Following statement will fail when the expression is negated
> select -(2 * 2) from ( values ( 1 ) ) T ( C1 )
> Error: SYSTEM ERROR: NullPointerException



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5687) Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5687:

Labels: ready-to-commit  (was: )

> Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns
> ---
>
> Key: DRILL-5687
> URL: https://issues.apache.org/jira/browse/DRILL-5687
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns until 
> DRILL-5612 is not fixed since the failure is random and can cause unit tests 
> to fail with further necessity to re-run the unit tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5687) Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1668#comment-1668
 ] 

ASF GitHub Bot commented on DRILL-5687:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/885
  
Yes, we have. I have included that Jira number in ignore annotation: 
`@Ignore("DRILL-5612")`.


> Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns
> ---
>
> Key: DRILL-5687
> URL: https://issues.apache.org/jira/browse/DRILL-5687
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.12.0
>
>
> Disable TestMergeJoinWithSchemaChanges#testMissingAndNewColumns until 
> DRILL-5612 is not fixed since the failure is random and can cause unit tests 
> to fail with further necessity to re-run the unit tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5645) negation of expression causes null pointer exception

[Josiah Yan (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111038#comment-16111038
 ] 

Josiah Yan commented on DRILL-5645:
---

Reproduced, getting stacktrace

{code}
[Error Id: bea50afd-515c-4afb-be0f-b07849945dee on psf806i3:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException


[Error Id: bea50afd-515c-4afb-be0f-b07849945dee on psf806i3:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:847)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:977) 
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:297) 
[drill-java-exec-1.11.0.jar:1.11.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_131]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Internal error: Error while applying 
rule ReduceExpressionsRule_Project, args [rel#12:L
ogicalProject.NONE.ANY([]).[](input=rel#11:Subset#0.NONE.ANY([]).[0],EXPR$0=-(*(2,
 2)))]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule 
ReduceExpressionsRule_Project, args 
[rel#12:LogicalProject.NONE.ANY([]).[](input=rel#11:Subset#0.NONE.ANY([]).[0],EXPR$0=-(*(2,
 2)))]
at org.apache.calcite.util.Util.newInternal(Util.java:792) 
~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
 ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:811)
 ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:310) 
~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:401)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:242)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:292)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:169)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050) 
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:280) 
[drill-java-exec-1.11.0.jar:1.11.0]
... 3 common frames omitted
Caused by: java.lang.NullPointerException: null
at 
org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:170)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:97)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) 
~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:94) 
~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:81) 
~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:124)
 ~[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:502)
 ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.calcite.rel.rules.ReduceExpressionsRule$1.onMatch(ReduceExpressionsRule.java:243)
 ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
 ~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
... 14 

[jira] [Updated] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5675:

Reviewer: Arina Ielchiieva

> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5675:

Fix Version/s: 1.12.0

> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5675:
---

Assignee: (was: Arina Ielchiieva)

> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5675:

Labels: ready-to-commit  (was: )

> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110723#comment-16110723
 ] 

ASF GitHub Bot commented on DRILL-5675:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/878
  
+1, LGTM.


> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5675) Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5675:
---

Assignee: Arina Ielchiieva

> Drill C++ Client Date Time Literals Support Metadata Mapping is Incorrect
> -
>
> Key: DRILL-5675
> URL: https://issues.apache.org/jira/browse/DRILL-5675
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Rob Wu
>Assignee: Arina Ielchiieva
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Small mistake in the dateTimeLiteralsSupport(1) function.
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L839
> and 
> https://github.com/laurentgo/drill/blob/f5f6bfdf7e1c9e97594956ef36c08cf2cfc9ddba/contrib/native/client/src/clientlib/metadata.cpp#L844



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5698:

Reviewer: Arina Ielchiieva

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5691) multiple count distinct query planning error at physical phase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110687#comment-16110687
 ] 

ASF GitHub Bot commented on DRILL-5691:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/889
  
@weijietong thanks for the PR.
1. Is it possible to add unit tests?
2. Can you please add in method description that a table scan with only one 
row should also be considered as a scalar as well.
3. Please fix indention in `if` statements and change `currentrel` to camel 
case [1].

[1] https://drill.apache.org/docs/apache-drill-contribution-guidelines/



> multiple count distinct query planning error at physical phase 
> ---
>
> Key: DRILL-5691
> URL: https://issues.apache.org/jira/browse/DRILL-5691
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.9.0, 1.10.0
>Reporter: weijie.tong
>
> I materialized the count distinct query result in a cache , added a plugin 
> rule to translate the (Aggregate、Aggregate、Project、Scan) or 
> (Aggregate、Aggregate、Scan) to (Project、Scan) at the PARTITION_PRUNING phase. 
> Then ,once user issue count distinct queries , it will be translated to query 
> the cache to get the result.
> eg1: " select count(*),sum(a) ,count(distinct b)  from t where dt=xx " 
> eg2:"select count(*),sum(a) ,count(distinct b) ,count(distinct c) from t 
> where dt=xxx "
> eg3:"select count(distinct b), count(distinct c) from t where dt=xxx"
> eg1 will be right and have a query result as I expected , but eg2 will be 
> wrong at the physical phase.The error info is here: 
> https://gist.github.com/weijietong/1b8ed12db9490bf006e8b3fe0ee52269. 
> eg3 will also get the similar error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5698:
---

Assignee: (was: Arina Ielchiieva)

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5698:

Fix Version/s: 1.12.0

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5698:

Labels: ready-to-commit  (was: )

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5698) Drill should start in embedded mode using java 1.8.0_144

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5698:
---

Assignee: Arina Ielchiieva

> Drill should start in embedded mode using java 1.8.0_144
> 
>
> Key: DRILL-5698
> URL: https://issues.apache.org/jira/browse/DRILL-5698
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Darren
>Assignee: Arina Ielchiieva
>
> Currently the start up script
> distribution/src/resources/drill-config.sh
> prevents drill from starting as the regex incorrectly captures the 144 
> portion of the version code
> this is because the regex isn't escaped 
> -"$JAVA" -version 2>&1 | grep "version" | egrep -e "1.4|1.5|1.6" > /dev/null
>  +"$JAVA" -version 2>&1 | grep "version" | egrep -e "1\.4|1\.5|1\.6" > 
> /dev/null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4790) DATA_READ ERROR: Failure while attempting to read from database.

[Abou Haydar Elias (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110505#comment-16110505
 ] 

Abou Haydar Elias commented on DRILL-4790:
--

The following issue : 
{code:java}
Caused by: java.sql.SQLException: Value '-00-00 00:00:00' can not be 
represented as java.sql.Timestamp
{code}

Is actually related to the jdbc connector, and like suggested by [~xwinie] 
adding *?zeroDateTimeBehavior=convertToNull* to the plugin storage url should 
solve this issue.

Thus there is no actually issue here.


> DATA_READ ERROR: Failure while attempting to read from database.
> 
>
> Key: DRILL-4790
> URL: https://issues.apache.org/jira/browse/DRILL-4790
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.6.0
> Environment: centos mysql56  
>Reporter: xwinie
> Fix For: Future
>
>
> 2016-07-20 10:08:51,318 [287122cc-54de-ad19-8235-a2ee1bc5cbc0:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 287122cc-54de-ad19-8235-a2ee1bc5cbc0:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-07-20 10:08:51,319 [287122cc-54de-ad19-8235-a2ee1bc5cbc0:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 287122cc-54de-ad19-8235-a2ee1bc5cbc0:0:0: State to report: RUNNING
> 2016-07-20 10:09:00,118 [287122cc-54de-ad19-8235-a2ee1bc5cbc0:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 287122cc-54de-ad19-8235-a2ee1bc5cbc0:0:0: State change requested RUNNING --> 
> FINISHED
> 2016-07-20 10:09:00,119 [287122cc-54de-ad19-8235-a2ee1bc5cbc0:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 287122cc-54de-ad19-8235-a2ee1bc5cbc0:0:0: State to report: FINISHED
> 2016-07-20 10:09:20,058 [287122ae-8f8c-b370-08fa-d9e6aa6566dd:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 287122ae-8f8c-b370-08fa-d9e6aa6566dd: select * from mysqlplugin.mfm.sys_dict
> 2016-07-20 10:09:20,130 [287122ae-8f8c-b370-08fa-d9e6aa6566dd:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 287122ae-8f8c-b370-08fa-d9e6aa6566dd:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-07-20 10:09:20,131 [287122ae-8f8c-b370-08fa-d9e6aa6566dd:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 287122ae-8f8c-b370-08fa-d9e6aa6566dd:0:0: State to report: RUNNING
> 2016-07-20 10:09:20,141 [287122ae-8f8c-b370-08fa-d9e6aa6566dd:frag:0:0] INFO  
> o.a.d.e.store.jdbc.JdbcRecordReader - User Error Occurred
> org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Failure 
> while attempting to read from database.
> sql SELECT *
> FROM `mfm`.`sys_dict`
> plugin mysqlplugin
> [Error Id: cf1ef1a8-2e88-4ce3-8e51-129f6e3b7f83 ]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:247)
>  [drill-jdbc-storage-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:178) 
> [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:257)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:251)
>  [drill-java-exec-1.7.0.jar:1.7.0]
>   at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.7.0_101]
>   at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_101]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>  [hadoop-common-2.7.1.jar:na]
>   at 

[jira] [Commented] (DRILL-5700) nohup support for sqlline

[Arjun KR (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110374#comment-16110374
 ] 

Arjun KR commented on DRILL-5700:
-

The issue seems to be related to JLine used with sqlline executed on 
background. Similar issue is reported with Hive beeline script and is fixed by 
adding jvm option '-Djline.terminal=jline.UnsupportedTerminal"'. This could be 
added in sqlline script as below.

{code:java}
if [[ ( ! $(ps -o stat= -p $$) =~ "+" ) && ! ( -p /dev/stdin ) ]]; then
export SQLLINE_JAVA_OPTS="$SQLLINE_JAVA_OPTS 
-Djline.terminal=jline.UnsupportedTerminal"
fi
SHELL_OPTS="$DRILL_SHELL_JAVA_OPTS $SQLLINE_JAVA_OPTS $DRILL_SHELL_LOG_OPTS 
$CLIENT_GC_OPTS"
CMD="$JAVA $SHELL_OPTS -cp $CP sqlline.SqlLine -d org.apache.drill.jdbc.Driver 
--maxWidth=1"

{code}


> nohup support for sqlline 
> --
>
> Key: DRILL-5700
> URL: https://issues.apache.org/jira/browse/DRILL-5700
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - CLI
>Reporter: Arjun KR
>Priority: Minor
>
> Sqlline script does support nohup mode for execution. On execution, it 
> remains stopped until it is made fore ground.
> {code:java}
> [mapr@node1 ~]$ cat test.sql
> select * from sys.drillbits
> [mapr@node1 ~]$
> [mapr@node1 ~]$ nohup sqlline -u "jdbc:drill:" -n mapr -p mapr -f test.sql  &
> [1] 24019
> [mapr@node1 ~]$ nohup: ignoring input and appending output to `nohup.out'
> [1]+  Stopped nohup sqlline -u "jdbc:drill:zk=node1:5181" -n 
> mapr -p mapr -f test.sql
> [mapr@node1 ~]$
> [mapr@node1 ~]$ fg
> nohup sqlline -u "jdbc:drill:zk=node1:5181" -n mapr -p mapr -f test.sql
> [mapr@node1 ~]$
> [mapr@node1 ~]$ cat nohup.out
> 0: jdbc:drill:zk=node1:5181> Closing: 
> org.apache.drill.jdbc.impl.DrillConnectionImpl
> output of ps: S
> 1/1  select * from sys.drillbits;
> +--++---++--+--+
> | hostname | user_port  | control_port  | data_port  | current  | 
> version  |
> +--++---++--+--+
> | node1  | 31010  | 31011 | 31012  | true | 1.10.0   |
> +--++---++--+--+
> 1 row selected (0.354 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.10.0
> "drill baby drill"
> [mapr@node1 ~]$
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5700) nohup support for sqlline

[Arjun KR (JIRA)]

Arjun KR created DRILL-5700:
---

 Summary: nohup support for sqlline 
 Key: DRILL-5700
 URL: https://issues.apache.org/jira/browse/DRILL-5700
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - CLI
Reporter: Arjun KR
Priority: Minor


Sqlline script does support nohup mode for execution. On execution, it remains 
stopped until it is made fore ground.

{code:java}
[mapr@node1 ~]$ cat test.sql
select * from sys.drillbits
[mapr@node1 ~]$
[mapr@node1 ~]$ nohup sqlline -u "jdbc:drill:" -n mapr -p mapr -f test.sql  &
[1] 24019
[mapr@node1 ~]$ nohup: ignoring input and appending output to `nohup.out'


[1]+  Stopped nohup sqlline -u "jdbc:drill:zk=node1:5181" -n 
mapr -p mapr -f test.sql
[mapr@node1 ~]$
[mapr@node1 ~]$ fg
nohup sqlline -u "jdbc:drill:zk=node1:5181" -n mapr -p mapr -f test.sql
[mapr@node1 ~]$
[mapr@node1 ~]$ cat nohup.out
0: jdbc:drill:zk=node1:5181> Closing: 
org.apache.drill.jdbc.impl.DrillConnectionImpl
output of ps: S
1/1  select * from sys.drillbits;
+--++---++--+--+
| hostname | user_port  | control_port  | data_port  | current  | 
version  |
+--++---++--+--+
| node1  | 31010  | 31011 | 31012  | true | 1.10.0   |
+--++---++--+--+
1 row selected (0.354 seconds)
Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
apache drill 1.10.0
"drill baby drill"
[mapr@node1 ~]$
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)