date:20161007

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557275#comment-15557275
 ] 

ASF GitHub Bot commented on DRILL-4870:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/605#discussion_r82496282
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -392,5 +392,7 @@ export DRILL_HOME
 export DRILL_CONF_DIR
 export DRILL_LOG_DIR
 export CP
-export JAVA_HOME
+# DRILL-4870: Don't export JAVA_HOME. Java can find it just fine from the 
java
+# command. If we attempt to work out out, we do so incorrectly for the Mac.
--- End diff --

+1. Looks good.


> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557189#comment-15557189
 ] 

ASF GitHub Bot commented on DRILL-4870:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/605#discussion_r82495596
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -392,5 +392,7 @@ export DRILL_HOME
 export DRILL_CONF_DIR
 export DRILL_LOG_DIR
 export CP
-export JAVA_HOME
+# DRILL-4870: Don't export JAVA_HOME. Java can find it just fine from the 
java
+# command. If we attempt to work out out, we do so incorrectly for the Mac.
--- End diff --

Yikes! Fixed the typo.

Changed the reference to JAVA_HOME to a reference relative to java, which 
we must already know.


> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557168#comment-15557168
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495497
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 ---
@@ -46,40 +46,83 @@
   // max off heap memory for planning (16G)
   private static final long MAX_OFF_HEAP_ALLOCATION_IN_BYTES = 16l * 1024 
* 1024 * 1024;
 
-  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true);
+  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true,
+  "If one side of a filter condition is a constant expression, 
constant folding evaluates the expression in the" +
+  " planning phase and replaces the expression with the constant 
value. For example, Drill can rewrite" +
+  " this clause ' WHERE age + 5 < 42 as WHERE age < 37'.");
+
   public static final OptionValidator EXCHANGE = new 
BooleanValidator("planner.disable_exchanges", false);
+
   public static final OptionValidator HASHAGG = new 
BooleanValidator("planner.enable_hashagg", true);
+
   public static final OptionValidator STREAMAGG = new 
BooleanValidator("planner.enable_streamagg", true);
-  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true);
-  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true);
+
+  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true,
+  "Enable the memory hungry hash join. Drill assumes that a query with 
have adequate memory to complete and" +
--- End diff --

with have --> will have


> Add Description Column in sys.options
> -
>
> Key: DRILL-4699
> URL: https://issues.apache.org/jira/browse/DRILL-4699
> Project: Apache Drill
>  Issue Type: Improvement
>  Components:  Server, Documentation
>Affects Versions: 1.6.0
>Reporter: John Omernik
>Assignee: Paul Rogers
>
> select * from sys.options provides a user with a strong understanding of what 
> options are available to Drill. These options are not well documented.  Some 
> options are "experimental" other options have a function only in specific 
> cases (writers vs readers for example).  If we had a large text field for 
> description, we could enforce documentation of the settings are option 
> creation time, and the description of the setting could change as the 
> versions change (i.e. when an option graduates to being supported from being 
> experimental, it would be changed in the version the user is using. I.e. when 
> they run select * from sys.options, they know the exact state of the option 
> every time they query. It could also facilitate better self documentation via 
> QA on pull requests "Did you update the sys.options.desc?"  This makes it 
> easier for users, and admins in the use of Drill in an enterprise.
> The first step is adding the field, and then going back and filling in the 
> desc for each option.  (Another JIRA after the option is available)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557165#comment-15557165
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485619
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS =

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557174#comment-15557174
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495511
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 ---
@@ -46,40 +46,83 @@
   // max off heap memory for planning (16G)
   private static final long MAX_OFF_HEAP_ALLOCATION_IN_BYTES = 16l * 1024 
* 1024 * 1024;
 
-  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true);
+  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true,
+  "If one side of a filter condition is a constant expression, 
constant folding evaluates the expression in the" +
+  " planning phase and replaces the expression with the constant 
value. For example, Drill can rewrite" +
+  " this clause ' WHERE age + 5 < 42 as WHERE age < 37'.");
+
   public static final OptionValidator EXCHANGE = new 
BooleanValidator("planner.disable_exchanges", false);
+
   public static final OptionValidator HASHAGG = new 
BooleanValidator("planner.enable_hashagg", true);
+
   public static final OptionValidator STREAMAGG = new 
BooleanValidator("planner.enable_streamagg", true);
-  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true);
-  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true);
+
+  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true,
+  "Enable the memory hungry hash join. Drill assumes that a query with 
have adequate memory to complete and" +
+  " tries to use the fastest operations possible to complete the 
planned inner, left, right, or full outer" +
+  " joins using a hash table. Does not write to disk. Disabling 
hash join allows Drill to manage arbitrarily" +
+  " large data in a small memory footprint.");
+
+  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true,
+  "Sort-based operation. A merge join is used for inner join, left and 
right outer joins. Inputs to the merge" +
+  " join must be sorted. It reads the sorted input streams from 
both sides and finds matching rows." +
+  " Writes to disk.");
+
   public static final OptionValidator NESTEDLOOPJOIN = new 
BooleanValidator("planner.enable_nestedloopjoin", true);
+
   public static final OptionValidator MULTIPHASE = new 
BooleanValidator("planner.enable_multiphase_agg", true);
-  public static final OptionValidator BROADCAST = new 
BooleanValidator("planner.enable_broadcast_join", true);
-  public static final OptionValidator BROADCAST_THRESHOLD = new 
PositiveLongValidator("planner.broadcast_threshold", MAX_BROADCAST_THRESHOLD, 
1000);
-  public static final OptionValidator BROADCAST_FACTOR = new 
RangeDoubleValidator("planner.broadcast_factor", 0, Double.MAX_VALUE, 1.0d);
-  public static final OptionValidator NESTEDLOOPJOIN_FACTOR = new 
RangeDoubleValidator("planner.nestedloopjoin_factor", 0, Double.MAX_VALUE, 
100.0d);
-  public static final OptionValidator NLJOIN_FOR_SCALAR = new 
BooleanValidator("planner.enable_nljoin_for_scalar_only", true);
-  public static final OptionValidator JOIN_ROW_COUNT_ESTIMATE_FACTOR = new 
RangeDoubleValidator("planner.join.row_count_estimate_factor", 0, 
Double.MAX_VALUE, 1.0d);
+
+  public static final OptionValidator BROADCAST = new 
BooleanValidator("planner.enable_broadcast_join", true,
+  "The broadcast join can be used for hash join, merge join and nested 
loop join. Use to join a large (fact)" +
+  " table to relatively smaller (dimension) tables. This should be 
enabled.");
+  public static final OptionValidator BROADCAST_THRESHOLD = new 
PositiveLongValidator("planner.broadcast_threshold",
+  MAX_BROADCAST_THRESHOLD, 1000, "The maximum number of records 
allowed to be broadcast as part of a query." +
+  " If the threshold is exceeded, Drill reshuffles data rather than 
doing a broadcast to one side of the" +
+  " join. Range: 0 - " + MAX_BROADCAST_THRESHOLD + ".");
+  public static final OptionValidator BROADCAST_FACTOR = new 
RangeDoubleValidator("planner.broadcast_factor", 0,
+  Double.MAX_VALUE, 1.0d, "A heuristic parameter for influencing the 
broadcast of records as part of a query.");
--- End diff --

Increasing the value does... what?


> Add Description Column in sys.options
>

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557171#comment-15557171
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495346
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557173#comment-15557173
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485277
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS =

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557169#comment-15557169
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495422
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
+  " Range: 0 - 9223372036854775807.");
 
-  String ENABLE_VERBOSE_ERRORS_KEY = "exec.errors.verbose";
-  OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false);
+  public static final String ENABLE_VERBOSE_ERRORS_KEY = 
"exec.errors.verbose";
+  public static final OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false,
+  "Toggles verbose output of error messages.");
 
-  String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
-  OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
+  public static final String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
+  public static final OptionValidator ENABLE_NEW_TEXT_READER = new

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557170#comment-15557170
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495404
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
--- End diff --

Units? Secs? ms?


> Add Description Column in sys.options
> -
>
> Key: DRILL-4699
> URL: https://issues.apache.org/jira/browse/DRILL-4699
> Project: Apache Drill
>  Issue Type: Improvement
>  Components:  Server, Documentation
>Affects Versions: 1.6.0
>Reporter: John Omernik
>Assignee: Paul Rogers
>
> select * from sys.options provides a user with a strong understanding of what 
> options are available to Drill. These options are not well documented.  Some 
> options are "experimental" other options have a function only in specific 
> cases (writers vs readers for example).  If we had a large text field for 
> description, we could enforce documentation of the settings are option 
> creation time, and the

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557163#comment-15557163
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495479
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/QueryClassLoader.java
 ---
@@ -42,7 +42,17 @@
   private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(QueryClassLoader.class);
 
   public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
-  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE =
+  new LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256 * 1024,
+  "See the " + JAVA_COMPILER_OPTION + ". Accepts inputs of type 
LONG.");
--- End diff --

LONG --> long. (Accepts the scalar, not the boxed object.)


> Add Description Column in sys.options
> -
>
> Key: DRILL-4699
> URL: https://issues.apache.org/jira/browse/DRILL-4699
> Project: Apache Drill
>  Issue Type: Improvement
>  Components:  Server, Documentation
>Affects Versions: 1.6.0
>Reporter: John Omernik
>Assignee: Paul Rogers
>
> select * from sys.options provides a user with a strong understanding of what 
> options are available to Drill. These options are not well documented.  Some 
> options are "experimental" other options have a function only in specific 
> cases (writers vs readers for example).  If we had a large text field for 
> description, we could enforce documentation of the settings are option 
> creation time, and the description of the setting could change as the 
> versions change (i.e. when an option graduates to being supported from being 
> experimental, it would be changed in the version the user is using. I.e. when 
> they run select * from sys.options, they know the exact state of the option 
> every time they query. It could also facilitate better self documentation via 
> QA on pull requests "Did you update the sys.options.desc?"  This makes it 
> easier for users, and admins in the use of Drill in an enterprise.
> The first step is adding the field, and then going back and filling in the 
> desc for each option.  (Another JIRA after the option is available)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557177#comment-15557177
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495464
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
+  " Range: 0 - 9223372036854775807.");
 
-  String ENABLE_VERBOSE_ERRORS_KEY = "exec.errors.verbose";
-  OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false);
+  public static final String ENABLE_VERBOSE_ERRORS_KEY = 
"exec.errors.verbose";
+  public static final OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false,
+  "Toggles verbose output of error messages.");
 
-  String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
-  OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
+  public static final String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
+  public static final OptionValidator ENABLE_NEW_TEXT_READER = new

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557172#comment-15557172
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485322
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS =

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557175#comment-15557175
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495374
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557167#comment-15557167
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495327
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
--- End diff --

Instead of saying that this takes a string, can we generate that part? We 
look up the default value, infer the type and say:

The column label ... Type: string, Default: "dir"

Or, add type and default as additional columns. (Or maybe the default does 
not matter, only the current value, so just list the type.)


> Add Description Column in sys.options
> -
>
> Key: DRILL-4699
> URL: https://issues.apache.org/jira/browse/DRILL-4699
> Project: Apache Drill
>  Issue Type: Improvement
>  Components:  Server, Documentation
>Affects Versions: 1.6.0
>Reporter: John Omernik
>Assignee: Paul Rogers
>
> select * from sys.options provides a user with a strong understanding of what 
> options are available to Drill. These options are not well documented.  Some 
> options are "experimental" other options have a function only in specific 
> cases (writers vs readers for example).  If we had a large text field for 
> description, we could enforce documentation of the settings are option 
> creation time, and the description of the setting could change as the 
> versions change (i.e. when an option graduates to being supported from being 
> experimental, it would be changed in the version the user is using. I.e. when 
> they run select * from sys.options, they know the exact state of the option 
> every time they query. It could also facilitate better self documentation via 
> QA on pull requests "Did you update the sys.options.desc?"  This makes it 
> easier for users, and admins in the use of Drill in an enterprise.
> The first step is adding the field, and then going back and filling in the 
> desc for each option.  (Another JIRA after the option is available)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557166#comment-15557166
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485491
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS =

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557164#comment-15557164
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495337
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557176#comment-15557176
 ] 

ASF GitHub Bot commented on DRILL-4699:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495353
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator

[jira] [Commented] (DRILL-3178) csv reader should allow newlines inside quotes

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557029#comment-15557029
 ] 

ASF GitHub Bot commented on DRILL-3178:
---

Github user fmethot commented on the issue:

https://github.com/apache/drill/pull/593
  
Thanks for the comments so far, See my newer changes,  I suggest that we 
remove the flag and add an extra method instead.
There is no more check for a boolean in nextChar, but instead there is an 
extra method call (readNext()->readNextNoNewLineCheck())



> csv reader should allow newlines inside quotes 
> ---
>
> Key: DRILL-3178
> URL: https://issues.apache.org/jira/browse/DRILL-3178
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Text & CSV
>Affects Versions: 1.0.0
> Environment: Ubuntu Trusty 14.04.2 LTS
>Reporter: Neal McBurnett
>Assignee: F Méthot
> Fix For: Future
>
> Attachments: drill-3178.patch
>
>
> When reading a csv file which contains newlines within quoted strings, e.g. 
> via
> select * from dfs.`/tmp/q.csv`;
> Drill 1.0 says:
> Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException:  
> Error processing input: Cannot use newline character within quoted string
> But many tools produce csv files with newlines in quoted strings.  Drill 
> should be able to handle them.
> Workaround: the csvquote program (https://github.com/dbro/csvquote) can 
> encode embedded commas and newlines, and even decode them later if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4841) Use user server event loop group for web clients

2016-10-07 Thread Sorabh Hamirwasia (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia reassigned DRILL-4841:


Assignee: Sudheesh Katkam  (was: Sorabh Hamirwasia)

Assigning to Sudheesh for resolving review comments.

> Use user server event loop group for web clients
> 
>
> Key: DRILL-4841
> URL: https://issues.apache.org/jira/browse/DRILL-4841
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - HTTP
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
>Priority: Minor
>
> Currently we spawn an event loop group for handling requests from clients. 
> This group should also be used to handles responses (from server) for web 
> clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4876) Remain disconnected connection

2016-10-07 Thread Sorabh Hamirwasia (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-4876:
-
Attachment: DrillClientConnectionResetTest.java

Test to show usage with JDBC connection and effect of loosing connection with 
foreman Drillbit.

> Remain disconnected connection
> --
>
> Key: DRILL-4876
> URL: https://issues.apache.org/jira/browse/DRILL-4876
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.6.0, 1.7.0, 1.8.0
> Environment: CentOS 7
>Reporter: Takuya Kojima
>Assignee: Sorabh Hamirwasia
>Priority: Minor
> Attachments: 1_normal.png, 2_after_restart.png, 
> 3_try_to_connect_after_restart.png, 
> 4_disconnected_after_minEvictableIdleTimeMillis.png, 
> 5_after_disconnected.png, DrillClientConnectionResetTest.java, 
> drill-connection-pool.txt
>
>
> I'm using drill via Java Application on Tomcat with JDBC driver.
> I found that disconnected connection is not released when restart a drillbit.
> Drillbit is restarted, but JDBC's connection keeps to try to connect the 
> connection which started before restart.
> Expected behavior is its connection release and reconnect when drillbit is 
> restarted, but as a matter of fact, the connection will be released after 
> elapsed time of "minEvictableIdleTimeMillis" setting.
> As a result, the application can't connect in the meantime despite drillbit 
> is active.
> I thought this is not major issue, but Postgres and Vertica's JDBC driver 
> works well in the same situation. I spend the much time to identify the 
> cause, so I create a new issue of this.
> The attached is log and JMX's monitor graph with 1.6.0's JDBC driver, but I 
> also get it with 1.7.0 and 1.8.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4876) Remain disconnected connection

2016-10-07 Thread Sorabh Hamirwasia (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556735#comment-15556735
 ] 

Sorabh Hamirwasia commented on DRILL-4876:
--

Hi Takuya,
Thanks for raising this issue. Today the way JDBC driver works is that once it 
looses connection with the drillbit it comes to know about it by the next query 
which is executed and fails. The application has to reconnect using new JDBC 
connection to be able to execute new queries. Part of the reason is we don't 
store any session info for each connection which is needed in cases like - "If 
an application set's a schema before executing any query". So just reconnecting 
will not help here as we will loose all the session specific information. 
Definitely that would be a great enhancement which we can target for future but 
unfortunately it's not supported today.

PFA a small test which I wrote to demonstrate the flow or usage using JDBC 
connection. Let me know if you have any other question.

Thanks,
Sorabh

> Remain disconnected connection
> --
>
> Key: DRILL-4876
> URL: https://issues.apache.org/jira/browse/DRILL-4876
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.6.0, 1.7.0, 1.8.0
> Environment: CentOS 7
>Reporter: Takuya Kojima
>Assignee: Sorabh Hamirwasia
>Priority: Minor
> Attachments: 1_normal.png, 2_after_restart.png, 
> 3_try_to_connect_after_restart.png, 
> 4_disconnected_after_minEvictableIdleTimeMillis.png, 
> 5_after_disconnected.png, DrillClientConnectionResetTest.java, 
> drill-connection-pool.txt
>
>
> I'm using drill via Java Application on Tomcat with JDBC driver.
> I found that disconnected connection is not released when restart a drillbit.
> Drillbit is restarted, but JDBC's connection keeps to try to connect the 
> connection which started before restart.
> Expected behavior is its connection release and reconnect when drillbit is 
> restarted, but as a matter of fact, the connection will be released after 
> elapsed time of "minEvictableIdleTimeMillis" setting.
> As a result, the application can't connect in the meantime despite drillbit 
> is active.
> I thought this is not major issue, but Postgres and Vertica's JDBC driver 
> works well in the same situation. I spend the much time to identify the 
> cause, so I create a new issue of this.
> The attached is log and JMX's monitor graph with 1.6.0's JDBC driver, but I 
> also get it with 1.7.0 and 1.8.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556690#comment-15556690
 ] 

ASF GitHub Bot commented on DRILL-4870:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/605#discussion_r82488367
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -392,5 +392,7 @@ export DRILL_HOME
 export DRILL_CONF_DIR
 export DRILL_LOG_DIR
 export CP
-export JAVA_HOME
+# DRILL-4870: Don't export JAVA_HOME. Java can find it just fine from the 
java
+# command. If we attempt to work out out, we do so incorrectly for the Mac.
--- End diff --

typo? 'work out out'


> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556689#comment-15556689
 ] 

ASF GitHub Bot commented on DRILL-4870:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/605#discussion_r82488363
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -392,5 +392,7 @@ export DRILL_HOME
 export DRILL_CONF_DIR
 export DRILL_LOG_DIR
 export CP
-export JAVA_HOME
--- End diff --

In drillbit.sh, we refer to JAVA_HOME with this '$JAVA_HOME/bin/jstack', is 
that piece of code still working after we do not export JAVA_HOME here?
Maybe it is better we limit the impact of not exporting JAVA_HOME to MAC 
and keep the same behavior on Linux(and other OSes)?


> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread Zelaine Fong (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong reassigned DRILL-4870:
---

Assignee: Chunhui Shi  (was: Paul Rogers)

Assigning to [~cshi] fo review.

> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Chunhui Shi
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556556#comment-15556556
 ] 

ASF GitHub Bot commented on DRILL-4870:
---

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/605

DRILL-4870 drill-config.sh sets JAVA_HOME incorrectly for the Mac

Recent script revisions uncommented a line to export the Drill-computed 
JAVA_HOME. Turns out, we should not do this; we should allow Java to figure it 
out as the location differs depending on platform. This change comments out 
that export line to restore the pre-1.8 functionality.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-rogers/drill DRILL-4870

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/605.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #605


commit 164a29d92011722f6234ed442e62919048d45fb4
Author: Paul Rogers 
Date:   2016-10-07T20:53:38Z

DRILL-4870 drill-config.sh sets JAVA_HOME incorrectly for the Mac




> drill-config.sh sets JAVA_HOME incorrectly for the Mac
> --
>
> Key: DRILL-4870
> URL: https://issues.apache.org/jira/browse/DRILL-4870
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: MacOS with unset JAVA_HOME
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> It turns out that drill-config.sh is both improperly and unnecessarily 
> setting the JAVA_HOME envrironment variable. That setting should be removed.
> In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment 
> variable is set. If not, it sets JAVA_HOME based on its guess as to the 
> proper value.
> In the 1.7 version, the veriable was set, but not exported, so the variable 
> was never actually used.
> The recent script fixes for 1.8 "fixed" the export problem. The fix works 
> fine on Linux. But, the Java install on the Mac has a different structure 
> than that on Linux. The value that drill-config.sh guesses is fine for Linux, 
> wrong for the Mac.
> When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME 
> will get the following error when using a Drill script:
> ./drill-embedded 
> Unable to locate an executable at 
> "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java"
> Mac users who do set JAVA_HOME will not encounter the problem (because 
> drill-config.sh does not change an existing value.)
> It seems likely that someone in the past ecountered the same problem and 
> removed the export of DRILL_HOME as an attempt to fix the problem.
> As it turns out, Java does know how to set JAVA_HOME properly if not set. So, 
> setting JAVA_HOME is unnecessary.
> The proper fix is to remove JAVA_HOME setting from drill-config.sh.
> The workaround for any 1.8 user who encounters the problem is to edit their 
> $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the 
> file:
> export JAVA_HOME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4203) Parquet File : Date is stored wrongly

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556065#comment-15556065
 ] 

ASF GitHub Bot commented on DRILL-4203:
---

Github user vdiravka commented on the issue:

https://github.com/apache/drill/pull/595
  
The branch is rebased to the master version.


> Parquet File : Date is stored wrongly
> -
>
> Key: DRILL-4203
> URL: https://issues.apache.org/jira/browse/DRILL-4203
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Stéphane Trou
>Assignee: Vitalii Diravka
>Priority: Critical
> Fix For: 1.9.0
>
>
> Hello,
> I have some problems when i try to read parquet files produce by drill with  
> Spark,  all dates are corrupted.
> I think the problem come from drill :)
> {code}
> cat /tmp/date_parquet.csv 
> Epoch,1970-01-01
> {code}
> {code}
> 0: jdbc:drill:zk=local> select columns[0] as name, cast(columns[1] as date) 
> as epoch_date from dfs.tmp.`date_parquet.csv`;
> ++-+
> |  name  | epoch_date  |
> ++-+
> | Epoch  | 1970-01-01  |
> ++-+
> {code}
> {code}
> 0: jdbc:drill:zk=local> create table dfs.tmp.`buggy_parquet`as select 
> columns[0] as name, cast(columns[1] as date) as epoch_date from 
> dfs.tmp.`date_parquet.csv`;
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> {code}
> When I read the file with parquet tools, i found  
> {code}
> java -jar parquet-tools-1.8.1.jar head /tmp/buggy_parquet/
> name = Epoch
> epoch_date = 4881176
> {code}
> According to 
> [https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md#date], 
> epoch_date should be equals to 0.
> Meta : 
> {code}
> java -jar parquet-tools-1.8.1.jar meta /tmp/buggy_parquet/
> file:file:/tmp/buggy_parquet/0_0_0.parquet 
> creator: parquet-mr version 1.8.1-drill-r0 (build 
> 6b605a4ea05b66e1a6bf843353abcb4834a4ced8) 
> extra:   drill.version = 1.4.0 
> file schema: root 
> 
> name:OPTIONAL BINARY O:UTF8 R:0 D:1
> epoch_date:  OPTIONAL INT32 O:DATE R:0 D:1
> row group 1: RC:1 TS:93 OFFSET:4 
> 
> name: BINARY SNAPPY DO:0 FPO:4 SZ:52/50/0,96 VC:1 
> ENC:RLE,BIT_PACKED,PLAIN
> epoch_date:   INT32 SNAPPY DO:0 FPO:56 SZ:45/43/0,96 VC:1 
> ENC:RLE,BIT_PACKED,PLAIN
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1939#comment-1939
 ] 

ASF GitHub Bot commented on DRILL-4653:
---

Github user ssriniva123 commented on the issue:

https://github.com/apache/drill/pull/518
  
I have a similar data set checked in. Hope that is good enough.


On Fri, Oct 7, 2016 at 10:00 AM, Paul Rogers 
wrote:

> Base data set:
>
> { "p": 1, "a": { "x": 10, "y": 20, "z": 30 }, "b": 50, "c": 60 }
> { "p": 2, "a": { "x": 11, "y": 21, "z": 31 }, "b": 51, "c": 61 }
> { "p": 3, "a": { "x": 12, "y": 22, "z": 32 }, "b": 52, "c": 62 }
> { "p": 4, "a": { "x": 13, "y": 23, "z": 33 }, "b": 53, "c": 63 }
> { "p": 5, "a": { "x": 14, "y": 24, "z": 34 }, "b": 54, "c": 64 }
> { "p": 6, "a": { "x": 15, "y": 25, "z": 35 }, "b": 55, "c": 65 }
>
> Create various errors:
>
> { "p": 2x, "a": { "x": 11, "y": 21, "z": 31 }, "b": 51, "c": 61 }
>
> and
>
> { "p": 2, "a": { "x": 11x, "y": 21, "z": 31 }, "b": 51, "c": 61 }
>
> And so on.
>
> In running the tests, I consistently saw that only the second (bad) row
> was omitted. Other rows properly appeared, and no partial row appeared.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
> Fix For: Future
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4934) ServiceEngine does not use property useIP for DrillbitStartup

2016-10-07 Thread Joe Swingle (JIRA)

Joe Swingle created DRILL-4934:
--

 Summary: ServiceEngine does not use property useIP for 
DrillbitStartup
 Key: DRILL-4934
 URL: https://issues.apache.org/jira/browse/DRILL-4934
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - RPC
Affects Versions: 1.8.0
 Environment: All Envrironments.
Reporter: Joe Swingle
Priority: Minor
 Fix For: 1.9.0


Our environment is configured such that two networks cannot resolve machines by 
hostname, but can connect by IP address.  This creates a problem when an 
ODBC/JDBC Connection requests a Drillbit from the Zookeeper Quorum.  The Quorum 
returns the hostname of the running drillbit.  The Quorum should be capable of 
returning the IP Address. Changing the existing property 
_'drill.exec.rpc.use.ip_' in *drill-override.conf* did not have desired effect. 
 

Reviewing code in org.apache.drill.exec.service.ServiceEngine.java, shows the 
boolean useIP is set to false, and never read from the configuration.

Simply adding the folllowing code at Line 76 resolved issue.:
{code:java}
useIP = context.getConfig().getBoolean(ExecConstants.USE_IP_ADDRESS);
{code}

With the above code,  the Drillbit is registerd in the quorum with the IP 
address, not the hostname.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1635#comment-1635
 ] 

ASF GitHub Bot commented on DRILL-4653:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/518
  
Base data set:

{ "p": 1, "a": { "x": 10, "y": 20, "z": 30 }, "b": 50, "c": 60 }
{ "p": 2, "a": { "x": 11, "y": 21, "z": 31 }, "b": 51, "c": 61 }
{ "p": 3, "a": { "x": 12, "y": 22, "z": 32 }, "b": 52, "c": 62 }
{ "p": 4, "a": { "x": 13, "y": 23, "z": 33 }, "b": 53, "c": 63 }
{ "p": 5, "a": { "x": 14, "y": 24, "z": 34 }, "b": 54, "c": 64 }
{ "p": 6, "a": { "x": 15, "y": 25, "z": 35 }, "b": 55, "c": 65 }

Create various errors:

{ "p": 2x, "a": { "x": 11, "y": 21, "z": 31 }, "b": 51, "c": 61 }

and

{ "p": 2, "a": { "x": 11x, "y": 21, "z": 31 }, "b": 51, "c": 61 }

And so on.

In running the tests, I consistently saw that only the second (bad) row was 
omitted. Other rows properly appeared, and no partial row appeared.


> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
> Fix For: Future
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

2016-10-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1218#comment-1218
 ] 

ASF GitHub Bot commented on DRILL-4653:
---

Github user zfong commented on the issue:

https://github.com/apache/drill/pull/518
  
@paul-rogers - since you had concerns about particular test cases, and it 
looks like you've confirmed that those are non-issues, would it make sense for 
you to share those with @ssriniva123 and he can then include them as unit tests 
with this pull request?


> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
> Fix For: Future
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-4699) Add Description Column in sys.options

[jira] [Commented] (DRILL-3178) csv reader should allow newlines inside quotes

[jira] [Assigned] (DRILL-4841) Use user server event loop group for web clients

[jira] [Updated] (DRILL-4876) Remain disconnected connection

[jira] [Commented] (DRILL-4876) Remain disconnected connection

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Assigned] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Commented] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac

[jira] [Commented] (DRILL-4203) Parquet File : Date is stored wrongly

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

[jira] [Created] (DRILL-4934) ServiceEngine does not use property useIP for DrillbitStartup

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

30 matches

Site Navigation

Mail list logo

Footer information