[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495422
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
+  " Range: 0 - 9223372036854775807.");
 
-  String ENABLE_VERBOSE_ERRORS_KEY = "exec.errors.verbose";
-  OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false);
+  public static final String ENABLE_VERBOSE_ERRORS_KEY = 
"exec.errors.verbose";
+  public static final OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false,
+  "Toggles verbose output of error messages.");
 
-  String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
-  OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
+  public static final String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
+  public static final OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
 
-  String BOOTSTRAP_STORAGE_PLUGINS_FILE = "bootstrap-storage-plugins.json";
-  String MAX_LOADING_CACHE_SIZE_CONFIG = 
"drill.exec.compile.cache_max_size";
+  public static final

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495374
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator 
MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
+  public static final String MONGO_BSON_RECORD_READER = 
"store.mongo.bson.record.reader";
+  public static final

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485277
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS = 
"drill.exec.sort.external.spill.directories";
-  String EXTERNAL_SORT_SPILL_FILESYSTEM = 
"drill.exec.sort.external.spill.fs";
-  String EXTERNAL_SORT_MSORT_MAX_BATCHSIZE = 
"drill.exec.sort.external.msort.batch.maxsize";
-  String

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495511
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 ---
@@ -46,40 +46,83 @@
   // max off heap memory for planning (16G)
   private static final long MAX_OFF_HEAP_ALLOCATION_IN_BYTES = 16l * 1024 
* 1024 * 1024;
 
-  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true);
+  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true,
+  "If one side of a filter condition is a constant expression, 
constant folding evaluates the expression in the" +
+  " planning phase and replaces the expression with the constant 
value. For example, Drill can rewrite" +
+  " this clause ' WHERE age + 5 < 42 as WHERE age < 37'.");
+
   public static final OptionValidator EXCHANGE = new 
BooleanValidator("planner.disable_exchanges", false);
+
   public static final OptionValidator HASHAGG = new 
BooleanValidator("planner.enable_hashagg", true);
+
   public static final OptionValidator STREAMAGG = new 
BooleanValidator("planner.enable_streamagg", true);
-  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true);
-  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true);
+
+  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true,
+  "Enable the memory hungry hash join. Drill assumes that a query with 
have adequate memory to complete and" +
+  " tries to use the fastest operations possible to complete the 
planned inner, left, right, or full outer" +
+  " joins using a hash table. Does not write to disk. Disabling 
hash join allows Drill to manage arbitrarily" +
+  " large data in a small memory footprint.");
+
+  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true,
+  "Sort-based operation. A merge join is used for inner join, left and 
right outer joins. Inputs to the merge" +
+  " join must be sorted. It reads the sorted input streams from 
both sides and finds matching rows." +
+  " Writes to disk.");
+
   public static final OptionValidator NESTEDLOOPJOIN = new 
BooleanValidator("planner.enable_nestedloopjoin", true);
+
   public static final OptionValidator MULTIPHASE = new 
BooleanValidator("planner.enable_multiphase_agg", true);
-  public static final OptionValidator BROADCAST = new 
BooleanValidator("planner.enable_broadcast_join", true);
-  public static final OptionValidator BROADCAST_THRESHOLD = new 
PositiveLongValidator("planner.broadcast_threshold", MAX_BROADCAST_THRESHOLD, 
1000);
-  public static final OptionValidator BROADCAST_FACTOR = new 
RangeDoubleValidator("planner.broadcast_factor", 0, Double.MAX_VALUE, 1.0d);
-  public static final OptionValidator NESTEDLOOPJOIN_FACTOR = new 
RangeDoubleValidator("planner.nestedloopjoin_factor", 0, Double.MAX_VALUE, 
100.0d);
-  public static final OptionValidator NLJOIN_FOR_SCALAR = new 
BooleanValidator("planner.enable_nljoin_for_scalar_only", true);
-  public static final OptionValidator JOIN_ROW_COUNT_ESTIMATE_FACTOR = new 
RangeDoubleValidator("planner.join.row_count_estimate_factor", 0, 
Double.MAX_VALUE, 1.0d);
+
+  public static final OptionValidator BROADCAST = new 
BooleanValidator("planner.enable_broadcast_join", true,
+  "The broadcast join can be used for hash join, merge join and nested 
loop join. Use to join a large (fact)" +
+  " table to relatively smaller (dimension) tables. This should be 
enabled.");
+  public static final OptionValidator BROADCAST_THRESHOLD = new 
PositiveLongValidator("planner.broadcast_threshold",
+  MAX_BROADCAST_THRESHOLD, 1000, "The maximum number of records 
allowed to be broadcast as part of a query." +
+  " If the threshold is exceeded, Drill reshuffles data rather than 
doing a broadcast to one side of the" +
+  " join. Range: 0 - " + MAX_BROADCAST_THRESHOLD + ".");
+  public static final OptionValidator BROADCAST_FACTOR = new 
RangeDoubleValidator("planner.broadcast_factor", 0,
+  Double.MAX_VALUE, 1.0d, "A heuristic parameter for influencing the 
broadcast of records as part of a query.");
--- End diff --

Increasing the value does... what?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485619
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS = 
"drill.exec.sort.external.spill.directories";
-  String EXTERNAL_SORT_SPILL_FILESYSTEM = 
"drill.exec.sort.external.spill.fs";
-  String EXTERNAL_SORT_MSORT_MAX_BATCHSIZE = 
"drill.exec.sort.external.msort.batch.maxsize";
-  String

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485491
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS = 
"drill.exec.sort.external.spill.directories";
-  String EXTERNAL_SORT_SPILL_FILESYSTEM = 
"drill.exec.sort.external.spill.fs";
-  String EXTERNAL_SORT_MSORT_MAX_BATCHSIZE = 
"drill.exec.sort.external.msort.batch.maxsize";
-  String

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495327
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
--- End diff --

Instead of saying that this takes a string, can we generate that part? We 
look up the default value, infer the type and say:

The column label ... Type: string, Default: "dir"

Or, add type and default as additional columns. (Or maybe the default does 
not matter, only the current value, so just list the type.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82485322
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -33,110 +33,122 @@
 import org.apache.drill.exec.testing.ExecutionControls;
 import org.apache.drill.exec.util.ImpersonationUtil;
 
-public interface ExecConstants {
-  String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
-  String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
-  String ZK_CONNECTION = "drill.exec.zk.connect";
-  String ZK_TIMEOUT = "drill.exec.zk.timeout";
-  String ZK_ROOT = "drill.exec.zk.root";
-  String ZK_REFRESH = "drill.exec.zk.refresh";
-  String BIT_RETRY_TIMES = "drill.exec.rpc.bit.server.retry.count";
-  String BIT_RETRY_DELAY = "drill.exec.rpc.bit.server.retry.delay";
-  String BIT_TIMEOUT = "drill.exec.bit.timeout" ;
-  String SERVICE_NAME = "drill.exec.cluster-id";
-  String INITIAL_BIT_PORT = "drill.exec.rpc.bit.server.port";
-  String BIT_RPC_TIMEOUT = "drill.exec.rpc.bit.timeout";
-  String INITIAL_USER_PORT = "drill.exec.rpc.user.server.port";
-  String USER_RPC_TIMEOUT = "drill.exec.rpc.user.timeout";
-  String METRICS_CONTEXT_NAME = "drill.exec.metrics.context";
-  String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
-  String CLIENT_RPC_THREADS = "drill.exec.rpc.user.client.threads";
-  String BIT_SERVER_RPC_THREADS = "drill.exec.rpc.bit.server.threads";
-  String USER_SERVER_RPC_THREADS = "drill.exec.rpc.user.server.threads";
-  String TRACE_DUMP_DIRECTORY = "drill.exec.trace.directory";
-  String TRACE_DUMP_FILESYSTEM = "drill.exec.trace.filesystem";
-  String TEMP_DIRECTORIES = "drill.exec.tmp.directories";
-  String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
-  String INCOMING_BUFFER_IMPL = "drill.exec.buffer.impl";
+public final class ExecConstants {
+
+  public static final String ZK_RETRY_TIMES = "drill.exec.zk.retry.count";
+  public static final String ZK_RETRY_DELAY = "drill.exec.zk.retry.delay";
+  public static final String ZK_CONNECTION = "drill.exec.zk.connect";
+  public static final String ZK_TIMEOUT = "drill.exec.zk.timeout";
+  public static final String ZK_ROOT = "drill.exec.zk.root";
+  public static final String ZK_REFRESH = "drill.exec.zk.refresh";
+  public static final String BIT_RETRY_TIMES = 
"drill.exec.rpc.bit.server.retry.count";
+  public static final String BIT_RETRY_DELAY = 
"drill.exec.rpc.bit.server.retry.delay";
+  public static final String BIT_TIMEOUT = "drill.exec.bit.timeout";
+  public static final String SERVICE_NAME = "drill.exec.cluster-id";
+  public static final String INITIAL_BIT_PORT = 
"drill.exec.rpc.bit.server.port";
+  public static final String BIT_RPC_TIMEOUT = 
"drill.exec.rpc.bit.timeout";
+  public static final String INITIAL_USER_PORT = 
"drill.exec.rpc.user.server.port";
+  public static final String USER_RPC_TIMEOUT = 
"drill.exec.rpc.user.timeout";
+  public static final String METRICS_CONTEXT_NAME = 
"drill.exec.metrics.context";
+  public static final String USE_IP_ADDRESS = "drill.exec.rpc.use.ip";
+  public static final String CLIENT_RPC_THREADS = 
"drill.exec.rpc.user.client.threads";
+  public static final String BIT_SERVER_RPC_THREADS = 
"drill.exec.rpc.bit.server.threads";
+  public static final String USER_SERVER_RPC_THREADS = 
"drill.exec.rpc.user.server.threads";
+  public static final String TRACE_DUMP_DIRECTORY = 
"drill.exec.trace.directory";
+  public static final String TRACE_DUMP_FILESYSTEM = 
"drill.exec.trace.filesystem";
+  public static final String TEMP_DIRECTORIES = 
"drill.exec.tmp.directories";
+  public static final String TEMP_FILESYSTEM = "drill.exec.tmp.filesystem";
+  public static final String INCOMING_BUFFER_IMPL = 
"drill.exec.buffer.impl";
   /** incoming buffer size (number of batches) */
-  String INCOMING_BUFFER_SIZE = "drill.exec.buffer.size";
-  String SPOOLING_BUFFER_DELETE = "drill.exec.buffer.spooling.delete";
-  String SPOOLING_BUFFER_MEMORY = "drill.exec.buffer.spooling.size";
-  String BATCH_PURGE_THRESHOLD = "drill.exec.sort.purge.threshold";
-  String EXTERNAL_SORT_TARGET_BATCH_SIZE = 
"drill.exec.sort.external.batch.size";
-  String EXTERNAL_SORT_TARGET_SPILL_BATCH_SIZE = 
"drill.exec.sort.external.spill.batch.size";
-  String EXTERNAL_SORT_SPILL_GROUP_SIZE = 
"drill.exec.sort.external.spill.group.size";
-  String EXTERNAL_SORT_SPILL_THRESHOLD = 
"drill.exec.sort.external.spill.threshold";
-  String EXTERNAL_SORT_SPILL_DIRS = 
"drill.exec.sort.external.spill.directories";
-  String EXTERNAL_SORT_SPILL_FILESYSTEM = 
"drill.exec.sort.external.spill.fs";
-  String EXTERNAL_SORT_MSORT_MAX_BATCHSIZE = 
"drill.exec.sort.external.msort.batch.maxsize";
-  String

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495479
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/compile/QueryClassLoader.java
 ---
@@ -42,7 +42,17 @@
   private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(QueryClassLoader.class);
 
   public static final String JAVA_COMPILER_OPTION = "exec.java_compiler";
-  public static final StringValidator JAVA_COMPILER_VALIDATOR = new 
StringValidator(JAVA_COMPILER_OPTION, CompilerPolicy.DEFAULT.toString()) {
+  public static final String JAVA_COMPILER_JANINO_MAXSIZE_OPTION = 
"exec.java_compiler_janino_maxsize";
+
+  public static final OptionValidator JAVA_COMPILER_JANINO_MAXSIZE =
+  new LongValidator(JAVA_COMPILER_JANINO_MAXSIZE_OPTION, 256 * 1024,
+  "See the " + JAVA_COMPILER_OPTION + ". Accepts inputs of type 
LONG.");
--- End diff --

LONG --> long. (Accepts the scalar, not the boxed object.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495346
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator 
MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
+  public static final String MONGO_BSON_RECORD_READER = 
"store.mongo.bson.record.reader";
+  public static final

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495464
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
+  " Range: 0 - 9223372036854775807.");
 
-  String ENABLE_VERBOSE_ERRORS_KEY = "exec.errors.verbose";
-  OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false);
+  public static final String ENABLE_VERBOSE_ERRORS_KEY = 
"exec.errors.verbose";
+  public static final OptionValidator ENABLE_VERBOSE_ERRORS = new 
BooleanValidator(ENABLE_VERBOSE_ERRORS_KEY, false,
+  "Toggles verbose output of error messages.");
 
-  String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
-  OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
+  public static final String ENABLE_NEW_TEXT_READER_KEY = 
"exec.storage.enable_new_text_reader";
+  public static final OptionValidator ENABLE_NEW_TEXT_READER = new 
BooleanValidator(ENABLE_NEW_TEXT_READER_KEY, true);
 
-  String BOOTSTRAP_STORAGE_PLUGINS_FILE = "bootstrap-storage-plugins.json";
-  String MAX_LOADING_CACHE_SIZE_CONFIG = 
"drill.exec.compile.cache_max_size";
+  public static final

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495404
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -237,71 +281,81 @@
* DEFAULT: 64 MB
* MAXIMUM: 2048 MB
*/
-  String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
-  OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
-NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
+  public static final String NON_BLOCKING_OPERATORS_MEMORY_KEY = 
"planner.memory.non_blocking_operators_memory";
+  public static final OptionValidator NON_BLOCKING_OPERATORS_MEMORY = new 
PowerOfTwoLongValidator(
+  NON_BLOCKING_OPERATORS_MEMORY_KEY, 1 << 11, 1 << 6);
 
-  String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
-  OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_JOIN_TABLE_FACTOR_KEY = 
"planner.memory.hash_join_table_factor";
+  public static final OptionValidator HASH_JOIN_TABLE_FACTOR = new 
DoubleValidator(HASH_JOIN_TABLE_FACTOR_KEY, 1.1d);
 
-  String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
-  OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
+  public static final String HASH_AGG_TABLE_FACTOR_KEY = 
"planner.memory.hash_agg_table_factor";
+  public static final OptionValidator HASH_AGG_TABLE_FACTOR = new 
DoubleValidator(HASH_AGG_TABLE_FACTOR_KEY, 1.1d);
 
-  String AVERAGE_FIELD_WIDTH_KEY = "planner.memory.average_field_width";
-  OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY, Long.MAX_VALUE, 8);
+  public static final String AVERAGE_FIELD_WIDTH_KEY = 
"planner.memory.average_field_width";
+  public static final OptionValidator AVERAGE_FIELD_WIDTH = new 
PositiveLongValidator(AVERAGE_FIELD_WIDTH_KEY,
+  Long.MAX_VALUE, 8);
 
-  BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
-  LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10);
-  LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100);
-  LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
-  Long.MAX_VALUE, 3000);
-  LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
-  Long.MAX_VALUE, 60 * 1000 * 5);
+  public static final BooleanValidator ENABLE_QUEUE = new 
BooleanValidator("exec.queue.enable", false);
+  public static final LongValidator LARGE_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.large", 1000, 10,
+  "Sets the number of large queries that can run concurrently in the 
cluster. Range: 0 - 1000.");
+  public static final LongValidator SMALL_QUEUE_SIZE = new 
PositiveLongValidator("exec.queue.small", 10, 100,
+  "Sets the number of small queries that can run concurrently in the 
cluster. Range: 0 - 10.");
+  public static final LongValidator QUEUE_THRESHOLD_SIZE = new 
PositiveLongValidator("exec.queue.threshold",
+  Long.MAX_VALUE, 3000, "Sets the cost threshold, which depends on 
the complexity of the queries in" +
+  " queue, for determining whether query is large or small. Complex 
queries have higher thresholds." +
+  " Range: 0 - 9223372036854775807.");
+  public static final LongValidator QUEUE_TIMEOUT = new 
PositiveLongValidator("exec.queue.timeout_millis",
+  Long.MAX_VALUE, 60 * 1000 * 5, "Indicates how long a query can wait 
in queue before the query fails." +
--- End diff --

Units? Secs? ms?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495353
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -146,89 +158,121 @@
*|-bar  -  a.parquet
*|-baz  -  b.parquet
*/
-  String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
-  OptionValidator FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new 
StringValidator(FILESYSTEM_PARTITION_COLUMN_LABEL, "dir");
+  public static final String FILESYSTEM_PARTITION_COLUMN_LABEL = 
"drill.exec.storage.file.partition.column.label";
+  public static final OptionValidator 
FILESYSTEM_PARTITION_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  FILESYSTEM_PARTITION_COLUMN_LABEL, "dir",
+  "The column label for directory levels in results of queries of 
files in a directory. Accepts a string input.");
 
   /**
* Implicit file columns
*/
-  String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
-  OptionValidator IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
-  String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
-  OptionValidator IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
-  String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
-  OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FQN_COLUMN_LABEL, "fqn");
-  String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
-  OptionValidator IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new 
StringValidator(IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
-
-  String JSON_READ_NUMBERS_AS_DOUBLE = "store.json.read_numbers_as_double";
-  BooleanValidator JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(JSON_READ_NUMBERS_AS_DOUBLE, false);
-
-  String MONGO_ALL_TEXT_MODE = "store.mongo.all_text_mode";
-  OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR = new 
BooleanValidator(MONGO_ALL_TEXT_MODE, false);
-  String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
-  OptionValidator MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new 
BooleanValidator(MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
-  String MONGO_BSON_RECORD_READER = "store.mongo.bson.record.reader";
-  OptionValidator MONGO_BSON_RECORD_READER_VALIDATOR = new 
BooleanValidator(MONGO_BSON_RECORD_READER, true);
-
-  BooleanValidator ENABLE_UNION_TYPE = new 
BooleanValidator("exec.enable_union_type", false);
+  public static final String IMPLICIT_FILENAME_COLUMN_LABEL = 
"drill.exec.storage.implicit.filename.column.label";
+  public static final OptionValidator 
IMPLICIT_FILENAME_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILENAME_COLUMN_LABEL, "filename");
+  public static final String IMPLICIT_SUFFIX_COLUMN_LABEL = 
"drill.exec.storage.implicit.suffix.column.label";
+  public static final OptionValidator 
IMPLICIT_SUFFIX_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_SUFFIX_COLUMN_LABEL, "suffix");
+  public static final String IMPLICIT_FQN_COLUMN_LABEL = 
"drill.exec.storage.implicit.fqn.column.label";
+  public static final OptionValidator IMPLICIT_FQN_COLUMN_LABEL_VALIDATOR 
= new StringValidator(
+  IMPLICIT_FQN_COLUMN_LABEL, "fqn");
+  public static final String IMPLICIT_FILEPATH_COLUMN_LABEL = 
"drill.exec.storage.implicit.filepath.column.label";
+  public static final OptionValidator 
IMPLICIT_FILEPATH_COLUMN_LABEL_VALIDATOR = new StringValidator(
+  IMPLICIT_FILEPATH_COLUMN_LABEL, "filepath");
+
+  public static final String JSON_READ_NUMBERS_AS_DOUBLE = 
"store.json.read_numbers_as_double";
+  public static final BooleanValidator 
JSON_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  JSON_READ_NUMBERS_AS_DOUBLE, false,
+  "Reads numbers with or without a decimal point as DOUBLE. Prevents 
schema change errors.");
+
+  public static final String MONGO_ALL_TEXT_MODE = 
"store.mongo.all_text_mode";
+  public static final OptionValidator MONGO_READER_ALL_TEXT_MODE_VALIDATOR 
= new BooleanValidator(MONGO_ALL_TEXT_MODE,
+  false);
+  public static final String MONGO_READER_READ_NUMBERS_AS_DOUBLE = 
"store.mongo.read_numbers_as_double";
+  public static final OptionValidator 
MONGO_READER_READ_NUMBERS_AS_DOUBLE_VALIDATOR = new BooleanValidator(
+  MONGO_READER_READ_NUMBERS_AS_DOUBLE, false);
+  public static final String MONGO_BSON_RECORD_READER = 
"store.mongo.bson.record.reader";
+  public static final

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-10-07 Thread paul-rogers 
Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/536#discussion_r82495497
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 ---
@@ -46,40 +46,83 @@
   // max off heap memory for planning (16G)
   private static final long MAX_OFF_HEAP_ALLOCATION_IN_BYTES = 16l * 1024 
* 1024 * 1024;
 
-  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true);
+  public static final OptionValidator CONSTANT_FOLDING = new 
BooleanValidator("planner.enable_constant_folding", true,
+  "If one side of a filter condition is a constant expression, 
constant folding evaluates the expression in the" +
+  " planning phase and replaces the expression with the constant 
value. For example, Drill can rewrite" +
+  " this clause ' WHERE age + 5 < 42 as WHERE age < 37'.");
+
   public static final OptionValidator EXCHANGE = new 
BooleanValidator("planner.disable_exchanges", false);
+
   public static final OptionValidator HASHAGG = new 
BooleanValidator("planner.enable_hashagg", true);
+
   public static final OptionValidator STREAMAGG = new 
BooleanValidator("planner.enable_streamagg", true);
-  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true);
-  public static final OptionValidator MERGEJOIN = new 
BooleanValidator("planner.enable_mergejoin", true);
+
+  public static final OptionValidator HASHJOIN = new 
BooleanValidator("planner.enable_hashjoin", true,
+  "Enable the memory hungry hash join. Drill assumes that a query with 
have adequate memory to complete and" +
--- End diff --

with have --> will have


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #536: DRILL-4699: Add description column to sys.options t...

2016-06-28 Thread sudheeshkatkam 
GitHub user sudheeshkatkam opened a pull request:

https://github.com/apache/drill/pull/536

DRILL-4699: Add description column to sys.options table

+ Add new constructor for each validator in TypeValidators with
  description field; deprecate old constructors
+ Add descriptions for some validators

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sudheeshkatkam/drill DRILL-4699

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #536


commit f8a78394423cca46aa3bfd63b2436c606df4b5b7
Author: Sudheesh Katkam 
Date:   2016-06-28T23:12:08Z

DRILL-4699: Add description column to sys.options table

+ Add new constructor for each validator in TypeValidators with
  description field; deprecate old constructors
+ Add descriptions for some validators




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---