[jira] [Commented] (DRILL-4047) Select with options

[Julien Le Dem (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096786#comment-15096786
 ] 

Julien Le Dem commented on DRILL-4047:
--

Here is how it works:
Context:
You can query files in a hadoop file system in the following way:
{noformat}
  select columns from dfs.`mytable/file.csv`;
{noformat}
The format plugin is selected based on file extension or header according to 
the configuration.

This feature enables providing the format plugin and its config in the SQL 
query instead as follows:
{noformat}
  select columns from table(dfs.`mytable/file.csv`(type => 'text', 
fieldDelimiter => ','))
{noformat}

The table(...) syntax tells the SQL parser that we're passing parameters to the 
table.
The type parameter is required and selects the Format plugin as defined by its 
Config class (as would be done when deserializing from json)
The other parameters are defined by the fields of the same class. (excluding 
"extensions" that does not apply in that case)
For example in the case of Text:
https://github.com/apache/drill/blob/1.4.0/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/TextFormatPlugin.java#L135
{noformat}
  @JsonTypeName("text") @JsonInclude(Include.NON_DEFAULT)
  public static class TextFormatConfig implements FormatPluginConfig {

public List extensions = ImmutableList.of();
public String lineDelimiter = "\n";
public char fieldDelimiter = '\n';
public char quote = '"';
public char escape = '"';
public char comment = '#';
public boolean skipFirstLine = false;
public boolean extractHeader = false;
{noformat}



> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[Julien Le Dem (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096789#comment-15096789
 ] 

Julien Le Dem commented on DRILL-4047:
--

You can refer to unit tests for more examples:
https://github.com/apache/drill/blob/1.4.0/exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java

> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093487#comment-15093487
 ] 

Khurram Faraaz commented on DRILL-4047:
---

[~ julienledem] Can you please provide a specification document that can be 
used to come up with a test plan to verify and test this feature ?

> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028119#comment-15028119
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/246


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024967#comment-15024967
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on the pull request:

https://github.com/apache/drill/pull/246#issuecomment-159355075
  
I ran the full test suite. It's green


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007391#comment-15007391
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44987917
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import static java.lang.String.format;
+import static org.apache.drill.TestBuilder.listOf;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+import org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class TestSelectWithOption extends BaseTestQuery {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(WorkspaceSchemaFactory.class);
+
+  private File genCSVFile(String name, String... rows) throws IOException {
+File file = new File(format("target/%s_%s.csv", 
this.getClass().getName(), name));
+try (FileWriter fw = new FileWriter(file)) {
+  for (int i = 0; i < rows.length; i++) {
+fw.append(rows[i] + "\n");
+  }
+}
+return file;
+  }
+
+  private String genCSVTable(String name, String... rows) throws 
IOException {
+File f = genCSVFile(name, rows);
+return format("dfs.`${WORKING_PATH}/%s`", f.getPath());
+  }
+
+  private void testWithResult(String query, Object... expectedResult) 
throws Exception {
+TestBuilder builder = testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("columns");
+for (Object o : expectedResult) {
+  builder = builder.baselineValues(o);
+}
+builder.build().run();
+  }
+
+  @Test
+  public void testTextFieldDelimiter() throws Exception {
+String tableName = genCSVTable("testTextFieldDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+String queryTemplate =
+"select columns from table(%s (type => 'TeXT', fieldDelimiter => 
'%s'))";
+testWithResult(format(queryTemplate, tableName, ","),
+listOf("b\"|\"0"),
+listOf("b\"|\"1"),
+listOf("b\"|\"2")
+  );
+testWithResult(format(queryTemplate, tableName, "|"),
+listOf("b", "0"),
+listOf("b", "1"),
+listOf("b", "2")
+  );
+  }
+
+  @Test @Ignore // It does not look like lineDelimiter is working
+  public void testTextLineDelimiter() throws Exception {
+String tableName = genCSVTable("testTextLineDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
lineDelimiter => '|'))", tableName),
+listOf("\"b\""),
+listOf("\"0\"", "\"b\""),
+listOf("\"1\"", "\"b\""),
+listOf("\"2\"")
+  );
+  }
+
+  @Test
+  public void testTextQuote() throws Exception {
+String tableName = genCSVTable("testTextQuote",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
fieldDelimiter => '|', quote => '@'))", tableName),
+listOf("\"b\"", "\"0\""),
+listOf("\"b\"", "\"1\""),
+listOf("\"b\"", "\"2\"")
+);
+
+String quoteTableName = genCSVTable("testTextQuote2",
+"@b@|@0@",
+"@b$@c@|@1@");
+// It seems that a parameter can not be called "escape"
+testWithResult(format("select columns from table(%s(`escape` => '$', 
type => 'TeXT', fieldDelimiter => '|', quote => '@'))", 

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006011#comment-15006011
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44876693
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import static java.lang.String.format;
+import static org.apache.drill.TestBuilder.listOf;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+import org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class TestSelectWithOption extends BaseTestQuery {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(WorkspaceSchemaFactory.class);
+
+  private File genCSVFile(String name, String... rows) throws IOException {
+File file = new File(format("target/%s_%s.csv", 
this.getClass().getName(), name));
+try (FileWriter fw = new FileWriter(file)) {
+  for (int i = 0; i < rows.length; i++) {
+fw.append(rows[i] + "\n");
+  }
+}
+return file;
+  }
+
+  private String genCSVTable(String name, String... rows) throws 
IOException {
+File f = genCSVFile(name, rows);
+return format("dfs.`${WORKING_PATH}/%s`", f.getPath());
+  }
+
+  private void testWithResult(String query, Object... expectedResult) 
throws Exception {
+TestBuilder builder = testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("columns");
+for (Object o : expectedResult) {
+  builder = builder.baselineValues(o);
+}
+builder.build().run();
+  }
+
+  @Test
+  public void testTextFieldDelimiter() throws Exception {
+String tableName = genCSVTable("testTextFieldDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+String queryTemplate =
+"select columns from table(%s (type => 'TeXT', fieldDelimiter => 
'%s'))";
+testWithResult(format(queryTemplate, tableName, ","),
+listOf("b\"|\"0"),
+listOf("b\"|\"1"),
+listOf("b\"|\"2")
+  );
+testWithResult(format(queryTemplate, tableName, "|"),
+listOf("b", "0"),
+listOf("b", "1"),
+listOf("b", "2")
+  );
+  }
+
+  @Test @Ignore // It does not look like lineDelimiter is working
+  public void testTextLineDelimiter() throws Exception {
+String tableName = genCSVTable("testTextLineDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
lineDelimiter => '|'))", tableName),
+listOf("\"b\""),
+listOf("\"0\"", "\"b\""),
+listOf("\"1\"", "\"b\""),
+listOf("\"2\"")
+  );
+  }
+
+  @Test
+  public void testTextQuote() throws Exception {
+String tableName = genCSVTable("testTextQuote",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
fieldDelimiter => '|', quote => '@'))", tableName),
+listOf("\"b\"", "\"0\""),
+listOf("\"b\"", "\"1\""),
+listOf("\"b\"", "\"2\"")
+);
+
+String quoteTableName = genCSVTable("testTextQuote2",
+"@b@|@0@",
+"@b$@c@|@1@");
+// It seems that a parameter can not be called "escape"
+testWithResult(format("select columns from table(%s(`escape` => '$', 
type => 'TeXT', fieldDelimiter => '|', quote => '@'))", 

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004537#comment-15004537
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44821481
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
+final Class pluginConfigClass;
+final String typeName;
+private final Map functionParamsByName;
+
+OptionsDescriptor(Class 
pluginConfigClass) {
+  this.pluginConfigClass = pluginConfigClass;
+  Map paramsByName = new LinkedHashMap<>();
+  Field[] fields = pluginConfigClass.getDeclaredFields();
+  // @JsonTypeName("text")
+  JsonTypeName annotation = 
pluginConfigClass.getAnnotation(JsonTypeName.class);
+  this.typeName = annotation != null ? annotation.value() : null;
+  if (this.typeName != null) {
+paramsByName.put("type", new TableParamDef("type", String.class));
+  }
+  for (Field field : fields) {
+if (Modifier.isStatic(field.getModifiers())
+// we want to deprecate this field
+|| (field.getName().equals("extensions") && field.getType() == 
List.class)) {
+  continue;
+}
+Class fieldType = field.getType();
+if (fieldType == char.class) {
+  // calcite does not like char type. Just use String and enforce 
later that length == 1
+  fieldType = String.class;
+}
+paramsByName.put(field.getName(), new 
TableParamDef(field.getName(), fieldType).optional());
+  }
+  this.functionParamsByName = unmodifiableMap(paramsByName);
+}
+
+TableSignature getTableSignature(String tableName) {
+  return new TableSignature(tableName, params());
+}
+
+private List params() {
+  return new ArrayList<>(functionParamsByName.values());
+}
+
+String presentParams() {
+  StringBuilder sb = new StringBuilder("(");
+  List params = params();
+  for (int i = 0; i < params.size(); i++) {
+TableParamDef paramDef = params.get(i);
+if (i != 0) {
+  sb.append(", ");
+}
+sb.append(paramDef.name).append(": 
").append(paramDef.type.getSimpleName());
+  }
+  sb.append(")");
+  return sb.toString();
+}
+
+FormatPluginConfig eval(TableInstance t) {
+  // Per the constructor, the first param is always "type"
+  TableParamDef typeParamDef = t.sig.params.get(0);
+  Object typeParam = t.params.get(0);
+  if (!typeParamDef.name.equals("type") || typeParamDef.type != 
String.class || !(typeParam instanceof String)
+  || !typeName.equalsIgnoreCase((String) typeParam)) {
+badInput(t);
+  }
+  FormatPluginConfig config;
+  try {
+config = pluginConfigClass.newInstance();
+  } catch (InstantiationException | IllegalAccessException e) {
+throw UserException.parseError(e)
+.message(
+"configuration for format of type %s can not be created 
(class: %s)",
+this.typeName, pluginConfigClass.getName())
+.addContext("table", t.sig.name)
+.build(logger);
+  }
+  for (int i = 1; i < t.params.size(); i++) {
+Object param = t.params.get(i);
+if (param == null)
+  continue;
+TableParamDef paramDef = t.sig.params.get(i);
+TableParamDef expectedParamDef = 
this.functionParamsByName.get(paramDef.name);
+if (expectedParamDef == null || expectedParamDef.type != 
paramDef.type) {
+  badInput(t);
+  

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004531#comment-15004531
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44821281
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
--- End diff --

Can you pull this out into a separate file?


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004540#comment-15004540
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44822308
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
+final Class pluginConfigClass;
+final String typeName;
+private final Map functionParamsByName;
+
+OptionsDescriptor(Class 
pluginConfigClass) {
+  this.pluginConfigClass = pluginConfigClass;
+  Map paramsByName = new LinkedHashMap<>();
+  Field[] fields = pluginConfigClass.getDeclaredFields();
+  // @JsonTypeName("text")
+  JsonTypeName annotation = 
pluginConfigClass.getAnnotation(JsonTypeName.class);
+  this.typeName = annotation != null ? annotation.value() : null;
+  if (this.typeName != null) {
+paramsByName.put("type", new TableParamDef("type", String.class));
+  }
+  for (Field field : fields) {
+if (Modifier.isStatic(field.getModifiers())
+// we want to deprecate this field
+|| (field.getName().equals("extensions") && field.getType() == 
List.class)) {
+  continue;
+}
+Class fieldType = field.getType();
+if (fieldType == char.class) {
+  // calcite does not like char type. Just use String and enforce 
later that length == 1
+  fieldType = String.class;
+}
+paramsByName.put(field.getName(), new 
TableParamDef(field.getName(), fieldType).optional());
+  }
+  this.functionParamsByName = unmodifiableMap(paramsByName);
+}
+
+TableSignature getTableSignature(String tableName) {
+  return new TableSignature(tableName, params());
+}
+
+private List params() {
+  return new ArrayList<>(functionParamsByName.values());
+}
+
+String presentParams() {
+  StringBuilder sb = new StringBuilder("(");
+  List params = params();
+  for (int i = 0; i < params.size(); i++) {
+TableParamDef paramDef = params.get(i);
+if (i != 0) {
+  sb.append(", ");
+}
+sb.append(paramDef.name).append(": 
").append(paramDef.type.getSimpleName());
+  }
+  sb.append(")");
+  return sb.toString();
+}
+
+FormatPluginConfig eval(TableInstance t) {
+  // Per the constructor, the first param is always "type"
+  TableParamDef typeParamDef = t.sig.params.get(0);
+  Object typeParam = t.params.get(0);
+  if (!typeParamDef.name.equals("type") || typeParamDef.type != 
String.class || !(typeParam instanceof String)
+  || !typeName.equalsIgnoreCase((String) typeParam)) {
+badInput(t);
+  }
+  FormatPluginConfig config;
+  try {
+config = pluginConfigClass.newInstance();
+  } catch (InstantiationException | IllegalAccessException e) {
+throw UserException.parseError(e)
+.message(
+"configuration for format of type %s can not be created 
(class: %s)",
+this.typeName, pluginConfigClass.getName())
+.addContext("table", t.sig.name)
+.build(logger);
+  }
+  for (int i = 1; i < t.params.size(); i++) {
+Object param = t.params.get(i);
+if (param == null)
+  continue;
+TableParamDef paramDef = t.sig.params.get(i);
+TableParamDef expectedParamDef = 
this.functionParamsByName.get(paramDef.name);
+if (expectedParamDef == null || expectedParamDef.type != 
paramDef.type) {
+  badInput(t);

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004548#comment-15004548
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44822446
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -148,12 +175,427 @@ private Path getViewPath(String name) {
 return DotDrillType.VIEW.getPath(config.getLocation(), name);
   }
 
-  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws  IOException {
+  public WorkspaceSchema createSchema(List parentSchemaPath, 
SchemaConfig schemaConfig) throws IOException {
 return new WorkspaceSchema(parentSchemaPath, schemaName, schemaConfig);
   }
 
-  public class WorkspaceSchema extends AbstractSchema implements 
ExpandingConcurrentMap.MapValueFactory {
-private final ExpandingConcurrentMap tables = new 
ExpandingConcurrentMap<>(this);
+  /**
+   * Describes the options for a format plugin
+   * extracted from the FormatPluginConfig subclass
+   */
+  static final class OptionsDescriptor {
+final Class pluginConfigClass;
+final String typeName;
+private final Map functionParamsByName;
+
+OptionsDescriptor(Class 
pluginConfigClass) {
+  this.pluginConfigClass = pluginConfigClass;
+  Map paramsByName = new LinkedHashMap<>();
+  Field[] fields = pluginConfigClass.getDeclaredFields();
+  // @JsonTypeName("text")
+  JsonTypeName annotation = 
pluginConfigClass.getAnnotation(JsonTypeName.class);
+  this.typeName = annotation != null ? annotation.value() : null;
+  if (this.typeName != null) {
+paramsByName.put("type", new TableParamDef("type", String.class));
+  }
+  for (Field field : fields) {
+if (Modifier.isStatic(field.getModifiers())
+// we want to deprecate this field
+|| (field.getName().equals("extensions") && field.getType() == 
List.class)) {
+  continue;
+}
+Class fieldType = field.getType();
+if (fieldType == char.class) {
+  // calcite does not like char type. Just use String and enforce 
later that length == 1
+  fieldType = String.class;
+}
+paramsByName.put(field.getName(), new 
TableParamDef(field.getName(), fieldType).optional());
+  }
+  this.functionParamsByName = unmodifiableMap(paramsByName);
+}
+
+TableSignature getTableSignature(String tableName) {
+  return new TableSignature(tableName, params());
+}
+
+private List params() {
+  return new ArrayList<>(functionParamsByName.values());
+}
+
+String presentParams() {
+  StringBuilder sb = new StringBuilder("(");
+  List params = params();
+  for (int i = 0; i < params.size(); i++) {
+TableParamDef paramDef = params.get(i);
+if (i != 0) {
+  sb.append(", ");
+}
+sb.append(paramDef.name).append(": 
").append(paramDef.type.getSimpleName());
+  }
+  sb.append(")");
+  return sb.toString();
+}
+
+FormatPluginConfig eval(TableInstance t) {
--- End diff --

Can you provide a more descriptive method name and short doc?


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994841#comment-14994841
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204833
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -221,9 +603,46 @@ public void dropView(String viewName) throws 
IOException {
   return viewSet;
 }
 
+private Set rawTableNames() {
+  return newHashSet(
+  transform(tables.keySet(), new 
com.google.common.base.Function() {
+@Override
+public String apply(TableInstance input) {
+  return input.sig.name;
+}
+  }));
+}
+
 @Override
 public Set getTableNames() {
-  return Sets.union(tables.keySet(), getViews());
+  System.out.println("getTableNames");
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994834#comment-14994834
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204776
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java
 ---
@@ -86,7 +86,8 @@ public FileSystemSchema(String name, SchemaConfig 
schemaConfig) throws IOExcepti
 
 void setPlus(SchemaPlus plusOfThis){
   for(WorkspaceSchema s : schemaMap.values()){
-plusOfThis.add(s.getName(), s);
+SchemaPlus schemaPlus = plusOfThis.add(s.getName(), s);
+//schemaPlus.add(arg0, arg1);
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994836#comment-14994836
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204801
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatCreator.java 
---
@@ -92,10 +118,39 @@
   logger.warn("Failure initializing storage config named '{}' of 
type '{}'.", e.getKey(), e.getValue().getClass().getName(), e1);
 }
   }
-
 }
+this.configuredPlugins = Collections.unmodifiableMap(plugins);
+  }
+
+  /**
+   * @param name the name of the formatplugin instance in the drill config
+   * @return The configured FormatPlugin for this name
+   */
+  FormatPlugin getFormatPluginByName(String name) {
+return configuredPlugins.get(name);
+  }
 
-return plugins;
+  /**
+   * @return all the format plugins from the Drill config
+   */
+  Collection getConfiguredFormatPlugins() {
+return configuredPlugins.values();
   }
 
+  /**
+   * Instantiate a new format plugin instance from the provided config 
object
+   * @param fpconfig the conf for the plugin
+   * @return the newly created instance of a FormatPlugin based on 
provided config
+   */
+  FormatPlugin newFormatPlugin(FormatPluginConfig fpconfig) {
+Constructor c = configConstructors.get(fpconfig.getClass());
+if (c == null) {
+  throw new RuntimeException("Unable to find constructor for storage 
config of type " + fpconfig.getClass().getName());
+}
+try {
+  return (FormatPlugin) c.newInstance(null, context, fsConf, 
storageConfig, fpconfig);
+} catch (InstantiationException | IllegalAccessException | 
IllegalArgumentException | InvocationTargetException e1) {
+  throw new RuntimeException("Failure initializing storage config of 
type " + fpconfig.getClass().getName(), e1);
--- End diff --

todo: exception


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994839#comment-14994839
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204830
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -221,9 +603,46 @@ public void dropView(String viewName) throws 
IOException {
   return viewSet;
 }
 
+private Set rawTableNames() {
+  return newHashSet(
+  transform(tables.keySet(), new 
com.google.common.base.Function() {
+@Override
+public String apply(TableInstance input) {
+  return input.sig.name;
+}
+  }));
+}
+
 @Override
 public Set getTableNames() {
-  return Sets.union(tables.keySet(), getViews());
+  System.out.println("getTableNames");
+  return Sets.union(rawTableNames(), getViews());
+}
+
+@Override
+public Set getFunctionNames() {
+  System.out.println("getFunctionNames");
+  return rawTableNames();
+}
+
+@Override
+public List getFunctions(String name) {
+  System.out.println("getFunctions(" + name + ")");
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994838#comment-14994838
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204824
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -221,9 +603,46 @@ public void dropView(String viewName) throws 
IOException {
   return viewSet;
 }
 
+private Set rawTableNames() {
+  return newHashSet(
+  transform(tables.keySet(), new 
com.google.common.base.Function() {
+@Override
+public String apply(TableInstance input) {
+  return input.sig.name;
+}
+  }));
+}
+
 @Override
 public Set getTableNames() {
-  return Sets.union(tables.keySet(), getViews());
+  System.out.println("getTableNames");
+  return Sets.union(rawTableNames(), getViews());
+}
+
+@Override
+public Set getFunctionNames() {
+  System.out.println("getFunctionNames");
+  return rawTableNames();
+}
+
+@Override
+public List getFunctions(String name) {
+  System.out.println("getFunctions(" + name + ")");
+  List sigs = optionExtractor.getTableSignatures(name);
+  System.out.println(sigs);
+//  List sigs = Arrays.asList(
+//  new TableSignature(name, new TableParamDef("delimiter", 
String.class, true)),
+//  new TableSignature(name, new TableParamDef("delimiter", 
Integer.TYPE, true)),
+//  new TableSignature(name, new TableParamDef("foo", 
Integer.TYPE, true), new TableParamDef("bar", Integer.TYPE, true)),
+//  new TableSignature(name, new TableParamDef("foo", 
String.class, true))
+//  );
--- End diff --

todo:cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994842#comment-14994842
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204847
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -313,19 +734,29 @@ public String getTypeName() {
   return FileSystemConfig.NAME;
 }
 
+private DrillTable isReadable(FormatMatcher m,  FileSelection 
fileSelection) throws IOException {
+  return m.isReadable(fs, fileSelection, plugin, storageEngineName, 
schemaConfig.getUserName());
+}
+
 @Override
-public DrillTable create(String key) {
+public DrillTable create(TableInstance key) {
   try {
 
-FileSelection fileSelection = FileSelection.create(fs, 
config.getLocation(), key);
+FileSelection fileSelection = FileSelection.create(fs, 
config.getLocation(), key.sig.name);
 if (fileSelection == null) {
   return null;
 }
-
+if (key.sig.params.size() > 0) {
+  FormatPluginConfig fconfig = optionExtractor.eval(key);
+//  TextFormatPlugin.TextFormatConfig fconfig = new 
TextFormatPlugin.TextFormatConfig();
+//  fconfig.extensions = Arrays.asList();
+//  fconfig.fieldDelimiter = ((String)key.params.get(0)).charAt(0);
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994835#comment-14994835
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204782
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java ---
@@ -86,7 +87,19 @@ public QueryContext(final UserSession session, final 
DrillbitContext drillbitCon
 executionControls = new ExecutionControls(queryOptions, 
drillbitContext.getEndpoint());
 plannerSettings = new PlannerSettings(queryOptions, 
getFunctionRegistry());
 plannerSettings.setNumEndPoints(drillbitContext.getBits().size());
+//boolean caseSensitive = false;
+//CalciteCatalogReader catalogReader =
+//new CalciteCatalogReader(
+//getRootSchema(),
+//caseSensitive,
+//getNewDefaultSchema(),
+//getTypeFactory());
 table = new DrillOperatorTable(getFunctionRegistry());
+//table = new ChainedSqlOperatorTable(asList(
+//  SqlStdOperatorTable.instance(),
+//  catalogReader,
+//  new DrillOperatorTable(getFunctionRegistry())
+//));
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994844#comment-14994844
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204861
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import static java.lang.String.format;
+import static org.apache.drill.TestBuilder.listOf;
+
+import java.io.File;
+import java.io.FileWriter;
+
+import org.junit.Test;
+
+public class TestSelectWithOption extends BaseTestQuery {
+
+//  @Test
+//  public void testBar() throws Exception {
+//test("select dfs.`${WORKING_PATH}/some/path`() from 
cp.`tpch/region.parquet`");
+//  }
+
+  @Test
+  public void testText() throws Exception {
+File input = new File("target/" + this.getClass().getName() + ".csv");
+String query = "select columns from table(dfs.`${WORKING_PATH}/" + 
input.getPath() +
+"`(type => 'TEXT', fieldDelimiter => '%s'))";
+String queryComma = format(query, ",");
+String queryPipe = format(query, "|");
+System.out.println(queryComma);
+System.out.println(queryPipe);
+TestBuilder builderComma = testBuilder()
+.sqlQuery(queryComma)
+.ordered()
+.baselineColumns("columns");
+TestBuilder builderPipe = testBuilder()
+.sqlQuery(queryPipe)
+.ordered()
+.baselineColumns("columns");
+try (FileWriter fw = new FileWriter(input)) {
+//  fw.append("a|b\n");
+  for (int i = 0; i < 3; i++) {
+fw.append("\"b\"|\"" + i + "\"\n");
+builderComma = builderComma.baselineValues(listOf("b\"|\"" + i));
+builderPipe = builderPipe.baselineValues(listOf("b", 
String.valueOf(i)));
+  }
+}
+
+test("select columns from dfs.`${WORKING_PATH}/" + input.getPath() + 
"`");
+
+builderComma.build().run();
+builderPipe.build().run();
+  }
+
+  @Test
+  public void testParquetFailure() throws Exception {
+File input = new File("target/" + this.getClass().getName() + ".csv");
+try (FileWriter fw = new FileWriter(input)) {
+//  fw.append("a|b\n");
+  for (int i = 0; i < 3; i++) {
+fw.append("\"b\"|\"" + i + "\"\n");
+  }
+}
+
+String query = "select columns from table(dfs.`${WORKING_PATH}/" + 
input.getPath() +
+"`(type => 'PARQUET'))";
+System.out.println(query);
+
+test(query);
+
+  }
--- End diff --

todo more granular tests


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994840#comment-14994840
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44204832
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
 ---
@@ -221,9 +603,46 @@ public void dropView(String viewName) throws 
IOException {
   return viewSet;
 }
 
+private Set rawTableNames() {
+  return newHashSet(
+  transform(tables.keySet(), new 
com.google.common.base.Function() {
+@Override
+public String apply(TableInstance input) {
+  return input.sig.name;
+}
+  }));
+}
+
 @Override
 public Set getTableNames() {
-  return Sets.union(tables.keySet(), getViews());
+  System.out.println("getTableNames");
+  return Sets.union(rawTableNames(), getViews());
+}
+
+@Override
+public Set getFunctionNames() {
+  System.out.println("getFunctionNames");
--- End diff --

todo: cleanup


> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994833#comment-14994833
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

GitHub user julienledem opened a pull request:

https://github.com/apache/drill/pull/246

DRILL-4047: Select with options

This is still work in progress and depends on the following PRs in Calcite:
https://github.com/dremio/calcite/pull/1
https://github.com/apache/calcite/pull/166


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/julienledem/drill select_with_options

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #246


commit 3e152e087ae71f323d09d8361389cc7d1fa2159a
Author: Julien Le Dem 
Date:   2015-10-22T17:49:34Z

initial TableMacro implementation

commit a963bbc9df79f51114037a59ba1a46cccd7a23cf
Author: Julien Le Dem 
Date:   2015-11-06T23:48:28Z

FormatCreator refactor

commit 495cd55894a554330918f87b83e9ebd4e184e02a
Author: Julien Le Dem 
Date:   2015-11-07T01:24:23Z

DRILL-4047: generic format plugin impl




> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)