[jira] [Commented] (DRILL-4514) Add describe schema command

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331419#comment-15331419
 ] 

Arina Ielchiieva commented on DRILL-4514:
-

h4. Implementation approach:

If schema doesn't have workspaces, all schema properties will be returned 
(information shown in webui storage tab).
If schema has workspaces, all schema properties will be returned + properties 
for indicated workspace ("location", "writable", "defaultInputFormat").

Example:

*1. Schema with workspaces*
When we indicate dfs without workspace, default workspace is implied. So result 
for both commands below will be the same.
{code:sql}
describe schema dfs;
describe schema dfs.default;
{code}
{noformat}
{
  "type" : "file",
  "enabled" : true,
  "connection" : "file:///",
  "config" : null,
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv", "bcp" ],
  "delimiter" : ","
},
   ... etc
  },
  "location" : "/",
  "writable" : false,
  "defaultInputFormat" : null
}
{noformat}

Results for dfs.tmp and dfs.default will slightly differ. All common properties 
that belong to schema will be the same ("type", "connection", "formats" etc).
Properties "location", "writable", "defaultInputFormat" will differ since they 
belong to workspace.
{code:sql}
describe schema dfs.tmp;
{code}
{noformat}
{
  "type" : "file",
  "enabled" : true,
  "connection" : "file:///",
  "config" : null,
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv", "bcp" ],
  "delimiter" : ","
},
   ... etc
  },
  "location" : "/tmp",
  "writable" : true,
  "defaultInputFormat" : null
}
{noformat}
*2. Schema without workspaces*
Hive schema doesn't have workspaces so we return all schema properties.
{code:sql}
describe schema hive;
{code}
{noformat}
{
  "type": "hive",
  "enabled": true,
  "configProps": {
"hive.metastore.uris": "",
"javax.jdo.option.ConnectionURL": 
"jdbc:derby:;databaseName=../sample-data/drill_hive_db;create=true",
"hive.metastore.warehouse.dir": "/tmp/drill_hive_wh",
"fs.default.name": "file:///",
"hive.metastore.sasl.enabled": "false"
  }
}
{noformat}

> Add describe schema  command
> -
>
> Key: DRILL-4514
> URL: https://issues.apache.org/jira/browse/DRILL-4514
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: Future
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>
> Add describe database  command which will return directory 
> associated with a database on the fly.
> Syntax:
> describe database 
> describe schema 
> Output:
> {noformat}
>  DESCRIBE SCHEMA xdf.proc;
> +-++
> |name | location   |
> +-++
> | xdf.proc| maprfs://dl.data/processed |
> +-++
> {noformat}
> Current implementation covers only dfs schema.
> For all other "" will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4514) Add describe schema command

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-4514:

Description: 
Add describe database  command which will return directory associated 
with a database on the fly.

Syntax:
describe database 
describe schema 

Output:
{code:sql}
 DESCRIBE SCHEMA dfs.tmp;
{code}
{noformat}
++
| properties |
++
| {
  "type" : "file",
  "enabled" : true,
  "connection" : "file:///",
  "config" : null,
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv" ],
  "delimiter" : ","
},
"tsv" : {
  "type" : "text",
  "extensions" : [ "tsv" ],
  "delimiter" : "\t"
},
"parquet" : {
  "type" : "parquet"
},
"json" : {
  "type" : "json",
  "extensions" : [ "json" ]
},
"avro" : {
  "type" : "avro"
},
"sequencefile" : {
  "type" : "sequencefile",
  "extensions" : [ "seq" ]
},
"csvh" : {
  "type" : "text",
  "extensions" : [ "csvh" ],
  "extractHeader" : true,
  "delimiter" : ","
}
  },
  "location" : "/tmp",
  "writable" : true,
  "defaultInputFormat" : null
} |
++

{noformat}


  was:
Add describe database  command which will return directory associated 
with a database on the fly.

Syntax:
describe database 
describe schema 

Output:
{code:sql}
 DESCRIBE SCHEMA dfs.tmp;
{code}
{noformat}
++
| properties |
++
| {
  "type" : "file",
  "enabled" : true,
  "connection" : "file:///",
  "config" : null,
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv" ],
  "delimiter" : ","
},
"tsv" : {
  "type" : "text",
  "extensions" : [ "tsv" ],
  "delimiter" : "\t"
},
"parquet" : {
  "type" : "parquet"
},
"json" : {
  "type" : "json",
  "extensions" : [ "json" ]
},
"avro" : {
  "type" : "avro"
},
"sequencefile" : {
  "type" : "sequencefile",
  "extensions" : [ "seq" ]
},
"csvh" : {
  "type" : "text",
  "extensions" : [ "csvh" ],
  "extractHeader" : true,
  "delimiter" : ","
}
  },
  "location" : "/tmp",
  "writable" : true,
  "defaultInputFormat" : null
} |
++

{noformat}

Current implementation covers only dfs schema.
For all other "" will be returned.




> Add describe schema  command
> -
>
> Key: DRILL-4514
> URL: https://issues.apache.org/jira/browse/DRILL-4514
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: Future
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>
> Add describe database  command which will return directory 
> associated with a database on the fly.
> Syntax:
> describe database 
> describe schema 
> Output:
> {code:sql}
>  DESCRIBE SCHEMA dfs.tmp;
> {code}
> {noformat}
> ++
> | properties |
> ++
> | {
>   "type" : "file",
>   "enabled" : true,
>   "connection" : "file:///",
>   "config" : null,
>   "formats" : {
> "psv" : {
>   "type" : "text",
>   "extensions" : [ "tbl" ],
>   "delimiter" : "|"
> },
> "csv" : {
>   "type" : "text",
>   "extensions" : [ "csv" ],
>   "delimiter" : ","
> },
> "tsv" : {
>   "type" : "text",
>   "extensions" : [ "tsv" ],
>   "delimiter" : "\t"
> },
> "parquet" : {
>   "type" : "parquet"
> },
> "json" : {
>   "type" : "json",
>   "extensions" : [ "json" ]
> },
> "avro" : {
>   "type" : "avro"
> },
> "sequencefile" : {
>   "type" : "sequencefile",
>   "extensions" : [ "seq" ]
> },
> "csvh" : {
>   "type" : "text",
>   "extensions" : [ "csvh" ],
>   "extractHeader" : true,
>   "delimiter" : ","
> }
>   },
>   "location" : "/tmp",
>   "writable" : true,
>   "defaultInputFormat" : null
> } |
> ++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4514) Add describe schema command

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-4514:

Description: 
Add describe database  command which will return directory associated 
with a database on the fly.

Syntax:
describe database 
describe schema 

Output:
{code:sql}
 DESCRIBE SCHEMA dfs.tmp;
{code}
{noformat}
++
| properties |
++
| {
  "type" : "file",
  "enabled" : true,
  "connection" : "file:///",
  "config" : null,
  "formats" : {
"psv" : {
  "type" : "text",
  "extensions" : [ "tbl" ],
  "delimiter" : "|"
},
"csv" : {
  "type" : "text",
  "extensions" : [ "csv" ],
  "delimiter" : ","
},
"tsv" : {
  "type" : "text",
  "extensions" : [ "tsv" ],
  "delimiter" : "\t"
},
"parquet" : {
  "type" : "parquet"
},
"json" : {
  "type" : "json",
  "extensions" : [ "json" ]
},
"avro" : {
  "type" : "avro"
},
"sequencefile" : {
  "type" : "sequencefile",
  "extensions" : [ "seq" ]
},
"csvh" : {
  "type" : "text",
  "extensions" : [ "csvh" ],
  "extractHeader" : true,
  "delimiter" : ","
}
  },
  "location" : "/tmp",
  "writable" : true,
  "defaultInputFormat" : null
} |
++

{noformat}

Current implementation covers only dfs schema.
For all other "" will be returned.



  was:
Add describe database  command which will return directory associated 
with a database on the fly.

Syntax:
describe database 
describe schema 

Output:

{noformat}
 DESCRIBE SCHEMA xdf.proc;

+-++
|name | location   |
+-++
| xdf.proc| maprfs://dl.data/processed |
+-++
{noformat}

Current implementation covers only dfs schema.
For all other "" will be returned.




> Add describe schema  command
> -
>
> Key: DRILL-4514
> URL: https://issues.apache.org/jira/browse/DRILL-4514
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: Future
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>
> Add describe database  command which will return directory 
> associated with a database on the fly.
> Syntax:
> describe database 
> describe schema 
> Output:
> {code:sql}
>  DESCRIBE SCHEMA dfs.tmp;
> {code}
> {noformat}
> ++
> | properties |
> ++
> | {
>   "type" : "file",
>   "enabled" : true,
>   "connection" : "file:///",
>   "config" : null,
>   "formats" : {
> "psv" : {
>   "type" : "text",
>   "extensions" : [ "tbl" ],
>   "delimiter" : "|"
> },
> "csv" : {
>   "type" : "text",
>   "extensions" : [ "csv" ],
>   "delimiter" : ","
> },
> "tsv" : {
>   "type" : "text",
>   "extensions" : [ "tsv" ],
>   "delimiter" : "\t"
> },
> "parquet" : {
>   "type" : "parquet"
> },
> "json" : {
>   "type" : "json",
>   "extensions" : [ "json" ]
> },
> "avro" : {
>   "type" : "avro"
> },
> "sequencefile" : {
>   "type" : "sequencefile",
>   "extensions" : [ "seq" ]
> },
> "csvh" : {
>   "type" : "text",
>   "extensions" : [ "csvh" ],
>   "extractHeader" : true,
>   "delimiter" : ","
> }
>   },
>   "location" : "/tmp",
>   "writable" : true,
>   "defaultInputFormat" : null
> } |
> ++
> {noformat}
> Current implementation covers only dfs schema.
> For all other "" will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4514) Add describe schema command

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331432#comment-15331432
 ] 

ASF GitHub Bot commented on DRILL-4514:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/436
  
@jinfengni 
PR is updated, physical_location is removed.


> Add describe schema  command
> -
>
> Key: DRILL-4514
> URL: https://issues.apache.org/jira/browse/DRILL-4514
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: Future
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>
> Add describe database  command which will return directory 
> associated with a database on the fly.
> Syntax:
> describe database 
> describe schema 
> Output:
> {code:sql}
>  DESCRIBE SCHEMA dfs.tmp;
> {code}
> {noformat}
> ++
> | properties |
> ++
> | {
>   "type" : "file",
>   "enabled" : true,
>   "connection" : "file:///",
>   "config" : null,
>   "formats" : {
> "psv" : {
>   "type" : "text",
>   "extensions" : [ "tbl" ],
>   "delimiter" : "|"
> },
> "csv" : {
>   "type" : "text",
>   "extensions" : [ "csv" ],
>   "delimiter" : ","
> },
> "tsv" : {
>   "type" : "text",
>   "extensions" : [ "tsv" ],
>   "delimiter" : "\t"
> },
> "parquet" : {
>   "type" : "parquet"
> },
> "json" : {
>   "type" : "json",
>   "extensions" : [ "json" ]
> },
> "avro" : {
>   "type" : "avro"
> },
> "sequencefile" : {
>   "type" : "sequencefile",
>   "extensions" : [ "seq" ]
> },
> "csvh" : {
>   "type" : "text",
>   "extensions" : [ "csvh" ],
>   "extractHeader" : true,
>   "delimiter" : ","
> }
>   },
>   "location" : "/tmp",
>   "writable" : true,
>   "defaultInputFormat" : null
> } |
> ++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-4724:
--
Assignee: Chunhui Shi

> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257) 
> [drill-java-ex

[jira] [Created] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4724:
-

 Summary: convert_from(binary_string(expression),'INT_BE') results 
in Exception
 Key: DRILL-4724
 URL: https://issues.apache.org/jira/browse/DRILL-4724
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types
Affects Versions: 1.7.0
Reporter: Khurram Faraaz


The below query that uses binary_string function results in Exception

Drill git commit ID : 6286c0a4

{noformat}
2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
289ee213-8ada-808f-e59d-5a6b67c53732: 
values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.


[Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalArgumentException: Wrong length 8(8-0) in the buffer 
'\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.


[Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
 [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_101]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Internal error: Error while applying 
rule ReduceExpressionsRule_Project, args 
[rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule 
ReduceExpressionsRule_Project, args 
[rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
at org.apache.calcite.util.Util.newInternal(Util.java:792) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
 ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978) 
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257) 
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
... 3 common frames omitted
Caused by: java.lang.RuntimeException: Error in evaluating function of 
convert_fromint_be
at 
org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:323)
 ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at 
org.apache.d

[jira] [Commented] (DRILL-4476) Enhance Union-All operator for dealing with empty left input or empty both inputs

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331511#comment-15331511
 ] 

Khurram Faraaz commented on DRILL-4476:
---

As per the last comment, this fix needs to be backed out of master ? Was this 
backed out ?

> Enhance Union-All operator for dealing with empty left input or empty both 
> inputs
> -
>
> Key: DRILL-4476
> URL: https://issues.apache.org/jira/browse/DRILL-4476
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.7.0
>
>
> Union-All operator does not deal with the situation where left side comes 
> from empty source.
> Due to DRILL-2288's enhancement for empty sources, Union-All operator now can 
> be allowed to support this scenario.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4476) Enhance Union-All operator for dealing with empty left input or empty both inputs

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331510#comment-15331510
 ] 

Khurram Faraaz commented on DRILL-4476:
---

As per the last comment, this fix needs to be backed out of master ? Was this 
backed out ?

> Enhance Union-All operator for dealing with empty left input or empty both 
> inputs
> -
>
> Key: DRILL-4476
> URL: https://issues.apache.org/jira/browse/DRILL-4476
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.7.0
>
>
> Union-All operator does not deal with the situation where left side comes 
> from empty source.
> Due to DRILL-2288's enhancement for empty sources, Union-All operator now can 
> be allowed to support this scenario.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4723) GC Memory Allocation Issues

[John Omernik (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331600#comment-15331600
 ] 

John Omernik commented on DRILL-4723:
-

In addition, One thing I noticed was that with my 5 nodes, only 2 or 3 of them 
(depending on which time I run the query) is actually writing during this query 
(before things get bad due to the Heap issues).  I may try reducing 
parallelization and see what happens.

What is level of effort for working to move the writers away from on heap 
memory?  I guess my concern here is doing a comparison to other tools with 
Parquet, large tables, and 1024m block sizes are not "crazy"  and other tools 
can handle this, is there incentive within the drill project to handle Parquet 
writing in a way that use the Direct Memory? 

Thanks

[~adeneche] Given [~jnadeau] and his post on the subject, is a Heap Dump still 
something to look at?  

> GC Memory Allocation Issues
> ---
>
> Key: DRILL-4723
> URL: https://issues.apache.org/jira/browse/DRILL-4723
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.6.0
> Environment: 5 Drill bits, MapR 1.6 release. 84GB of Direct Memory, 
> 24GB of Heap
>Reporter: John Omernik
>  Labels: memory, stability
> Fix For: Future
>
> Attachments: node3.log
>
>
> This issue is reposted from the Drill User List.  More logs available on 
> request in comments (please specify which logs you may want)
> High level: Drill Cluster gets into a bad state when this occurs. 
> This is what I have thus far... I can provide more complete logs on a one on 
> one basis. 
> The cluster was completely mine, with fresh logs. I ran a CTAS query on a 
> large table that over 100 fields. This query works well in other cases, 
> however I was working with the Block size, both in MapR FS and Drill Parquet. 
> I had successfully tested 512m on each, this case was different. Here are the 
> facts in this setup:
> - No Compression in MapRFS - Using Standard Parquet Snappy Compression
> - MapR Block Size 1024m
> - Parquet Block size 1024m
> - Query  ends up disappearing in the profiles
> - The UI page listing bits only show 4 bits however 5 are running (node 03 
> process is running, but no longer in the bit)
> - Error (copied below)  from rest API
> - No output in STD out or STD error on node3. Only two nodes actually had 
> "Parquet Writing" logs. The other three on Stdout, did not have any 
> issues/errors/
> - I have standard log files, gclogs, the profile.json (before it 
> disappeared), and the physical plan.  Only some components that looked 
> possibly at issue included here
> - The Node 3 GC log shows a bunch of "Full GC Allocation Failures"  that take 
> 4 seconds or more (When I've seen this in other cases, this time can balloon 
> to 8 secs or more)
> - The node 3 output log show some issues with really long RPC issues. Perhaps 
> the GCs prevent RPC communication and create a snowball loop effect?
> Other logs if people are interested can be provided upon request. I just 
> didn't want to flood the whole list with all the logs. 
> Thanks!
> John
> Rest Error:
> ./load_day.py 2016-05-09
> Drill Rest Endpoint: https://drillprod.marathonprod.zeta:2
> Day: 2016-05-09
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: 
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
> certificate verification is strongly advised. See: 
> https://urllib3.readthedocs.org/en/latest/security.html
>   InsecureRequestWarning)
> Authentication successful
> Error encountered: 500
> {
>   "errorMessage" : "SYSTEM ERROR: ForemanException: One more more nodes lost 
> connectivity during query.  Identified nodes were 
> [atl1ctuzeta03:20001].\n\n\n[Error Id: d7dd0120-f7c0-44ef-ac54-29c746b26354 
> on atl1ctuzeta01:20001"
> }
> Possible issue in Node3 Log:
> 2016-06-14 17:25:27,860 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:90] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:90: State to report: RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State to report: RUNNING
> 2016-06-14 17:43:41,869 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 
> took longer than 500ms.  Actual duration was 4192ms.
> 2016-06-14 17:45:36,720 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0

[jira] [Commented] (DRILL-4723) GC Memory Allocation Issues

[Deneche A. Hakim (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331607#comment-15331607
 ] 

Deneche A. Hakim commented on DRILL-4723:
-

The heap dump will most likely confirm what [~jacq...@dremio.com] said, but I 
would still like to take a look at it, thanks.

> GC Memory Allocation Issues
> ---
>
> Key: DRILL-4723
> URL: https://issues.apache.org/jira/browse/DRILL-4723
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.6.0
> Environment: 5 Drill bits, MapR 1.6 release. 84GB of Direct Memory, 
> 24GB of Heap
>Reporter: John Omernik
>  Labels: memory, stability
> Fix For: Future
>
> Attachments: node3.log
>
>
> This issue is reposted from the Drill User List.  More logs available on 
> request in comments (please specify which logs you may want)
> High level: Drill Cluster gets into a bad state when this occurs. 
> This is what I have thus far... I can provide more complete logs on a one on 
> one basis. 
> The cluster was completely mine, with fresh logs. I ran a CTAS query on a 
> large table that over 100 fields. This query works well in other cases, 
> however I was working with the Block size, both in MapR FS and Drill Parquet. 
> I had successfully tested 512m on each, this case was different. Here are the 
> facts in this setup:
> - No Compression in MapRFS - Using Standard Parquet Snappy Compression
> - MapR Block Size 1024m
> - Parquet Block size 1024m
> - Query  ends up disappearing in the profiles
> - The UI page listing bits only show 4 bits however 5 are running (node 03 
> process is running, but no longer in the bit)
> - Error (copied below)  from rest API
> - No output in STD out or STD error on node3. Only two nodes actually had 
> "Parquet Writing" logs. The other three on Stdout, did not have any 
> issues/errors/
> - I have standard log files, gclogs, the profile.json (before it 
> disappeared), and the physical plan.  Only some components that looked 
> possibly at issue included here
> - The Node 3 GC log shows a bunch of "Full GC Allocation Failures"  that take 
> 4 seconds or more (When I've seen this in other cases, this time can balloon 
> to 8 secs or more)
> - The node 3 output log show some issues with really long RPC issues. Perhaps 
> the GCs prevent RPC communication and create a snowball loop effect?
> Other logs if people are interested can be provided upon request. I just 
> didn't want to flood the whole list with all the logs. 
> Thanks!
> John
> Rest Error:
> ./load_day.py 2016-05-09
> Drill Rest Endpoint: https://drillprod.marathonprod.zeta:2
> Day: 2016-05-09
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: 
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
> certificate verification is strongly advised. See: 
> https://urllib3.readthedocs.org/en/latest/security.html
>   InsecureRequestWarning)
> Authentication successful
> Error encountered: 500
> {
>   "errorMessage" : "SYSTEM ERROR: ForemanException: One more more nodes lost 
> connectivity during query.  Identified nodes were 
> [atl1ctuzeta03:20001].\n\n\n[Error Id: d7dd0120-f7c0-44ef-ac54-29c746b26354 
> on atl1ctuzeta01:20001"
> }
> Possible issue in Node3 Log:
> 2016-06-14 17:25:27,860 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:90] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:90: State to report: RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State to report: RUNNING
> 2016-06-14 17:43:41,869 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 
> took longer than 500ms.  Actual duration was 4192ms.
> 2016-06-14 17:45:36,720 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> 2016-06-14 17:45:45,740 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State to report: 
> CANCELLATION_REQUESTED
> 2016-06-14 17:46:15,318 [BitServer-3] WARN  
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
> took longer than 500ms.  Actual duration was 55328ms.
> 2016-06-14 17:46:36,057 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:5: State change requested RUNNI

[jira] [Commented] (DRILL-4723) GC Memory Allocation Issues

[John Omernik (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331778#comment-15331778
 ] 

John Omernik commented on DRILL-4723:
-

[~adeneche] Do you want me to manually force a heap dump when the Allocation 
Failures occur (at this point, I don't think we are out of memory, the Full GC, 
and the delays I think are actually causing cluster stability issues... 
sometimes they get to OOM conditions (I setup the dump on OOM settings). 
However, I wanted to be clear if you wanted the dump prior to OOM. 

In addition, I set my max width per node from 40 to 20 and things worked with 
the 1024 block size. I still had some Full GC issues, but not nearly as many 
and it completed. 

Also, any thoughts on why only 2 out of the 5 nodes are actually doing "writes" 
to parquet would be welcome. 

I would love to see the Parquet write using Direct Memory still this, is a 
challenge to work my way through :) 



> GC Memory Allocation Issues
> ---
>
> Key: DRILL-4723
> URL: https://issues.apache.org/jira/browse/DRILL-4723
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.6.0
> Environment: 5 Drill bits, MapR 1.6 release. 84GB of Direct Memory, 
> 24GB of Heap
>Reporter: John Omernik
>  Labels: memory, stability
> Fix For: Future
>
> Attachments: node3.log
>
>
> This issue is reposted from the Drill User List.  More logs available on 
> request in comments (please specify which logs you may want)
> High level: Drill Cluster gets into a bad state when this occurs. 
> This is what I have thus far... I can provide more complete logs on a one on 
> one basis. 
> The cluster was completely mine, with fresh logs. I ran a CTAS query on a 
> large table that over 100 fields. This query works well in other cases, 
> however I was working with the Block size, both in MapR FS and Drill Parquet. 
> I had successfully tested 512m on each, this case was different. Here are the 
> facts in this setup:
> - No Compression in MapRFS - Using Standard Parquet Snappy Compression
> - MapR Block Size 1024m
> - Parquet Block size 1024m
> - Query  ends up disappearing in the profiles
> - The UI page listing bits only show 4 bits however 5 are running (node 03 
> process is running, but no longer in the bit)
> - Error (copied below)  from rest API
> - No output in STD out or STD error on node3. Only two nodes actually had 
> "Parquet Writing" logs. The other three on Stdout, did not have any 
> issues/errors/
> - I have standard log files, gclogs, the profile.json (before it 
> disappeared), and the physical plan.  Only some components that looked 
> possibly at issue included here
> - The Node 3 GC log shows a bunch of "Full GC Allocation Failures"  that take 
> 4 seconds or more (When I've seen this in other cases, this time can balloon 
> to 8 secs or more)
> - The node 3 output log show some issues with really long RPC issues. Perhaps 
> the GCs prevent RPC communication and create a snowball loop effect?
> Other logs if people are interested can be provided upon request. I just 
> didn't want to flood the whole list with all the logs. 
> Thanks!
> John
> Rest Error:
> ./load_day.py 2016-05-09
> Drill Rest Endpoint: https://drillprod.marathonprod.zeta:2
> Day: 2016-05-09
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: 
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
> certificate verification is strongly advised. See: 
> https://urllib3.readthedocs.org/en/latest/security.html
>   InsecureRequestWarning)
> Authentication successful
> Error encountered: 500
> {
>   "errorMessage" : "SYSTEM ERROR: ForemanException: One more more nodes lost 
> connectivity during query.  Identified nodes were 
> [atl1ctuzeta03:20001].\n\n\n[Error Id: d7dd0120-f7c0-44ef-ac54-29c746b26354 
> on atl1ctuzeta01:20001"
> }
> Possible issue in Node3 Log:
> 2016-06-14 17:25:27,860 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:90] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:90: State to report: RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State to report: RUNNING
> 2016-06-14 17:43:41,869 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 
> took longer than 500ms.  Actual duration was 4192ms.
> 2016-06-14 17:45:36,720 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
>

[jira] [Commented] (DRILL-4723) GC Memory Allocation Issues

[Deneche A. Hakim (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331851#comment-15331851
 ] 

Deneche A. Hakim commented on DRILL-4723:
-

Ideally we want the dump that will get generated automatically when a node 
crashes. But if you don't see any node dying because of OOM then you could try 
to get a manual heap dump.

> GC Memory Allocation Issues
> ---
>
> Key: DRILL-4723
> URL: https://issues.apache.org/jira/browse/DRILL-4723
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.6.0
> Environment: 5 Drill bits, MapR 1.6 release. 84GB of Direct Memory, 
> 24GB of Heap
>Reporter: John Omernik
>  Labels: memory, stability
> Fix For: Future
>
> Attachments: node3.log
>
>
> This issue is reposted from the Drill User List.  More logs available on 
> request in comments (please specify which logs you may want)
> High level: Drill Cluster gets into a bad state when this occurs. 
> This is what I have thus far... I can provide more complete logs on a one on 
> one basis. 
> The cluster was completely mine, with fresh logs. I ran a CTAS query on a 
> large table that over 100 fields. This query works well in other cases, 
> however I was working with the Block size, both in MapR FS and Drill Parquet. 
> I had successfully tested 512m on each, this case was different. Here are the 
> facts in this setup:
> - No Compression in MapRFS - Using Standard Parquet Snappy Compression
> - MapR Block Size 1024m
> - Parquet Block size 1024m
> - Query  ends up disappearing in the profiles
> - The UI page listing bits only show 4 bits however 5 are running (node 03 
> process is running, but no longer in the bit)
> - Error (copied below)  from rest API
> - No output in STD out or STD error on node3. Only two nodes actually had 
> "Parquet Writing" logs. The other three on Stdout, did not have any 
> issues/errors/
> - I have standard log files, gclogs, the profile.json (before it 
> disappeared), and the physical plan.  Only some components that looked 
> possibly at issue included here
> - The Node 3 GC log shows a bunch of "Full GC Allocation Failures"  that take 
> 4 seconds or more (When I've seen this in other cases, this time can balloon 
> to 8 secs or more)
> - The node 3 output log show some issues with really long RPC issues. Perhaps 
> the GCs prevent RPC communication and create a snowball loop effect?
> Other logs if people are interested can be provided upon request. I just 
> didn't want to flood the whole list with all the logs. 
> Thanks!
> John
> Rest Error:
> ./load_day.py 2016-05-09
> Drill Rest Endpoint: https://drillprod.marathonprod.zeta:2
> Day: 2016-05-09
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: 
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
> certificate verification is strongly advised. See: 
> https://urllib3.readthedocs.org/en/latest/security.html
>   InsecureRequestWarning)
> Authentication successful
> Error encountered: 500
> {
>   "errorMessage" : "SYSTEM ERROR: ForemanException: One more more nodes lost 
> connectivity during query.  Identified nodes were 
> [atl1ctuzeta03:20001].\n\n\n[Error Id: d7dd0120-f7c0-44ef-ac54-29c746b26354 
> on atl1ctuzeta01:20001"
> }
> Possible issue in Node3 Log:
> 2016-06-14 17:25:27,860 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:90] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:90: State to report: RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State to report: RUNNING
> 2016-06-14 17:43:41,869 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 
> took longer than 500ms.  Actual duration was 4192ms.
> 2016-06-14 17:45:36,720 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> 2016-06-14 17:45:45,740 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State to report: 
> CANCELLATION_REQUESTED
> 2016-06-14 17:46:15,318 [BitServer-3] WARN  
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
> took longer than 500ms.  Actual duration was 55328ms.
> 2016-06-14 17:46:36,057 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266

[jira] [Commented] (DRILL-4723) GC Memory Allocation Issues

[John Omernik (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331857#comment-15331857
 ] 

John Omernik commented on DRILL-4723:
-

I have the dumps (I just artificially dropped the heap from 24 to 8 GB,
they all crashed).  I will run the MAT tools on it and pass it along.

Thanks!

On Wed, Jun 15, 2016 at 9:40 AM, Deneche A. Hakim (JIRA) 



> GC Memory Allocation Issues
> ---
>
> Key: DRILL-4723
> URL: https://issues.apache.org/jira/browse/DRILL-4723
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Affects Versions: 1.6.0
> Environment: 5 Drill bits, MapR 1.6 release. 84GB of Direct Memory, 
> 24GB of Heap
>Reporter: John Omernik
>  Labels: memory, stability
> Fix For: Future
>
> Attachments: node3.log
>
>
> This issue is reposted from the Drill User List.  More logs available on 
> request in comments (please specify which logs you may want)
> High level: Drill Cluster gets into a bad state when this occurs. 
> This is what I have thus far... I can provide more complete logs on a one on 
> one basis. 
> The cluster was completely mine, with fresh logs. I ran a CTAS query on a 
> large table that over 100 fields. This query works well in other cases, 
> however I was working with the Block size, both in MapR FS and Drill Parquet. 
> I had successfully tested 512m on each, this case was different. Here are the 
> facts in this setup:
> - No Compression in MapRFS - Using Standard Parquet Snappy Compression
> - MapR Block Size 1024m
> - Parquet Block size 1024m
> - Query  ends up disappearing in the profiles
> - The UI page listing bits only show 4 bits however 5 are running (node 03 
> process is running, but no longer in the bit)
> - Error (copied below)  from rest API
> - No output in STD out or STD error on node3. Only two nodes actually had 
> "Parquet Writing" logs. The other three on Stdout, did not have any 
> issues/errors/
> - I have standard log files, gclogs, the profile.json (before it 
> disappeared), and the physical plan.  Only some components that looked 
> possibly at issue included here
> - The Node 3 GC log shows a bunch of "Full GC Allocation Failures"  that take 
> 4 seconds or more (When I've seen this in other cases, this time can balloon 
> to 8 secs or more)
> - The node 3 output log show some issues with really long RPC issues. Perhaps 
> the GCs prevent RPC communication and create a snowball loop effect?
> Other logs if people are interested can be provided upon request. I just 
> didn't want to flood the whole list with all the logs. 
> Thanks!
> John
> Rest Error:
> ./load_day.py 2016-05-09
> Drill Rest Endpoint: https://drillprod.marathonprod.zeta:2
> Day: 2016-05-09
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: 
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding 
> certificate verification is strongly advised. See: 
> https://urllib3.readthedocs.org/en/latest/security.html
>   InsecureRequestWarning)
> Authentication successful
> Error encountered: 500
> {
>   "errorMessage" : "SYSTEM ERROR: ForemanException: One more more nodes lost 
> connectivity during query.  Identified nodes were 
> [atl1ctuzeta03:20001].\n\n\n[Error Id: d7dd0120-f7c0-44ef-ac54-29c746b26354 
> on atl1ctuzeta01:20001"
> }
> Possible issue in Node3 Log:
> 2016-06-14 17:25:27,860 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:90] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:90: State to report: RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-06-14 17:25:27,871 [289fc208-7266-6a81-73a1-709efff6c412:frag:1:70] INFO 
>  o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:70: State to report: RUNNING
> 2016-06-14 17:43:41,869 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 
> took longer than 500ms.  Actual duration was 4192ms.
> 2016-06-14 17:45:36,720 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State change requested RUNNING --> 
> CANCELLATION_REQUESTED
> 2016-06-14 17:45:45,740 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 289fc208-7266-6a81-73a1-709efff6c412:1:0: State to report: 
> CANCELLATION_REQUESTED
> 2016-06-14 17:46:15,318 [BitServer-3] WARN  
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
> took longer than 500ms.  Actual duration was 55328ms.
> 2016-06-14 17:46:36,057 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecut

[jira] [Commented] (DRILL-4476) Enhance Union-All operator for dealing with empty left input or empty both inputs

[Zelaine Fong (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331914#comment-15331914
 ] 

Zelaine Fong commented on DRILL-4476:
-

I don't believe it has been.  [~amansinha100]?

> Enhance Union-All operator for dealing with empty left input or empty both 
> inputs
> -
>
> Key: DRILL-4476
> URL: https://issues.apache.org/jira/browse/DRILL-4476
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.7.0
>
>
> Union-All operator does not deal with the situation where left side comes 
> from empty source.
> Due to DRILL-2288's enhancement for empty sources, Union-All operator now can 
> be allowed to support this scenario.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3149) TextReader should support multibyte line delimiters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332015#comment-15332015
 ] 

ASF GitHub Bot commented on DRILL-3149:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/500
  
+1 looks good. Can you squash and rebase ?


> TextReader should support multibyte line delimiters
> ---
>
> Key: DRILL-3149
> URL: https://issues.apache.org/jira/browse/DRILL-3149
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Text & CSV
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Jim Scott
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: Future
>
>
> lineDelimiter in the TextFormatConfig doesn't support \r\n for record 
> delimiters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4714) Add metadata and prepared statement APIs to DrillClient<->Drillbit interface

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti reassigned DRILL-4714:
--

Assignee: Venki Korukanti

> Add metadata and prepared statement APIs to DrillClient<->Drillbit interface
> 
>
> Key: DRILL-4714
> URL: https://issues.apache.org/jira/browse/DRILL-4714
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
>
> Currently ODBC/JDBC drivers spawn a set of queries on INFORMATION_SCHEMA for 
> metadata. Client has to deal with submitting a query, reading query results 
> and constructing required objects. Sometimes same work is done twice 
> (planning work in case of prepare statements) to get the metadata and execute 
> query. Instead we could simplify the client by providing APIs on the client 
> interface and let the server construct the required objects and send them to 
> client directly. These APIs provide common info that can be consumed by the 
> JDBC/ODBC driver.
> [Here|https://docs.google.com/document/d/1ytwGDYGbEFuS3_eUEqRikh7wQABFyHjS0fXJ6wr1eSU/edit?usp=sharing]
>  is the doc explaining the new APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4725) Improvements to InfoSchema RecordGenerator needed for DRILL-4714

[Venki Korukanti (JIRA)]

Venki Korukanti created DRILL-4725:
--

 Summary: Improvements to InfoSchema RecordGenerator needed for 
DRILL-4714
 Key: DRILL-4725
 URL: https://issues.apache.org/jira/browse/DRILL-4725
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Metadata
Reporter: Venki Korukanti
Assignee: Venki Korukanti


1. Add support for pushing the filter on following fields into 
InfoSchemaRecordGenerator:
   - CATALOG_NAME
   - COLUMN_NAME

2. Pushdown LIKE with ESCAPE. Add test.

3. Add a method visitCatalog() to InfoSchemaRecordGenerator to decide whether 
to explore the catalog or not





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4714) Add metadata and prepared statement APIs to DrillClient<->Drillbit interface

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti updated DRILL-4714:
---
Component/s: Metadata

> Add metadata and prepared statement APIs to DrillClient<->Drillbit interface
> 
>
> Key: DRILL-4714
> URL: https://issues.apache.org/jira/browse/DRILL-4714
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
>
> Currently ODBC/JDBC drivers spawn a set of queries on INFORMATION_SCHEMA for 
> metadata. Client has to deal with submitting a query, reading query results 
> and constructing required objects. Sometimes same work is done twice 
> (planning work in case of prepare statements) to get the metadata and execute 
> query. Instead we could simplify the client by providing APIs on the client 
> interface and let the server construct the required objects and send them to 
> client directly. These APIs provide common info that can be consumed by the 
> JDBC/ODBC driver.
> [Here|https://docs.google.com/document/d/1ytwGDYGbEFuS3_eUEqRikh7wQABFyHjS0fXJ6wr1eSU/edit?usp=sharing]
>  is the doc explaining the new APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332282#comment-15332282
 ] 

ASF GitHub Bot commented on DRILL-4724:
---

GitHub user vkorukanti opened a pull request:

https://github.com/apache/drill/pull/524

DRILL-4724: Improvements to InfoSchema RecordGenerator needed for DRILL-4714

1. Add support for pushing the filter on following fields into 
InfoSchemaRecordGenerator:
   - CATALOG_NAME
   - COLUMN_NAME

2. Pushdown LIKE with ESCAPE. Add test 
TestInfoSchemaFilterPushDown#testFilterPushdown_LikeWithEscape

3. Add a method visitCatalog() to InfoSchemaRecordGenerator to decide 
whether to explore the catalog or not

4. Refactor CATALOG_DESCRIPTION and CATALOG_CONNECT as constant strings in 
InfoSchemaConstants.java

5. Update TestInfoSchemaFilterPushDown#testPartialFilterPushDownWithProject 
as
   we are now pushing the filter on COLUMN_NAME field

6. Cleanup:
   Rename RecordGenerator -> InfoSchemaRecordGenerator
   Add comments in RecordGenerator
   Rename SelectedTable -> InfoSchemaTableType

@parthchandra Can you please review this change?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vkorukanti/drill DRILL-4725

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #524


commit f331507af044d42b74b1fedbf7b9de8adc861334
Author: vkorukanti 
Date:   2016-06-09T07:00:31Z

DRILL-4724: Improvements to InfoSchema RecordGenerator needed for DRILL-4714

1. Add support for pushing the filter on following fields into 
InfoSchemaRecordGenerator:
   - CATALOG_NAME
   - COLUMN_NAME

2. Pushdown LIKE with ESCAPE. Add test 
TestInfoSchemaFilterPushDown#testFilterPushdown_LikeWithEscape

3. Add a method visitCatalog() to InfoSchemaRecordGenerator to decide 
whether to explore the catalog or not

4. Refactor CATALOG_DESCRIPTION and CATALOG_CONNECT as constant strings in 
InfoSchemaConstants.java

5. Update TestInfoSchemaFilterPushDown#testPartialFilterPushDownWithProject 
as
   we are now pushing the filter on COLUMN_NAME field

6. Cleanup:
   Rename RecordGenerator -> InfoSchemaRecordGenerator
   Add comments in RecordGenerator
   Rename SelectedTable -> InfoSchemaTableType

Change-Id: I0b2e16d04cb72fe3ce5961f5f357a00655f1cb05




> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thr

[jira] [Commented] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Venki Korukanti (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332295#comment-15332295
 ] 

Venki Korukanti commented on DRILL-4724:


Ignore the PR, wrong JIRA number

> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.dril

[jira] [Commented] (DRILL-4725) Improvements to InfoSchema RecordGenerator needed for DRILL-4714

[Venki Korukanti (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332297#comment-15332297
 ] 

Venki Korukanti commented on DRILL-4725:


PR: https://github.com/apache/drill/pull/524

> Improvements to InfoSchema RecordGenerator needed for DRILL-4714
> 
>
> Key: DRILL-4725
> URL: https://issues.apache.org/jira/browse/DRILL-4725
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
>
> 1. Add support for pushing the filter on following fields into 
> InfoSchemaRecordGenerator:
>- CATALOG_NAME
>- COLUMN_NAME
> 2. Pushdown LIKE with ESCAPE. Add test.
> 3. Add a method visitCatalog() to InfoSchemaRecordGenerator to decide whether 
> to explore the catalog or not



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4697) Editing of multi-line queries in sqlline

[Matt Keranen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Keranen updated DRILL-4697:

Issue Type: Improvement  (was: Wish)

> Editing of multi-line queries in sqlline
> 
>
> Key: DRILL-4697
> URL: https://issues.apache.org/jira/browse/DRILL-4697
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - CLI
>Affects Versions: 1.6.0
>Reporter: Matt Keranen
>Priority: Minor
>
> Being used to the PostgreSQL psql client, editing multiline queries in 
> sqlline seems difficult. Each line of an unterminated (semicolon) query 
> appears to be a separate history entry, thus uneditable.
> The ability to up-arrow to bring the entire previous query would be very 
> helpful, or the option to send the query text to an external shell editor 
> (Vi).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2385) count on complex objects failed with missing function implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332525#comment-15332525
 ] 

ASF GitHub Bot commented on DRILL-2385:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/501
  
+1 LGTM


> count on complex objects failed with missing function implementation
> 
>
> Key: DRILL-2385
> URL: https://issues.apache.org/jira/browse/DRILL-2385
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 0.8.0
>Reporter: Chun Chang
>Assignee: Vitalii Diravka
>Priority: Minor
> Fix For: 1.7.0
>
>
> #Wed Mar 04 01:23:42 EST 2015
> git.commit.id.abbrev=71b6bfe
> Have a complex type looks like the following:
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.sia from 
> `complex.json` t limit 1;
> ++
> |sia |
> ++
> | [1,11,101,1001] |
> ++
> {code}
> A count on the complex type will fail with missing function implementation:
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.gbyi, count(t.sia) 
> countsia from `complex.json` t group by t.gbyi;
> Query failed: RemoteRpcException: Failure while running fragment., Schema is 
> currently null.  You must call buildSchema(SelectionVectorMode) before this 
> container can return a schema. [ 12856530-3133-45be-bdf4-ef8cc784f7b3 on 
> qa-node119.qa.lab:31010 ]
> [ 12856530-3133-45be-bdf4-ef8cc784f7b3 on qa-node119.qa.lab:31010 ]
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-03-04 13:44:51,383 [2b08832b-9247-e90c-785d-751f02fc1548:frag:2:0] ERROR 
> o.a.drill.exec.ops.FragmentContext - Fragment Context received failure.
> org.apache.drill.exec.exception.SchemaChangeException: Failure while 
> materializing expression.
> Error in expression at index 0.  Error: Missing function implementation: 
> [count(BIGINT-REPEATED)].  Full expression: null.
> at 
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.createAggregatorInternal(HashAggBatch.java:210)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.createAggregator(HashAggBatch.java:158)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.buildSchema(HashAggBatch.java:101)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:130)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:114)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:121)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> 2015-03-04 13:44:51,383 [2b08832b-9247-e90c-785d-751f02fc1548:frag:2:0] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing 
> fragment
> java.lang.NullPointerException: Schema is currently null.  You must call 
> buildSchema(SelectionVectorMode) before this container can return a schema.
> at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208) 
> ~[guava-14.0.1.jar:na]
> at 
> org.apache.drill.exec.record.VectorContainer.getSchema(VectorContainer.java:261)
>  ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.getSchema(AbstractRecordBatch.java:155)

[jira] [Commented] (DRILL-4719) Need To Support IAM role based access for supporting Amazon S3

[subbu srinivasan (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332707#comment-15332707
 ] 

subbu srinivasan commented on DRILL-4719:
-

There is a fix already from the hadoop common project.

In the core-site.xml set the following:



fs.s3a.aws.credentials.provider
com.amazonaws.auth.DefaultAWSCredentialsProviderChain


The access key/id are no longer required.





> Need To Support IAM role based access for supporting Amazon S3
> --
>
> Key: DRILL-4719
> URL: https://issues.apache.org/jira/browse/DRILL-4719
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Other
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
>  Labels: security
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> We need amazon secret accessid/credentials as part of the core-site.xml.
> This is not ideal in many deployments, we would use IAM roles to accomplish
> access to s3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Chunhui Shi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332880#comment-15332880
 ] 

Chunhui Shi commented on DRILL-4724:


Hi Khurram, it should not be double slash '\\' but one slash instead.

My test result is this:
0: jdbc:drill:zk=local> select
convert_from(binary_string('\x99\x8c\x2f\x77'),'INT_BE') from (values(1));
+--+
+--+
+--+


On Wed, Jun 15, 2016 at 3:09 AM, Khurram Faraaz (JIRA) 
wrote:



> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.plan

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332946#comment-15332946
 ] 

ASF GitHub Bot commented on DRILL-4653:
---

Github user ssriniva123 commented on the issue:

https://github.com/apache/drill/pull/518
  
For some reason the build terminated- What needs to be done to fire it 
again?



> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-4688) String functions may produce incorrect result when input has multi-byte character

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni closed DRILL-4688.
-
   Resolution: Fixed
Fix Version/s: 1.7.0

Fixed in commit: 2e2b54af5379f7046e84390e70f3862cddd93195

> String functions may produce incorrect result when input has multi-byte 
> character
> -
>
> Key: DRILL-4688
> URL: https://issues.apache.org/jira/browse/DRILL-4688
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jinfeng Ni
>Priority: Critical
> Fix For: 1.7.0
>
>
> As discussed in DRILL-4573, the patch of DRILL-4573 would cause regression  
> of query correctness, when the input, encoded as utf-8, contains multi-byte 
> characters.
> For example,
> {code:sql}
> select regexp_matches('München', 'München') res3 from (values(1));
> ++
> | res3 |
> ++
> | false |
> ++
> {code}
> Here is the result before the patch of DRILL-4573. 
>  
> {code}
> select regexp_matches('München', 'München') res3 from (values(1));
> +---+
> | res3 |
> +---+
> | true |
> +---+
> {code}
> Once this issue has been resolved, QA would add functional testcases to cover 
> the case of multip-byte characters, so that we will be able to catch such 
> regression in the first place in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-4573) Zero copy LIKE, REGEXP_MATCHES, SUBSTR

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni closed DRILL-4573.
-
Resolution: Fixed

Fixed in commit: 2e2b54af5379f7046e84390e70f3862cddd93195

> Zero copy LIKE, REGEXP_MATCHES, SUBSTR
> --
>
> Key: DRILL-4573
> URL: https://issues.apache.org/jira/browse/DRILL-4573
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: jean-claude
>Priority: Critical
> Fix For: 1.7.0
>
> Attachments: DRILL-4573-3.patch.txt, DRILL-4573.patch.txt
>
>
> All the functions using the java.util.regex.Matcher are currently creating 
> Java string objects to pass into the matcher.reset().
> However this creates unnecessary copy of the bytes and a Java string object.
> The matcher uses a CharSequence, so instead of making a copy we can create an 
> adapter from the DrillBuffer to the CharSequence interface.
> Gains of 25% in execution speed are possible when going over VARCHAR of 36 
> chars. The gain will be proportional to the size of the VARCHAR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333047#comment-15333047
 ] 

Khurram Faraaz commented on DRILL-4724:
---

Ok, but the data used in the unit tests had two back slashes used in it.
Do we plan to fix the assertion error seen in the stack trace ?

> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL

[jira] [Commented] (DRILL-4530) Improve metadata cache performance for queries with single partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333132#comment-15333132
 ] 

ASF GitHub Bot commented on DRILL-4530:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/519#discussion_r67289117
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
 ---
@@ -208,8 +209,18 @@ public DrillTable isReadable(DrillFileSystem fs, 
FileSelection selection,
 FileSystemPlugin fsPlugin, String storageEngineName, String 
userName)
 throws IOException {
   // TODO: we only check the first file for directory reading.
-  if(selection.containsDirectories(fs)){
-if(isDirReadable(fs, selection.getFirstPath(fs))){
+  if(selection.containsDirectories(fs)) {
+Path dirMetaPath = new Path(selection.getSelectionRoot(), 
Metadata.METADATA_DIRECTORIES_FILENAME);
+if (fs.exists(dirMetaPath)) {
+  ParquetTableMetadataDirs mDirs = Metadata.readMetadataDirs(fs, 
dirMetaPath.toString());
+  if (mDirs.getDirectories().size() > 0) {
+FileSelection dirSelection = 
FileSelection.createFromDirectories(mDirs.getDirectories(), selection);
+dirSelection.setExpandedPartial();
+return new DynamicDrillTable(fsPlugin, storageEngineName, 
userName,
--- End diff --

I intentionally don't call isDirReadable() here because that method returns 
true if a metadata cache file exists and I am doing a similar check for the 
directories file here with fs.exists(dirMetaPath).If this check fails,  we 
will fall through to the old code path (line 223) which does check 
isDirReadable().   


> Improve metadata cache performance for queries with single partition 
> -
>
> Key: DRILL-4530
> URL: https://issues.apache.org/jira/browse/DRILL-4530
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 1.6.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.7.0
>
>
> Consider two types of queries which are run with Parquet metadata caching: 
> {noformat}
> query 1:
> SELECT col FROM  `A/B/C`;
> query 2:
> SELECT col FROM `A` WHERE dir0 = 'B' AND dir1 = 'C';
> {noformat}
> For a certain dataset, the query1 elapsed time is 1 sec whereas query2 
> elapsed time is 9 sec even though both are accessing the same amount of data. 
>  The user expectation is that they should perform roughly the same.  The main 
> difference comes from reading the bigger metadata cache file at the root 
> level 'A' for query2 and then applying the partitioning filter.  query1 reads 
> a much smaller metadata cache file at the subdirectory level. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333142#comment-15333142
 ] 

ASF GitHub Bot commented on DRILL-4653:
---

Github user ssriniva123 commented on the issue:

https://github.com/apache/drill/pull/518
  
I updated the pull request with more changes requested by the reviewers.

On Wed, Jun 15, 2016 at 5:28 PM, Aman Sinha 
wrote:

> Yes, it does in fact have a conflict with DRILL-3764 which has changes to
> the JsonRecordReader, although this issue is still in progress. I noticed
> that @adeneche  mentioned this in the JIRA.
> @ssriniva123  did you get a chance to
> look at DRILL-3764 ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4724) convert_from(binary_string(expression),'INT_BE') results in Exception

[Chunhui Shi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333158#comment-15333158
 ] 

Chunhui Shi commented on DRILL-4724:


When the format is not valid, we raised some exception, While the exception 
text is not ideal I think it is still acceptable. 

> convert_from(binary_string(expression),'INT_BE') results in Exception
> -
>
> Key: DRILL-4724
> URL: https://issues.apache.org/jira/browse/DRILL-4724
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
>
> The below query that uses binary_string function results in Exception
> Drill git commit ID : 6286c0a4
> {noformat}
> 2016-06-15 09:20:43,623 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 289ee213-8ada-808f-e59d-5a6b67c53732: 
> values(convert_from(binary_string('\\x99\\x8c\\x2f\\x77'),'INT_BE'))
> 2016-06-15 09:20:43,666 [289ee213-8ada-808f-e59d-5a6b67c53732:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException: 
> Wrong length 8(8-0) in the buffer '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalArgumentException: Wrong length 8(8-0) in the buffer 
> '\x5C\x99\x5C\x8C\x5C/\x5Cw', expected 4.
> [Error Id: bb6968cd-44c2-4c48-bb12-865f8709167e on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#1460:LogicalProject.NONE.ANY([]).[](input=rel#1459:Subset#0.NONE.ANY([]).[0],EXPR$0=CONVERT_FROMINT_BE(BINARY_STRING('\\x99\\x8c\\x2f\\x77')))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r11.jar:1.4.0-drill-r11]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:339)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:237)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:286)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:9