Drill 1.8.0 User Authentication with a custom authenticator

[Sudip Mukherjee]

Hi,
I'm using drill 1.8.0 and I have a custom authenticator implementation 
following below steps :
https://drill.apache.org/docs/configuring-user-authentication/

Implementing and Configuring a Custom Authenticator
Administrators can use the template provided here to develop and implement a 
custom username/password based authenticator.
Complete the following steps to build and implement a custom authenticator:


When I try to get the logged in user from schemaConfig.getUserName() it gives 
me process user name which was not the case while I was using drill 1.4.0.

Could you please help. I use the logged in user name to validate against a SOLR 
source (this storage plugin is not published)

Thanks,
Sudip
***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

DRILL JDBC Driver setFetchSize

[Sudip Mukherjee]

Hi,

Should the setFetchSize work similar to applying LIMIT clause with drill JDBC 
driver? I've set setFetchSize to 
java.sql.Statement
 but the query still gets records beyond the fetchSIze.

Thanks,
Sudip

RE: Load JDBC service ServiceLoader

[Sudip Mukherjee]

Hi Laurent,
Thanks for your comment. I've created a Jira DRILL-4880.

Sudip

-Original Message-
From: Laurent Goujon [mailto:laur...@dremio.com] 
Sent: Wednesday, September 07, 2016 4:33 AM
To: dev@drill.apache.org
Subject: Re: Load JDBC service ServiceLoader

That sounds like a good idea. The JDBC example has a
Class.forName("org.apache.drill.jdbc.Driver") statement to force driver 
registration, but a ServiceLoader registration is pretty standard. Maybe you 
should open a bug for it?

Laurent

On Tue, Sep 6, 2016 at 9:53 AM, Sudip Mukherjee <smukher...@commvault.com>
wrote:

> Hi,
> I am trying out a java program where I want to load drill JDBC driver 
> using ServiceLoader.load like all other jdbc drivers. But the code is 
> failing to load org.apache.drill.jdbc.Driver class. One possible 
> reason Is that the drill jdbc driver jar doesn't have a 
> java.sql.Driver file under META_INF/services which is common for all the 
> other driver jars.
>
> Can the drill jdbc driver jar come with the driver file?
>
> Thanks,
> Sudip
>

[jira] [Created] (DRILL-4880) Support JDBC driver registration using ServiceLoader

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-4880:
--

 Summary: Support JDBC driver registration using ServiceLoader 
 Key: DRILL-4880
 URL: https://issues.apache.org/jira/browse/DRILL-4880
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 1.8.0
 Environment: Windows Server 2012
Reporter: Sudip Mukherjee
 Fix For: 1.9.0


Currently drill-jdbc-all*.jar doesn't contain a 
META_INF/services/java.sql.Driver file which is apparently used to discover a 
service by Java ServiceLoader API.

Can drill jdbc driver have this file like all the other jdbc drivers so that 
the driver can be loaded using ServiceLoader instead of a direct Class.forName?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Load JDBC service ServiceLoader

[Sudip Mukherjee]

Hi,
I am trying out a java program where I want to load drill JDBC driver using 
ServiceLoader.load like all other jdbc drivers. But the code is failing to load 
org.apache.drill.jdbc.Driver class. One possible reason
Is that the drill jdbc driver jar doesn't have a java.sql.Driver file under 
META_INF/services which is common for all the other driver jars.

Can the drill jdbc driver jar come with the driver file?

Thanks,
Sudip

RE: Drill service start/stop

[Sudip Mukherjee]

Hi,
Drill service was stopped while I was using it through tableau. I restarted the 
service but I couldn't continue with the current tableau workspace as the 
queries were not reaching the drill server.
Is this something expected. Could you please suggest troubleshooting steps if 
there's any?

Thanking you.

Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Drill service start/stop

[Sudip Mukherjee]

Hi,
Drill service was stopped while I was using it through tableau. I restarted the 
service but I couldn't continue with the current tableau workspace as the 
queries were not reaching the drill server.
Is this something expected. Could you please suggest troubleshooting steps if 
there's any?

Thanking you.

Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Schema validation in Drill

[Sudip Mukherjee]

Hi,

I've been experimenting with Apache Solr storage plugin with DRILL and have a 
doubt whether we can query on solr dynamic fields existing in a solr core's 
schema.
there is a schema validation which says that column not found which is 
expected. Is there way to loosen the schema check or may be allow dynamic field?

It might not sound practical but just asking...

Thanks,
Sudip




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[jira] [Created] (DRILL-4516) Transform SUM(1) query to COUNT(1)

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-4516:
--

 Summary: Transform SUM(1) query to COUNT(1)
 Key: DRILL-4516
 URL: https://issues.apache.org/jira/browse/DRILL-4516
 Project: Apache Drill
  Issue Type: Improvement
  Components: Query Planning & Optimization
Reporter: Sudip Mukherjee


If we connect drill with tableau we see some query requests like , select 
sum(1) tablename. 
This results in pulling all the records out of the underlying datasource and 
aggregate them to get row count.
The behavior can be optimized if the query gets transformed into a count(1) 
query which is likely to be optimized at the datasource level



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Optimizing SUM(1) query

[Sudip Mukherjee]

Hi Jacques,

Thanks for your comments. I've created a JIRA for improvement in future release
https://issues.apache.org/jira/browse/DRILL-4516

Thanks,
Sudip

-Original Message-
From: Jacques Nadeau [mailto:jacq...@dremio.com] 
Sent: 17 March 2016 AM 02:33
To: dev
Subject: Re: Optimizing SUM(1) query

I don't think Julian is saying it does this, I think he is saying it should. I 
agree. (This actually is very common Tableau query pattern among other things.)

Sudip, do you want to open an enhancement JIRA where we rewrite SUM(1) to 
COUNT(1). Then our existing count optimizations can take over.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Mar 16, 2016 at 8:38 AM, Sudip Mukherjee <smukher...@commvault.com>
wrote:

> I don't see DRILL is transforming the query. Tried with a CSV file.
> Please let me know if I am missing something.
>
> 00-00Screen : rowType = RecordType(INTEGER EXPR$0): rowcount = 1.0,
> cumulative cost = {3.1 rows, 17.1 cpu, 0.0 io, 0.0 network, 0.0 
> memory}, id = 260
> 00-01  Project(EXPR$0=[$0]) : rowType = RecordType(INTEGER EXPR$0):
> rowcount = 1.0, cumulative cost = {3.0 rows, 17.0 cpu, 0.0 io, 0.0 
> network,
> 0.0 memory}, id = 259
> 00-02StreamAgg(group=[{}], EXPR$0=[SUM($0)]) : rowType =
> RecordType(INTEGER EXPR$0): rowcount = 1.0, cumulative cost = {3.0 
> rows,
> 17.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 258
> 00-03  Project($f0=[1]) : rowType = RecordType(INTEGER $f0):
> rowcount = 1.0, cumulative cost = {2.0 rows, 5.0 cpu, 0.0 io, 0.0 
> network,
> 0.0 memory}, id = 257
> 00-04Scan(groupscan=[EasyGroupScan
> [selectionRoot=file:/C:/data/company.csv, numFiles=1, columns=[`*`],
> files=[file:/C:/data/company.csv]]]) : rowType = RecordType(): 
> rowcount = 1.0, cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 256
>
> Thanks,
> Sudip
>
> -Original Message-
> From: Julian Hyde [mailto:jh...@apache.org]
> Sent: 16 March 2016 AM 12:50
> To: dev@drill.apache.org
> Subject: Re: Optimizing SUM(1) query
>
> Is there any reason why Drill cannot transform SUM(1) to COUNT(*) at 
> an early stage (i.e. using a logical optimization rule) so that this 
> optimization does not need to be done for each engine?
>
> > On Mar 15, 2016, at 5:29 AM, Sudip Mukherjee 
> > <smukher...@commvault.com>
> wrote:
> >
> > I was trying to have an Optimizer rule for the solr storage plugin 
> > that
> I'm working on for this query. Trying to use SOLR field stats for this 
> , so that the query is faster..
> > Getting the below exception while transforming project to scan. 
> > Could
> you please advise?
> >
> >
> > 2016-03-15 08:20:35,149 
> > [291801ee-33fc-064d-7aff-18391f15ae0e:foreman]
> DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - Drill Logical :
> > DrillScreenRel: rowcount = 1.0, cumulative cost = {60.1 rows, 320.1 
> > cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 49
> >  DrillProjectRel(EXPR$0=[$0]): rowcount = 1.0, cumulative cost = 
> > {60.0
> rows, 320.0 cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 48
> >DrillAggregateRel(group=[{}], EXPR$0=[SUM($0)]): rowcount = 1.0,
> cumulative cost = {60.0 rows, 320.0 cpu, 0.0 io, 0.0 network, 176.0 
> memory}, id = 46
> >  DrillProjectRel($f0=[1]): rowcount = 20.0, cumulative cost = 
> > {40.0
> rows, 80.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 44
> >DrillScanRel(table=[[solr, ANalert_494]], 
> > groupscan=[SolrGroupScan [SolrScanSpec=SolrScanSpec 
> > [solrCoreName=ANalert_494, solrUrl=http://localhost:2/solr/
> > filter=[], solrDocFetchCount=-1, aggreegation=[]], columns=[`*`]]]):
> > rowcount = 20.0, cumulative cost = {20.0 rows, 0.0 cpu, 0.0 io, 0.0 
> > network, 0.0 memory}, id = 26
> >
> > 2016-03-15 08:20:35,201 
> > [291801ee-33fc-064d-7aff-18391f15ae0e:foreman]
> > DEBUG o.a.drill.exec.work.foreman.Foreman -
> > 291801ee-33fc-064d-7aff-18391f15ae0e: State change requested PENDING
> > --> FAILED
> > org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: index (0) must be less than 
> size
> (0)
> >   at
> org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255)
> [drill-java-exec.jar:1.4.0]
> >   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.8.0_65]
> >   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) [na:1.8.0_65]
> >   at java.lang.Thread.run(Unknown Source) [na:1.8.0_65] Caused by:
> > java.lang.IndexOutOfBoundsException: index (0) must be less than size (0)
> >   at
> com.google.common.base.Prec

RE: Optimizing SUM(1) query

[Sudip Mukherjee]

I don't see DRILL is transforming the query. Tried with a CSV file.  Please let 
me know if I am missing something.

00-00Screen : rowType = RecordType(INTEGER EXPR$0): rowcount = 1.0, 
cumulative cost = {3.1 rows, 17.1 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 
260
00-01  Project(EXPR$0=[$0]) : rowType = RecordType(INTEGER EXPR$0): 
rowcount = 1.0, cumulative cost = {3.0 rows, 17.0 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 259
00-02StreamAgg(group=[{}], EXPR$0=[SUM($0)]) : rowType = 
RecordType(INTEGER EXPR$0): rowcount = 1.0, cumulative cost = {3.0 rows, 17.0 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 258
00-03  Project($f0=[1]) : rowType = RecordType(INTEGER $f0): rowcount = 
1.0, cumulative cost = {2.0 rows, 5.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id 
= 257
00-04Scan(groupscan=[EasyGroupScan 
[selectionRoot=file:/C:/data/company.csv, numFiles=1, columns=[`*`], 
files=[file:/C:/data/company.csv]]]) : rowType = RecordType(): rowcount = 1.0, 
cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 256

Thanks,
Sudip

-Original Message-
From: Julian Hyde [mailto:jh...@apache.org] 
Sent: 16 March 2016 AM 12:50
To: dev@drill.apache.org
Subject: Re: Optimizing SUM(1) query

Is there any reason why Drill cannot transform SUM(1) to COUNT(*) at an early 
stage (i.e. using a logical optimization rule) so that this optimization does 
not need to be done for each engine?

> On Mar 15, 2016, at 5:29 AM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> I was trying to have an Optimizer rule for the solr storage plugin that I'm 
> working on for this query. Trying to use SOLR field stats for this , so that 
> the query is faster..
> Getting the below exception while transforming project to scan. Could you 
> please advise?
> 
> 
> 2016-03-15 08:20:35,149 [291801ee-33fc-064d-7aff-18391f15ae0e:foreman] DEBUG 
> o.a.d.e.p.s.h.DefaultSqlHandler - Drill Logical :
> DrillScreenRel: rowcount = 1.0, cumulative cost = {60.1 rows, 320.1 
> cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 49
>  DrillProjectRel(EXPR$0=[$0]): rowcount = 1.0, cumulative cost = {60.0 rows, 
> 320.0 cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 48
>DrillAggregateRel(group=[{}], EXPR$0=[SUM($0)]): rowcount = 1.0, 
> cumulative cost = {60.0 rows, 320.0 cpu, 0.0 io, 0.0 network, 176.0 memory}, 
> id = 46
>  DrillProjectRel($f0=[1]): rowcount = 20.0, cumulative cost = {40.0 rows, 
> 80.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 44
>DrillScanRel(table=[[solr, ANalert_494]], 
> groupscan=[SolrGroupScan [SolrScanSpec=SolrScanSpec 
> [solrCoreName=ANalert_494, solrUrl=http://localhost:2/solr/ 
> filter=[], solrDocFetchCount=-1, aggreegation=[]], columns=[`*`]]]): 
> rowcount = 20.0, cumulative cost = {20.0 rows, 0.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 26
> 
> 2016-03-15 08:20:35,201 [291801ee-33fc-064d-7aff-18391f15ae0e:foreman] 
> DEBUG o.a.drill.exec.work.foreman.Foreman - 
> 291801ee-33fc-064d-7aff-18391f15ae0e: State change requested PENDING 
> --> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception 
> during fragment initialization: index (0) must be less than size (0)
>   at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
> [drill-java-exec.jar:1.4.0]
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> [na:1.8.0_65]
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> [na:1.8.0_65]
>   at java.lang.Thread.run(Unknown Source) [na:1.8.0_65] Caused by: 
> java.lang.IndexOutOfBoundsException: index (0) must be less than size (0)
>   at 
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
>  ~[com.google.guava-guava.jar:na]
>   at 
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
>  ~[com.google.guava-guava.jar:na]
>   at 
> com.google.common.collect.EmptyImmutableList.get(EmptyImmutableList.java:80) 
> ~[com.google.guava-guava.jar:na]
>   at org.apache.calcite.util.Pair$6.get(Pair.java:335) 
> ~[org.apache.calcite-calcite-core.jar:1.4.0-drill-r10]
>   at 
> org.apache.drill.exec.planner.StarColumnHelper.containsStarColumnInProject(StarColumnHelper.java:60)
>  ~[drill-java-exec.jar:1.4.0]
>   at 
> org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitProject(StarColumnConverter.java:138)
>  ~[drill-java-exec.jar:1.4.0]
>   at 
> org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitProject(StarColumnConverter.java:45)
>  ~[drill-java-exec.jar:1.4.0]
>   at 
> org.apache.drill.exec.planner.physical.ProjectPrel.accept(Project

RE: Optimizing SUM(1) query

[Sudip Mukherjee]

I was trying to have an Optimizer rule for the solr storage plugin that I'm 
working on for this query. Trying to use SOLR field stats for this , so that 
the query is faster..
Getting the below exception while transforming project to scan. Could you 
please advise?


2016-03-15 08:20:35,149 [291801ee-33fc-064d-7aff-18391f15ae0e:foreman] DEBUG 
o.a.d.e.p.s.h.DefaultSqlHandler - Drill Logical :
DrillScreenRel: rowcount = 1.0, cumulative cost = {60.1 rows, 320.1 cpu, 0.0 
io, 0.0 network, 176.0 memory}, id = 49
  DrillProjectRel(EXPR$0=[$0]): rowcount = 1.0, cumulative cost = {60.0 rows, 
320.0 cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 48
DrillAggregateRel(group=[{}], EXPR$0=[SUM($0)]): rowcount = 1.0, cumulative 
cost = {60.0 rows, 320.0 cpu, 0.0 io, 0.0 network, 176.0 memory}, id = 46
  DrillProjectRel($f0=[1]): rowcount = 20.0, cumulative cost = {40.0 rows, 
80.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 44
DrillScanRel(table=[[solr, ANalert_494]], groupscan=[SolrGroupScan 
[SolrScanSpec=SolrScanSpec [solrCoreName=ANalert_494, 
solrUrl=http://localhost:2/solr/ filter=[], solrDocFetchCount=-1, 
aggreegation=[]], columns=[`*`]]]): rowcount = 20.0, cumulative cost = {20.0 
rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 26

2016-03-15 08:20:35,201 [291801ee-33fc-064d-7aff-18391f15ae0e:foreman] DEBUG 
o.a.drill.exec.work.foreman.Foreman - 291801ee-33fc-064d-7aff-18391f15ae0e: 
State change requested PENDING --> FAILED
org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception 
during fragment initialization: index (0) must be less than size (0)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
[drill-java-exec.jar:1.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
[na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
[na:1.8.0_65]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_65]
Caused by: java.lang.IndexOutOfBoundsException: index (0) must be less than 
size (0)
at 
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305) 
~[com.google.guava-guava.jar:na]
at 
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284) 
~[com.google.guava-guava.jar:na]
at 
com.google.common.collect.EmptyImmutableList.get(EmptyImmutableList.java:80) 
~[com.google.guava-guava.jar:na]
at org.apache.calcite.util.Pair$6.get(Pair.java:335) 
~[org.apache.calcite-calcite-core.jar:1.4.0-drill-r10]
at 
org.apache.drill.exec.planner.StarColumnHelper.containsStarColumnInProject(StarColumnHelper.java:60)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitProject(StarColumnConverter.java:138)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitProject(StarColumnConverter.java:45)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.ProjectPrel.accept(ProjectPrel.java:69) 
~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitScreen(StarColumnConverter.java:77)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.visitScreen(StarColumnConverter.java:45)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.ScreenPrel.accept(ScreenPrel.java:64) 
~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.physical.visitor.StarColumnConverter.insertRenameProject(StarColumnConverter.java:72)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:326)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:175)
 ~[drill-java-exec.jar:1.4.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:197)
 ~[drill-java-exec.jar:1.4.0]


Thanks,
Sudip

-Original Message-
From: Sudip Mukherjee [mailto:smukher...@commvault.com] 
Sent: 07 March 2016 PM 04:45
To: dev@drill.apache.org
Subject: RE: Optimizing SUM(1) query

Hi,

Extremely sorry for the delayed response!

Basically , I am experimenting an Apache Solr plugin for Drill and tried to use 
Tableau. On intial selection of a "Table" , I see these queries from drill 
webui profile

SELECT * FROM (SELECT * FROM `solr`.`CSVStringData`) T LIMIT 0

SELECT * FROM (SELECT SUM(1) AS `COL` FROM `solr`.`CSVStringData` 
`CSVStringData` HAVING COUNT(1)>0) T LIMIT 0 

I have a dataset of 2,297,451,8 and when I try access that,the tableau screen 
in kind of stuck in "Processing Request" for 5 minutes.

Thanks,
Sudip

-Original Message-
From: Andries Engelbrecht [mailto:aengelbre...@maprtech.com]
Sent: 20 February 2016 AM 03:42
To: dev@dr

RE: Optimizing SUM(1) query

[Sudip Mukherjee]

Hi,

Extremely sorry for the delayed response!

Basically , I am experimenting an Apache Solr plugin for Drill and tried to use 
Tableau. On intial selection of a "Table" , I see these queries from drill 
webui profile

SELECT * FROM (SELECT * FROM `solr`.`CSVStringData`) T LIMIT 0

SELECT * FROM (SELECT SUM(1) AS `COL` FROM `solr`.`CSVStringData` 
`CSVStringData` HAVING COUNT(1)>0) T LIMIT 0 

I have a dataset of 2,297,451,8 and when I try access that,the tableau screen 
in kind of stuck in "Processing Request" for 5 minutes.

Thanks,
Sudip

-Original Message-
From: Andries Engelbrecht [mailto:aengelbre...@maprtech.com] 
Sent: 20 February 2016 AM 03:42
To: dev@drill.apache.org
Subject: Re: Optimizing SUM(1) query

What are you trying to do in Tableau that generates these queries?

Do you have the actual full query that is being generated?

Tableau has a number of customization features, wonder if it is possible to 
optimize those settings to avoid this issue.

--Andries


> On Feb 19, 2016, at 10:16 AM, Sudip Mukherjee <smukher...@commvault.com> 
> wrote:
> 
> Hi,
> 
> Have anyone tried optimizing SUM(1) query in drill? Or is it implemented?
> Getting these query while using Tableau. Mostly probably it is trying to 
> figure out NUMBER_OF_RECORDS.
> 
> Thanks,
> Sudip
> 
> 
> 
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **





***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Optimizing SUM(1) query

[Sudip Mukherjee]

Hi,

Have anyone tried optimizing SUM(1) query in drill? Or is it implemented?
Getting these query while using Tableau. Mostly probably it is trying to figure 
out NUMBER_OF_RECORDS.

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: TIMESTAMPADD sql jdbc function execution in drill

[Sudip Mukherjee]

Thanks a lot for the information.

Regards,
Sudip

From: Arina Yelchiyeva [mailto:arina.yelchiy...@gmail.com]
Sent: 08 February 2016 PM 01:13
To: dev@drill.apache.org; Sudip Mukherjee
Subject: Re: TIMESTAMPADD sql jdbc function execution in drill

Hi Sudip,

timestampadd / timestampdiff functions are not currently implemented in Drill 
but they are in development.
See Jira https://issues.apache.org/jira/browse/DRILL-3610


Kind regards
Arina
On Mon, Feb 8, 2016 at 9:29 AM Sudip Mukherjee 
<smukher...@commvault.com<mailto:smukher...@commvault.com>> wrote:
Thanks Khurram.

I am not sure if I am looking at the correct place. Seeing it in calcite,
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/SqlJdbcFunctionCall.java

I'll try to check further on the date-time functions.

Thanks,
Sudip

-Original Message-
From: Khurram Faraaz [mailto:kfar...@maprtech.com<mailto:kfar...@maprtech.com>]
Sent: 08 February 2016 PM 12:14
To: dev@drill.apache.org<mailto:dev@drill.apache.org>
Subject: Re: TIMESTAMPADD sql jdbc function execution in drill

I didn't find it on the documented date time functions here 
https://drill.apache.org/docs/date-time-functions-and-arithmetic/

On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee 
<smukher...@commvault.com<mailto:smukher...@commvault.com>>
wrote:

> Hi,
> Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I
> am getting an error while running the below query
>
> SELECT CAST(EXTRACT(YEAR FROM CAST({fn
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> '1970-01-01'})} AS DATE)) AS INTEGER) AS
> `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> FROM `dfs.data`.`MyFileShare_2214` `myshare` GROUP BY
> CAST(EXTRACT(YEAR FROM CAST({fn
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d
> '1970-01-01'})} AS DATE)) AS INTEGER)
>
>
> VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column
> 'SQL_TSI_YEAR' not found in any table
>
> Thanks,
> Sudip
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply
> email and delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: TIMESTAMPADD sql jdbc function execution in drill

[Sudip Mukherjee]

Thanks Khurram.

I am not sure if I am looking at the correct place. Seeing it in calcite,
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/sql/SqlJdbcFunctionCall.java
 

I'll try to check further on the date-time functions.

Thanks,
Sudip

-Original Message-
From: Khurram Faraaz [mailto:kfar...@maprtech.com] 
Sent: 08 February 2016 PM 12:14
To: dev@drill.apache.org
Subject: Re: TIMESTAMPADD sql jdbc function execution in drill

I didn't find it on the documented date time functions here 
https://drill.apache.org/docs/date-time-functions-and-arithmetic/

On Mon, Feb 8, 2016 at 12:09 PM, Sudip Mukherjee <smukher...@commvault.com>
wrote:

> Hi,
> Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I 
> am getting an error while running the below query
>
> SELECT CAST(EXTRACT(YEAR FROM CAST({fn 
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d 
> '1970-01-01'})} AS DATE)) AS INTEGER) AS 
> `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
> FROM `dfs.data`.`MyFileShare_2214` `myshare` GROUP BY 
> CAST(EXTRACT(YEAR FROM CAST({fn 
> TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d 
> '1970-01-01'})} AS DATE)) AS INTEGER)
>
>
> VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column 
> 'SQL_TSI_YEAR' not found in any table
>
> Thanks,
> Sudip
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

TIMESTAMPADD sql jdbc function execution in drill

[Sudip Mukherjee]

Hi,
Are  TIMESTAMPADD / TIMESTAMPDIFF functions supported through drill? I am 
getting an error while running the below query

SELECT CAST(EXTRACT(YEAR FROM CAST({fn 
TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d '1970-01-01'})} 
AS DATE)) AS INTEGER) AS `yr_Calculation_IGIAGIIHHFEBCCIFEE_ok`
FROM `dfs.data`.`MyFileShare_2214` `myshare`
GROUP BY CAST(EXTRACT(YEAR FROM CAST({fn 
TIMESTAMPADD(SQL_TSI_YEAR,CAST(`myshare`.`mtm` AS INTEGER),{d '1970-01-01'})} 
AS DATE)) AS INTEGER)


VALIDATION ERROR: From line 3, column 55 to line 3, column 66: Column 
'SQL_TSI_YEAR' not found in any table

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: [jira] [Created] (DRILL-4294) InfoSchema is not returning column metadata if the query contains escape character

[Sudip Mukherjee]

Hi,

Could you please advise if this is the correct way to fix the problem. Wanted 
to use Power BI with Drill and got into the below issue.
https://github.com/apache/drill/commit/19886c11352cfa45bdb2da885a86e6500986e1e3 

Thanks,
Sudip

-Original Message-
From: Sudip Mukherjee (JIRA) [mailto:j...@apache.org] 
Sent: 21 January 2016 PM 04:16
To: dev@drill.apache.org
Subject: [jira] [Created] (DRILL-4294) InfoSchema is not returning column 
metadata if the query contains escape character

Sudip Mukherjee created DRILL-4294:
--

 Summary: InfoSchema is not returning column metadata if the query 
contains escape character
 Key: DRILL-4294
 URL: https://issues.apache.org/jira/browse/DRILL-4294
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: Future, 1.4.0
Reporter: Sudip Mukherjee


Below Query returns noting where escape character is "\\":

SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, CASE DATA_TYPE 
WHEN 'ANY', 'MAP' THEN 'CHARACTER VARYING' ELSE DATA_TYPE END as DATA_TYPE, 
CASE DATA_TYPE WHEN 'ANY', 'MAP' THEN 'CHARACTER VARYING' ELSE DATA_TYPE END as 
DATA_TYPE_NAME, CASE DATA_TYPE WHEN 'BINARY' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'CHARACTER VARYING' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'CHARACTER' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'DATE' THEN DATETIME_PRECISION
 WHEN 'INTERVAL' THEN INTERVAL_PRECISION WHEN 'VARCHAR' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'VARBINARY' THEN 
CHARACTER_MAXIMUM_LENGTH ELSE NUMERIC_PRECISION END as 
COLUMN_SIZE, NUMERIC_SCALE as DECIMAL_DIGITS, CASE DATA_TYPE
 WHEN 'CHARACTER VARYING' THEN CHARACTER_MAXIMUM_LENGTH 
WHEN 'CHARACTER' THEN CHARACTER_MAXIMUM_LENGTH WHEN 'VARCHAR' THEN 
CHARACTER_MAXIMUM_LENGTH ELSE 9 END as BUFFER_LENGTH,   
  NUMERIC_PRECISION_RADIX, IS_NULLABLE as NULLABLE, 12 as 
REMARKS, 13 as COLUMN_DEF, 14 as SQL_DATA_TYPE, 15 as 
SQL_DATETIME_SUB, CHARACTER_MAXIMUM_LENGTH as CHAR_OCTET_LENGTH,
 ORDINAL_POSITION, IS_NULLABLE, 18 as USER_DATA_TYPE FROM 
INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG LIKE 'DRILL' ESCAPE '\' AND 
TABLE_SCHEMA LIKE 'dfs.tmp' ESCAPE '\' AND TABLE_NAME LIKE 'app\_client\_view' 
ESCAPE '\' ORDER BY TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, ORDINAL_POSITION, 
COLUMN_NAME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)




***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: [GitHub] drill pull request: Apache Solr Storage plugin

[Sudip Mukherjee]

Hi Stefan,

I've the initial code to query solr server with some of the filter, limit 
push-downs. Used solrj to query on solr.

Thanks,
Sudip

-Original Message-
From: Stefán Baxter [mailto:ste...@activitystream.com] 
Sent: 22 January 2016 PM 02:56
To: dev
Subject: Re: [GitHub] drill pull request: Apache Solr Storage plugin

Hi sudipmukherjee,

This is great. Can you please share with me the status of this initiative?

Regards,
  -Stefán


On Fri, Jan 22, 2016 at 9:16 AM, sudipmukherjee  wrote:

> Github user sudipmukherjee closed the pull request at:
>
> https://github.com/apache/drill/pull/100
>
>
> ---
> If your project is set up for it, you can reply to this email and have 
> your reply appear on GitHub as well. If your project does not have 
> this feature enabled and wishes so, or if the feature is enabled but 
> not working, please contact infrastructure at 
> infrastruct...@apache.org or file a JIRA ticket with INFRA.
> ---
>



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[jira] [Created] (DRILL-4294) InfoSchema is not returning column metadata if the query contains escape character

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-4294:
--

 Summary: InfoSchema is not returning column metadata if the query 
contains escape character
 Key: DRILL-4294
 URL: https://issues.apache.org/jira/browse/DRILL-4294
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: Future, 1.4.0
Reporter: Sudip Mukherjee


Below Query returns noting where escape character is "\\":

SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, CASE DATA_TYPE 
WHEN 'ANY', 'MAP' THEN 'CHARACTER VARYING' ELSE DATA_TYPE END as DATA_TYPE, 
CASE DATA_TYPE WHEN 'ANY', 'MAP' THEN 'CHARACTER VARYING' ELSE DATA_TYPE END as 
DATA_TYPE_NAME, CASE DATA_TYPE WHEN 'BINARY' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'CHARACTER VARYING' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'CHARACTER' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'DATE' THEN DATETIME_PRECISION
 WHEN 'INTERVAL' THEN INTERVAL_PRECISION WHEN 'VARCHAR' THEN 
CHARACTER_MAXIMUM_LENGTH WHEN 'VARBINARY' THEN 
CHARACTER_MAXIMUM_LENGTH ELSE NUMERIC_PRECISION END as 
COLUMN_SIZE, NUMERIC_SCALE as DECIMAL_DIGITS, CASE DATA_TYPE
 WHEN 'CHARACTER VARYING' THEN CHARACTER_MAXIMUM_LENGTH 
WHEN 'CHARACTER' THEN CHARACTER_MAXIMUM_LENGTH WHEN 'VARCHAR' THEN 
CHARACTER_MAXIMUM_LENGTH ELSE 9 END as BUFFER_LENGTH,   
  NUMERIC_PRECISION_RADIX, IS_NULLABLE as NULLABLE, 12 as 
REMARKS, 13 as COLUMN_DEF, 14 as SQL_DATA_TYPE, 15 as 
SQL_DATETIME_SUB, CHARACTER_MAXIMUM_LENGTH as CHAR_OCTET_LENGTH,
 ORDINAL_POSITION, IS_NULLABLE, 18 as USER_DATA_TYPE FROM 
INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG LIKE 'DRILL' ESCAPE '\' AND 
TABLE_SCHEMA LIKE 'dfs.tmp' ESCAPE '\' AND TABLE_NAME LIKE 'app\_client\_view' 
ESCAPE '\' ORDER BY TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, ORDINAL_POSITION, 
COLUMN_NAME



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Does DRILL support connectivity with Microsoft Power BI?

[Sudip Mukherjee]

Hi,
Does drill support connectivity with Microsoft Power BI? Couldn't find any 
documentation stating that it is supported.


Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[jira] [Created] (DRILL-4172) Need stop , port as startup parameters in case drill is installed as windows service

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-4172:
--

 Summary: Need stop , port as startup parameters in case drill is 
installed as windows service
 Key: DRILL-4172
 URL: https://issues.apache.org/jira/browse/DRILL-4172
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.3.0
 Environment: Windows
Reporter: Sudip Mukherjee
 Fix For: Future


I am trying to install Drill using procrun in windows server for persistent 
service other than running batch file.
I was in need of start , stop and port parameter for the flexibility to 
start/stop the service from windows services.msc

Does it make sense to introduce these as startup params ( optional )?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Can Drill distribution be deployed in an existing jetty/tomcat server

[Sudip Mukherjee]

Hi,

Could you please advise if there is a way I can use one of my existing 
jetty/tomcat server and deploy drill as a service?

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Drill Custom Authentication

[Sudip Mukherjee]

Hi,
I am trying to implement a custom authenticator so that I can validate against 
some of existing service that we use. Wanted to know how can I apply the 
security rules i.e. user1 has access to these many tables etc. going further 
ahead with using BI Tools.

https://drill.apache.org/docs/configuring-user-authentication/

Please let me know your thoughts.

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: Channel closed exception

[Sudip Mukherjee]

Ohh..Sorry about that,
Could you please try again? Or I'll put it in github
https://drive.google.com/drive/folders/0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ

Thanks,
Sudip

-Original Message-
From: Sudheesh Katkam [mailto:skat...@maprtech.com] 
Sent: 02 October 2015 AM 02:04
To: dev@drill.apache.org
Subject: Re: Channel closed exception

I cannot open the shared documents (“Whoops! There was a problem previewing 
this document.”) nor can I download them ("We're sorry, but you do not have 
access to this page.”). The folder link in this thread is also broken (“The 
open folder can no longer be viewed”).

- Sudheesh

> On Oct 1, 2015, at 12:43 PM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> I uploaded to google drive. Should it be okay?
> 
> https://drive.google.com/drive/folders/0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ
> 
> Thanks,
> Sudip
> 
> -Original Message-
> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
> Sent: 01 October 2015 AM 10:01
> To: dev@drill.apache.org
> Subject: Re: Channel closed exception
> 
> Ah there, I assumed you were not on Windows. We do not have the drillbit.out 
> file for Windows (I think).
> 
> Anyway, the error messages you see are symptoms of the real issue, which is 
> most likely an “out of memory” scenario. I think that the drillbit did not 
> crash either (in which case you might see a JVM core dump). Instead the 
> drillbit gracefully shutdown unable to handle the error. Could you share the 
> query (if possible), data set details, and configuration?
> 
> It would also help if you put the entire log files somewhere (Github gist or 
> Dropbox) and shared the links.
> 
> - Sudheesh
> 
>> On Sep 30, 2015, at 9:10 PM, Sudip Mukherjee <smukher...@commvault.com> 
>> wrote:
>> 
>> Yes. I was running drill in embedded mode to try it out with  3.9 
>> lakhs records , near about 50+ columns.[ sqlline.bat -u 
>> "jdbc:drill:zk=local" ] I see sqlline.log and sqlline_queries.log under log 
>> folder. Did I miss some steps?
>> 
>> But was able run queries from the command prompt after that
>> 
>> Thanks,
>> Sudip
>> 
>> -Original Message-
>> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
>> Sent: 01 October 2015 AM 09:24
>> To: dev@drill.apache.org
>> Subject: Re: Channel closed exception
>> 
>> Are you running drill in embedded mode? The directory containing the 
>> drillbit.log file contains the drillbit.out file too. There might be a JVM 
>> core dump in the /tmp directory.
>> 
>> Usually "Received shutdown request” message means that’s the drillbit ran 
>> out of memory. Given that you mentioned the query is on a large document 
>> set, this scenario is likely.
>> 
>> - Sudheesh
>> 
>>> On Sep 30, 2015, at 8:46 PM, Sudip Mukherjee <smukher...@commvault.com> 
>>> wrote:
>>> 
>>> Hi,
>>> This is the excerpt of exception in sqlline.log. Could you please refer me 
>>> to the location of drillbit.out file?
>>> 
>>> 
>>> 2015-09-30 22:23:36,286 [BitServer-4] WARN  
>>> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 
>>> 6 took longer than 500ms.  Actual duration was 16047ms.
>>> 2015-09-30 22:24:13,581
>>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>>> o.a.d.e.w.fragment.FragmentExecutor -
>>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State change requested 
>>> CANCELLATION_REQUESTED --> FINISHED
>>> 2015-09-30 22:24:13,581
>>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>>> o.a.d.e.w.f.FragmentStatusReporter -
>>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State to report: CANCELLED
>>> 2015-09-30 22:24:13,581 [BitServer-4] INFO 
>>> o.a.drill.exec.work.foreman.Foreman - State change requested.
>>> CANCELLATION_REQUESTED --> COMPLETED
>>> 2015-09-30 22:24:13,581 [BitServer-4] INFO  
>>> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
>>> 2015-09-30 22:24:13,597 [UserServer-1] INFO  
>>> o.a.drill.exec.work.foreman.Foreman - Failure while trying communicate 
>>> query result to initiating client. This would happen if a client is 
>>> disconnected before response notice can be sent.
>>> org.apache.drill.exec.rpc.ChannelClosedException: null
>>> at 
>>> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:89)
>>>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>>> at 
>>> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComp

RE: Channel closed exception

[Sudip Mukherjee]

I uploaded to google drive. Should it be okay?

https://drive.google.com/drive/folders/0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ

Thanks,
Sudip

-Original Message-
From: Sudheesh Katkam [mailto:skat...@maprtech.com] 
Sent: 01 October 2015 AM 10:01
To: dev@drill.apache.org
Subject: Re: Channel closed exception

Ah there, I assumed you were not on Windows. We do not have the drillbit.out 
file for Windows (I think).

Anyway, the error messages you see are symptoms of the real issue, which is 
most likely an “out of memory” scenario. I think that the drillbit did not 
crash either (in which case you might see a JVM core dump). Instead the 
drillbit gracefully shutdown unable to handle the error. Could you share the 
query (if possible), data set details, and configuration?

It would also help if you put the entire log files somewhere (Github gist or 
Dropbox) and shared the links.

- Sudheesh

> On Sep 30, 2015, at 9:10 PM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> Yes. I was running drill in embedded mode to try it out with  3.9 
> lakhs records , near about 50+ columns.[ sqlline.bat -u "jdbc:drill:zk=local" 
> ] I see sqlline.log and sqlline_queries.log under log folder. Did I miss some 
> steps?
> 
> But was able run queries from the command prompt after that
> 
> Thanks,
> Sudip
> 
> -Original Message-
> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
> Sent: 01 October 2015 AM 09:24
> To: dev@drill.apache.org
> Subject: Re: Channel closed exception
> 
> Are you running drill in embedded mode? The directory containing the 
> drillbit.log file contains the drillbit.out file too. There might be a JVM 
> core dump in the /tmp directory.
> 
> Usually "Received shutdown request” message means that’s the drillbit ran out 
> of memory. Given that you mentioned the query is on a large document set, 
> this scenario is likely.
> 
> - Sudheesh
> 
>> On Sep 30, 2015, at 8:46 PM, Sudip Mukherjee <smukher...@commvault.com> 
>> wrote:
>> 
>> Hi,
>> This is the excerpt of exception in sqlline.log. Could you please refer me 
>> to the location of drillbit.out file?
>> 
>> 
>> 2015-09-30 22:23:36,286 [BitServer-4] WARN  
>> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
>> took longer than 500ms.  Actual duration was 16047ms.
>> 2015-09-30 22:24:13,581
>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>> o.a.d.e.w.fragment.FragmentExecutor -
>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State change requested 
>> CANCELLATION_REQUESTED --> FINISHED
>> 2015-09-30 22:24:13,581
>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>> o.a.d.e.w.f.FragmentStatusReporter -
>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State to report: CANCELLED
>> 2015-09-30 22:24:13,581 [BitServer-4] INFO 
>> o.a.drill.exec.work.foreman.Foreman - State change requested.
>> CANCELLATION_REQUESTED --> COMPLETED
>> 2015-09-30 22:24:13,581 [BitServer-4] INFO  
>> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
>> 2015-09-30 22:24:13,597 [UserServer-1] INFO  
>> o.a.drill.exec.work.foreman.Foreman - Failure while trying communicate query 
>> result to initiating client. This would happen if a client is disconnected 
>> before response notice can be sent.
>> org.apache.drill.exec.rpc.ChannelClosedException: null
>>  at 
>> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:89)
>>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>>  at 
>> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:67)
>>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>>  at 
>> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>>  at 
>> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>>  at 
>> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>>  at
>> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.jav
>> a
>> :424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
>> 
>> Thanks,
>> Sudip
>> 
>> -Original Message-
>> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
>> Sent: 01 October 2015 AM 09:09
>> To: dev@drill.apache.org
>> Subject: Re: Channel closed exception
>> 
>> Can you check if there are any error messages in the drillbit.out f

Apache_Drill_Exception - Invitation to view

[Sudip Mukherjee (via Google Drive)]

I've shared an item with you:

Apache_Drill_Exception
https://drive.google.com/folderview?id=0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ=sharing=CPOnotUP=560d87a6

It's not an attachment -- it's stored online. To open this item, just click  
the link above.


Apache drill exception while getting data to tableau..

logs and data source info shared.

RE: Channel closed exception

[Sudip Mukherjee]

Uploaded the entire log.

Thanks,
Sudip

-Original Message-
From: Sudheesh Katkam [mailto:skat...@maprtech.com] 
Sent: 02 October 2015 AM 02:40
To: dev@drill.apache.org
Subject: Re: Channel closed exception

Could you share the entire log file?

- Sudheesh

> On Oct 1, 2015, at 1:57 PM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> Ohh..Sorry about that,
> Could you please try again? Or I'll put it in github 
> https://drive.google.com/drive/folders/0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ
> 
> Thanks,
> Sudip
> 
> -Original Message-
> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
> Sent: 02 October 2015 AM 02:04
> To: dev@drill.apache.org
> Subject: Re: Channel closed exception
> 
> I cannot open the shared documents (“Whoops! There was a problem previewing 
> this document.”) nor can I download them ("We're sorry, but you do not have 
> access to this page.”). The folder link in this thread is also broken (“The 
> open folder can no longer be viewed”).
> 
> - Sudheesh
> 
>> On Oct 1, 2015, at 12:43 PM, Sudip Mukherjee <smukher...@commvault.com> 
>> wrote:
>> 
>> I uploaded to google drive. Should it be okay?
>> 
>> https://drive.google.com/drive/folders/0ByRlB6iLvhRvSm9UQWJ4b2ZjVDQ
>> 
>> Thanks,
>> Sudip
>> 
>> -Original Message-
>> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
>> Sent: 01 October 2015 AM 10:01
>> To: dev@drill.apache.org
>> Subject: Re: Channel closed exception
>> 
>> Ah there, I assumed you were not on Windows. We do not have the drillbit.out 
>> file for Windows (I think).
>> 
>> Anyway, the error messages you see are symptoms of the real issue, which is 
>> most likely an “out of memory” scenario. I think that the drillbit did not 
>> crash either (in which case you might see a JVM core dump). Instead the 
>> drillbit gracefully shutdown unable to handle the error. Could you share the 
>> query (if possible), data set details, and configuration?
>> 
>> It would also help if you put the entire log files somewhere (Github gist or 
>> Dropbox) and shared the links.
>> 
>> - Sudheesh
>> 
>>> On Sep 30, 2015, at 9:10 PM, Sudip Mukherjee <smukher...@commvault.com> 
>>> wrote:
>>> 
>>> Yes. I was running drill in embedded mode to try it out with  3.9 
>>> lakhs records , near about 50+ columns.[ sqlline.bat -u 
>>> "jdbc:drill:zk=local" ] I see sqlline.log and sqlline_queries.log under log 
>>> folder. Did I miss some steps?
>>> 
>>> But was able run queries from the command prompt after that
>>> 
>>> Thanks,
>>> Sudip
>>> 
>>> -Original Message-
>>> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
>>> Sent: 01 October 2015 AM 09:24
>>> To: dev@drill.apache.org
>>> Subject: Re: Channel closed exception
>>> 
>>> Are you running drill in embedded mode? The directory containing the 
>>> drillbit.log file contains the drillbit.out file too. There might be a JVM 
>>> core dump in the /tmp directory.
>>> 
>>> Usually "Received shutdown request” message means that’s the drillbit ran 
>>> out of memory. Given that you mentioned the query is on a large document 
>>> set, this scenario is likely.
>>> 
>>> - Sudheesh
>>> 
>>>> On Sep 30, 2015, at 8:46 PM, Sudip Mukherjee <smukher...@commvault.com> 
>>>> wrote:
>>>> 
>>>> Hi,
>>>> This is the excerpt of exception in sqlline.log. Could you please refer me 
>>>> to the location of drillbit.out file?
>>>> 
>>>> 
>>>> 2015-09-30 22:23:36,286 [BitServer-4] WARN  
>>>> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 
>>>> 6 took longer than 500ms.  Actual duration was 16047ms.
>>>> 2015-09-30 22:24:13,581
>>>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>>>> o.a.d.e.w.fragment.FragmentExecutor -
>>>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State change requested 
>>>> CANCELLATION_REQUESTED --> FINISHED
>>>> 2015-09-30 22:24:13,581
>>>> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO 
>>>> o.a.d.e.w.f.FragmentStatusReporter -
>>>> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State to report: 
>>>> CANCELLED
>>>> 2015-09-30 22:24:13,581 [BitServer-4] INFO 
>>>> o.a.drill.exec.work.foreman.Foreman - State change req

RE: Channel closed exception

[Sudip Mukherjee]

Hi,
This is the excerpt of exception in sqlline.log. Could you please refer me to 
the location of drillbit.out file?


2015-09-30 22:23:36,286 [BitServer-4] WARN  
o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
took longer than 500ms.  Actual duration was 16047ms.
2015-09-30 22:24:13,581 [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: 
State change requested CANCELLATION_REQUESTED --> FINISHED
2015-09-30 22:24:13,581 [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: 
State to report: CANCELLED
2015-09-30 22:24:13,581 [BitServer-4] INFO  o.a.drill.exec.work.foreman.Foreman 
- State change requested.  CANCELLATION_REQUESTED --> COMPLETED
2015-09-30 22:24:13,581 [BitServer-4] INFO  o.a.drill.exec.work.foreman.Foreman 
- foreman cleaning up.
2015-09-30 22:24:13,597 [UserServer-1] INFO  
o.a.drill.exec.work.foreman.Foreman - Failure while trying communicate query 
result to initiating client. This would happen if a client is disconnected 
before response notice can be sent.
org.apache.drill.exec.rpc.ChannelClosedException: null
at 
org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:89)
 [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:67)
 [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) 
[netty-common-4.0.27.Final.jar:4.0.27.Final]

Thanks,
Sudip

-Original Message-
From: Sudheesh Katkam [mailto:skat...@maprtech.com] 
Sent: 01 October 2015 AM 09:09
To: dev@drill.apache.org
Subject: Re: Channel closed exception

Can you check if there are any error messages in the drillbit.out file?

- Sudheesh

> On Sep 30, 2015, at 8:35 PM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> Hi Devs,
> 
> Getting into a channel closed exception when I tried to connect drill to 
> tableau and query a large document set. Could it be because tableau dropped 
> the connection due to datasize. Any thoughts?
> 
> 2015-09-30 21:13:43,716 
> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2015-09-30 21:13:43,716 
> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State to report: RUNNING
> 2015-09-30 21:14:36,875 
> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State change requested 
> RUNNING --> FINISHED
> 2015-09-30 21:14:36,876 
> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State to report: FINISHED
> 2015-09-30 21:14:36,877 [BitServer-4] INFO  
> o.a.drill.exec.work.foreman.Foreman - State change requested.  RUNNING 
> --> COMPLETED
> 2015-09-30 21:14:36,877 [BitServer-4] INFO  
> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
> 2015-09-30 21:33:09,745 [Client-1] INFO  o.a.drill.exec.rpc.user.UserClient - 
> Channel closed /172.19.77.101:57162 <--> 
> DEVSTAGING2.testlab.commvault.com/172.19.77.101:31010.
> 2015-09-30 21:33:12,859 [BitServer-3] INFO  
> o.a.d.exec.rpc.control.ControlClient - Channel closed /172.19.77.101:57207 
> <--> DEVSTAGING2.testlab.commvault.com/172.19.77.101:31011.
> 2015-09-30 21:33:16,077 [main] INFO  o.apache.drill.exec.server.Drillbit - 
> Shutdown completed (6331 ms).
> 2015-09-30 21:33:16,102 [Drillbit-ShutdownHook#0] INFO  
> o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 
> Thanks,
> Sudip
> 
> 
> 
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the mess

Channel closed exception

[Sudip Mukherjee]

Hi Devs,

Getting into a channel closed exception when I tried to connect drill to 
tableau and query a large document set. Could it be because tableau dropped the 
connection due to datasize. Any thoughts?

2015-09-30 21:13:43,716 [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 29f3777a-d407-295a-d45e-243fb70efc07:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2015-09-30 21:13:43,716 [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 29f3777a-d407-295a-d45e-243fb70efc07:0:0: 
State to report: RUNNING
2015-09-30 21:14:36,875 [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 29f3777a-d407-295a-d45e-243fb70efc07:0:0: 
State change requested RUNNING --> FINISHED
2015-09-30 21:14:36,876 [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 29f3777a-d407-295a-d45e-243fb70efc07:0:0: 
State to report: FINISHED
2015-09-30 21:14:36,877 [BitServer-4] INFO  o.a.drill.exec.work.foreman.Foreman 
- State change requested.  RUNNING --> COMPLETED
2015-09-30 21:14:36,877 [BitServer-4] INFO  o.a.drill.exec.work.foreman.Foreman 
- foreman cleaning up.
2015-09-30 21:33:09,745 [Client-1] INFO  o.a.drill.exec.rpc.user.UserClient - 
Channel closed /172.19.77.101:57162 <--> 
DEVSTAGING2.testlab.commvault.com/172.19.77.101:31010.
2015-09-30 21:33:12,859 [BitServer-3] INFO  
o.a.d.exec.rpc.control.ControlClient - Channel closed /172.19.77.101:57207 <--> 
DEVSTAGING2.testlab.commvault.com/172.19.77.101:31011.
2015-09-30 21:33:16,077 [main] INFO  o.apache.drill.exec.server.Drillbit - 
Shutdown completed (6331 ms).
2015-09-30 21:33:16,102 [Drillbit-ShutdownHook#0] INFO  
o.apache.drill.exec.server.Drillbit - Received shutdown request.

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: Channel closed exception

[Sudip Mukherjee]

Yes. I was running drill in embedded mode to try it out with  3.9 lakhs records 
, near about 50+ columns.[ sqlline.bat -u "jdbc:drill:zk=local" ]
I see sqlline.log and sqlline_queries.log under log folder. Did I miss some 
steps?

But was able run queries from the command prompt after that

Thanks,
Sudip

-Original Message-
From: Sudheesh Katkam [mailto:skat...@maprtech.com] 
Sent: 01 October 2015 AM 09:24
To: dev@drill.apache.org
Subject: Re: Channel closed exception

Are you running drill in embedded mode? The directory containing the 
drillbit.log file contains the drillbit.out file too. There might be a JVM core 
dump in the /tmp directory.

Usually "Received shutdown request” message means that’s the drillbit ran out 
of memory. Given that you mentioned the query is on a large document set, this 
scenario is likely.

- Sudheesh

> On Sep 30, 2015, at 8:46 PM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> Hi,
> This is the excerpt of exception in sqlline.log. Could you please refer me to 
> the location of drillbit.out file?
> 
> 
> 2015-09-30 22:23:36,286 [BitServer-4] WARN  
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type 6 
> took longer than 500ms.  Actual duration was 16047ms.
> 2015-09-30 22:24:13,581 
> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State change requested 
> CANCELLATION_REQUESTED --> FINISHED
> 2015-09-30 22:24:13,581 
> [29f36725-9de3-b370-7f38-50ebbe677bf7:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 29f36725-9de3-b370-7f38-50ebbe677bf7:0:0: State to report: CANCELLED
> 2015-09-30 22:24:13,581 [BitServer-4] INFO  
> o.a.drill.exec.work.foreman.Foreman - State change requested.  
> CANCELLATION_REQUESTED --> COMPLETED
> 2015-09-30 22:24:13,581 [BitServer-4] INFO  
> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
> 2015-09-30 22:24:13,597 [UserServer-1] INFO  
> o.a.drill.exec.work.foreman.Foreman - Failure while trying communicate query 
> result to initiating client. This would happen if a client is disconnected 
> before response notice can be sent.
> org.apache.drill.exec.rpc.ChannelClosedException: null
>   at 
> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:89)
>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:67)
>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>   at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>   at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>   at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
>   at 
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java
> :424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
> 
> Thanks,
> Sudip
> 
> -Original Message-
> From: Sudheesh Katkam [mailto:skat...@maprtech.com]
> Sent: 01 October 2015 AM 09:09
> To: dev@drill.apache.org
> Subject: Re: Channel closed exception
> 
> Can you check if there are any error messages in the drillbit.out file?
> 
> - Sudheesh
> 
>> On Sep 30, 2015, at 8:35 PM, Sudip Mukherjee <smukher...@commvault.com> 
>> wrote:
>> 
>> Hi Devs,
>> 
>> Getting into a channel closed exception when I tried to connect drill to 
>> tableau and query a large document set. Could it be because tableau dropped 
>> the connection due to datasize. Any thoughts?
>> 
>> 2015-09-30 21:13:43,716
>> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO 
>> o.a.d.e.w.fragment.FragmentExecutor -
>> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State change requested 
>> AWAITING_ALLOCATION --> RUNNING
>> 2015-09-30 21:13:43,716
>> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO 
>> o.a.d.e.w.f.FragmentStatusReporter -
>> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State to report: RUNNING
>> 2015-09-30 21:14:36,875
>> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO 
>> o.a.d.e.w.fragment.FragmentExecutor -
>> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State change requested 
>> RUNNING --> FINISHED
>> 2015-09-30 21:14:36,876
>> [29f3777a-d407-295a-d45e-243fb70efc07:frag:0:0] INFO 
>> o.a.d.e.w.f.FragmentStatusReporter -
>> 29f3777a-d407-295a-d45e-243fb70efc07:0:0: State to report: FINISHED
>> 

RE: Drill- Query execution plan

[Sudip Mukherjee]

Hi Abdel,

Sorry for that. The code flow that I am trying to understand is below 

Like in my query::
SELECT SUM(1) AS `COL` FROM mydb.testtable HAVING COUNT(1)>0

Is the Projection broken down to 1) Final record output , 2) Projection on 
Aggregation  something like that. Please correct me if I am wrong.



00-00Screen : rowType = RecordType(INTEGER COL): rowcount = 1.0, cumulative 
cost = {62.1 rows, 607.1 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 217
00-01  Project(COL=[$0]) : rowType = RecordType(INTEGER COL): rowcount = 
1.0, cumulative cost = {62.0 rows, 607.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, 
id = 216
00-02SelectionVectorRemover : rowType = RecordType(INTEGER COL, BIGINT 
$f1): rowcount = 1.0, cumulative cost = {62.0 rows, 607.0 cpu, 0.0 io, 0.0 
network, 0.0 memory}, id = 215
00-03  Filter(condition=[>($1, 0)]) : rowType = RecordType(INTEGER COL, 
BIGINT $f1): rowcount = 1.0, cumulative cost = {61.0 rows, 606.0 cpu, 0.0 io, 
0.0 network, 0.0 memory}, id = 214
00-04StreamAgg(group=[{}], COL=[SUM($0)], agg#1=[COUNT()]) : 
rowType = RecordType(INTEGER COL, BIGINT $f1): rowcount = 1.0, cumulative cost 
= {60.0 rows, 600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 213
00-05  Project($f0=[1]) : rowType = RecordType(INTEGER $f0): 
rowcount = 20.0, cumulative cost = {40.0 rows, 120.0 cpu, 0.0 io, 0.0 network, 
0.0 memory}, id = 212
00-06Scan(groupscan=[GroupScan [ScanSpec=ScanSpec [tableName= 
testtable, filter=null], columns=[`*`]]]) : rowType = 
RecordType(VARCHAR(2147483647) Company, VARCHAR(2147483647) Name): rowcount = 
20.0, cumulative cost = {20.0 rows, 40.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, 
id = 211

Thanks,
Sudip

-Original Message-
From: Abdel Hakim Deneche [mailto:adene...@maprtech.com] 
Sent: 14 September 2015 PM 08:12
To: dev@drill.apache.org
Subject: Re: Drill- Query execution plan

Hi Sudip,

Did you include an image in your email ? unfortunately, Drill's mailing list 
doesn't allow any attachments, so we are not able to see the image

On Sun, Sep 13, 2015 at 10:51 PM, Sudip Mukherjee <smukher...@commvault.com>
wrote:

> Hi,
>
> Need some help understanding the below steps of a query execution. The 
> below query is broken down in to these steps.
>
> If you could just explain me in short or point to a documentation link 
> would be great for me as I am trying to dig into drill code and logics.
>
>
>
> *SELECT SUM(1) AS `COL` FROM mydb.testtable HAVING COUNT(1)>0 [query 
> is sent from tableau]*
>
>
>
>
>
>
>
>
>
> Sudip
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **
>
>


-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training 
<http://www.mapr.com/training?utm_source=Email_medium=Signature_campaign=Free%20available>



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Drill- Query execution plan

[Sudip Mukherjee]

Hi,
Need some help understanding the below steps of a query execution. The below 
query is broken down in to these steps.
If you could just explain me in short or point to a documentation link would be 
great for me as I am trying to dig into drill code and logics.

SELECT SUM(1) AS `COL` FROM mydb.testtable HAVING COUNT(1)>0 [query is sent 
from tableau]




Sudip[cid:image001.png@01D0EED8.6C54AE60]



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Drill - Error creating views

[Sudip Mukherjee]

Hi Devs,

I am getting the below exception while trying to create view from Web UI. Could 
you please take a look on how to troubleshoot this?

org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 
java.lang.IndexOutOfBoundsException: index (43) must be less than size (43)


QUERY that I gave is below :

CREATE OR REPLACE VIEW dfs.myviews.downloadcenterdataview as SELECT 
`bollink`,`category`,`categoryname`,`categoryid`,`downloadtype`,`earlypreviewusers`,`installpreference`,`notificationcontent`,`notificationusers`,`notvisibleto`,`packagedescription`,`packageid`,`packageplatformmappingid`,`packagelocation`,`packagename`,`packagesize`,`packagestatus`,`platform`,`platformname`,`productversion`,`productversionname`,`readmelocation`,`recutnumber`,`subcategory`,`subcategoryid`,`subcategoryname`,`validfrom`,`validto`,`vendor`,`visibleto`,`bolcontent`,`readmecontent`,`content`,`createtime`,`modifiedtime`,`serverid`,`softwareicon`,`reportname`,`reportdescription`,`reportguid`,`reportrevision`,`reportformat`,`includeTable`,`includeChart`,`itemrank`,`priceweightage`
 from mydb.downloadcenterdata

Stack trace :

2015-09-10 01:09:55,109 [qtp727140336-1839] ERROR 
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 
java.lang.IndexOutOfBoundsException: index (43) must be less than size (43)

at 
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118)
 ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:111) 
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47)
 ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32)
 ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) 
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) 
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) 
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
 ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: Drill Sql Max row size

[Sudip Mukherjee]

Hi,
I have somewhere around a million records. But columns are less in numbers (up 
to 10)

Thanks,
Sudip

-Original Message-
From: Jacques Nadeau [mailto:jacq...@dremio.com] 
Sent: 08 September 2015 AM 06:58
To: dev@drill.apache.org
Subject: Re: Drill Sql Max row size

Generally, no. That being said, Drill will probably struggle if you start 
reading records where one or more cells is greater than a few hundred kilobytes 
(or mbs) or more than several hundred columns/fields. What size records are you 
working with?

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee <smukher...@commvault.com>
wrote:

> Hi Devs,
>
> Is there a max row limit which I can configure while pulling data from 
> underlying datasource? If there is a large data-set would drill fetch 
> like page by page?
>
> Thanks,
> Sudip
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

RE: Drill Sql Max row size

[Sudip Mukherjee]

Yes, there is no problem in that . Was just wondering if there is a mechanism 
of getting data by batches from the record reader. 
Actually , I wrote a basic plugin to solr and was thinking if I paginate or use 
cursormark params in the recordreader.

Thanks,
Sudip

-Original Message-
From: Tomer Shiran [mailto:tshi...@dremio.com] 
Sent: 08 September 2015 PM 08:16
To: dev@drill.apache.org
Subject: Re: Drill Sql Max row size

That won't be a problem. There's actually no limit on how many records/rows you 
can have.  

> On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <smukher...@commvault.com> wrote:
> 
> Hi,
> I have somewhere around a million records. But columns are less in 
> numbers (up to 10)
> 
> Thanks,
> Sudip
> 
> -Original Message-
> From: Jacques Nadeau [mailto:jacq...@dremio.com]
> Sent: 08 September 2015 AM 06:58
> To: dev@drill.apache.org
> Subject: Re: Drill Sql Max row size
> 
> Generally, no. That being said, Drill will probably struggle if you start 
> reading records where one or more cells is greater than a few hundred 
> kilobytes (or mbs) or more than several hundred columns/fields. What size 
> records are you working with?
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee 
> <smukher...@commvault.com>
> wrote:
> 
>> Hi Devs,
>> 
>> Is there a max row limit which I can configure while pulling data 
>> from underlying datasource? If there is a large data-set would drill 
>> fetch like page by page?
>> 
>> Thanks,
>> Sudip
>> 
>> 
>> 
>> ***Legal 
>> Disclaimer***
>> "This communication may contain confidential and privileged material 
>> for the sole use of the intended recipient. Any unauthorized review, 
>> use or distribution by others is strictly prohibited. If you have 
>> received the message by mistake, please advise the sender by reply 
>> email and delete the message. Thank you."
>> *
>> *
> 
> 
> 
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Drill Sql Max row size

[Sudip Mukherjee]

Hi Devs,

Is there a max row limit which I can configure while pulling data from 
underlying datasource? If there is a large data-set would drill fetch like page 
by page?

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[Drill] Doubt in code flow

[Sudip Mukherjee]

Hi Devs,

I added some log-in in the drill code (copying excerpt of logs). I am seeing 
some repetitive log lines when is do a "show databases" . wondering why would 
schema scan would happen multiple times. Or is something I am missing. Could 
you please help? [below are the log-lines that I added]

2015-09-04 16:14:50,108 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.SchemaFactory - registering schema for 
plugin:org.apache.drill.exec.store.dfs.FileSystemPlugin@3cc861f9
2015-09-04 16:14:50,108 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs]
2015-09-04 16:14:50,108 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs, root]
2015-09-04 16:14:50,109 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs, tmp]
2015-09-04 16:14:50,109 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs, donuts]
2015-09-04 16:14:50,110 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs, default]

2015-09-04 16:14:50,145 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs.default]
2015-09-04 16:14:50,145 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs.donuts]
2015-09-04 16:14:50,145 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs.root]
2015-09-04 16:14:50,145 [2a1689de-9581-f912-3dc6-f4b61fc2e676:frag:0:0] INFO  
o.a.drill.exec.store.AbstractSchema - schemapath is...[dfs.tmp]


Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

Apache drill jdbc driver - can i connect to a drillbit?

[Sudip Mukherjee]

Hi Devs,

Is there way to connect a drillbit using the jdbc driver. Could you please 
point me to an example if there is one?

Thanks,
Sudip



***Legal Disclaimer***
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**

[Apache Drill] Creating Views

[Sudip Mukherjee]

Hi Devs,

I was trying to create views on top of my mongo datasource recently so that I 
can use them from BI tools. I was wondering if there is a rest api to create 
the views programmatically .
Actually the use case I was thinking that I will use the rest api to configure 
the FS workspace for the schema and I will start creating the views from a 
program..

Could you  clarify this part for me please.

Thanks,
Sudip




***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**

RE: Apache Drill: How does the plug-in know that an aggregate function is applied

[Sudip Mukherjee]

Hi,

Thanks a lot for this. I could use isSkipQuery for the count(*) in sql 
statement. Need to figure out if I can optimize the solr query (if applicable) 
for other aggregate functions like SUM where I can use a simple stats query on 
the given field.

Regards,
Sudip

-Original Message-
From: Hanifi Gunes [mailto:hgu...@maprtech.com] 
Sent: 10 August 2015 PM 11:27
To: user; dev@drill.apache.org
Subject: Re: Apache Drill: How does the plug-in know that an aggregate function 
is applied

+dev

+1 to Hakim. AbstractRR#isSkipQuery is the way to go. If you want more
details on this you should check out DRILL-2358[1] that is an umbrella issue 
targeting to make count(*) queries more efficient per storage plugin.
Currently (I guess) JSON and Mongo(?) readers support it though.

1:
https://github.com/apache/drill/commit/54df129cab544c3df8e75a7dae3f85a91a9ded5a

On Thu, Aug 6, 2015 at 6:50 PM, Abdel Hakim Deneche adene...@maprtech.com
wrote:

 Hi Sudip,

 I'm not really an expert in this matter but I came recently across
 isSkipQuery() method in AbstractRecordReader, it's javadoc states:

 *Returns true if reader should skip all of the columns, reporting 
 number of
  records only. Handling of a skip query is storage plugin-specific.*


 You can take a look at JSONRecordReader for an example on how to use
 isSkipQuery() to optimize the reading.

 Thanks

 On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee  
 mukherjeesud...@hotmail.com
  wrote:

  Hi,
  I am using apache drill recently with mongodb and trying to write a 
  basic plug-in for apache solr. Wanted to know how does the plug-in 
  know that an
  count(*) query has been applied so that the query to data-source can 
  be optimized? Can i get it if I extend  AbstractExprVisitor class?
 
  Thanks,Sudip Mukherjee




 --

 Abdelhakim Deneche

 Software Engineer

   http://www.mapr.com/


 Now Available - Free Hadoop On-Demand Training  
 http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm
 _campaign=Free%20available
 




***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**

RE: Aggregate queries in drill

[Sudip Mukherjee]

Hi Rahul,

I was trying something below where I am trying to see what is in the sql query 
but doesn't seem get the aggr functions!
https://github.com/sudipmukherjee/drill/blob/master/contrib/storage-solr/src/main/java/org/apache/drill/exec/store/solr/SolrQueryFilterRule.java
could you please have a look if you get a chance?

example physical plan for a query (select count(*) from solr.`bootstrap_5`; ) 
[bootstrap_5 is one of the cores I have in my solr engine]

2015-08-10 18:04:04,007 [2a3765c5-0e91-1f6e-5462-b134759bc9b7:foreman] DEBUG 
o.a.d.e.p.s.h.DefaultSqlHandler - Drill Physical : 
00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, 
cumulative cost = {60.1 rows, 340.1 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 
147
00-01  Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount 
= 1.0, cumulative cost = {60.0 rows, 340.0 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 146
00-02StreamAgg(group=[{}], EXPR$0=[COUNT()]) : rowType = 
RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {60.0 rows, 340.0 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 145
00-03  Project($f0=[0]) : rowType = RecordType(INTEGER $f0): rowcount = 
20.0, cumulative cost = {40.0 rows, 100.0 cpu, 0.0 io, 0.0 network, 0.0 
memory}, id = 144
00-04Scan(groupscan=[SolrGroupScan [SolrScanSpec=SolrScanSpec 
[solrCoreName=bootstrap_5, filter=null], columns=[`*`]]]) : rowType = 
(DrillRecordRow[*]): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 
0.0 io, 0.0 network, 0.0 memory}, id = 143

Excerpt of the plan :

graph : [ {
pop : solr-scan,
@id : 4,
solrPluginConfig : {
  type : solr,
  solrServer : http://localhost:2/solr/;,
  enabled : true
},
solrScanSpec : {
  solrCoreName : bootstrap_5,
  filter : null
},
columns : [ `*` ],
userName : smukherjee,
cost : 20.0
  }, {
pop : project,
@id : 3,
exprs : [ {
  ref : `$f0`,
  expr : 0
} ],
child : 4,
initialAllocation : 100,
maxAllocation : 100,
cost : 20.0
  }, {
pop : streaming-aggregate,
@id : 2,
child : 3,
keys : [ ],
exprs : [ {
  ref : `EXPR$0`,
  expr : count(1) 
} ],
initialAllocation : 100,
maxAllocation : 100,
cost : 1.0
  }

Thanks,
Sudip
-Original Message-
From: rahul challapalli [mailto:challapallira...@gmail.com] 
Sent: 07 August 2015 PM 01:23
To: dev@drill.apache.org
Subject: Re: Aggregate queries in drill

Sudip,

In your case, I would assume that you would construct something similar to the 
below :

1. Create your own optimizer rule (SolrPushAggIntoScan). Take a look at 
PruneScanRule. You should gather the LogicalAggregate and DrillScanRel objects 
from the RelOptRuleCall. Now from a high level you need to re-create the group 
scan with the aggregate information. Most likely you might to need to use an 
expression visitor in your SolrPushAggIntoScan class to figure out what 
aggregate functions you want to push into the scan
2. Now add your new rule(s) to the StoragePlugin.getOptimizerRules() method.

- Rahul


On Thu, Aug 6, 2015 at 10:00 PM, Sudip Mukherjee smukher...@commvault.com
wrote:

 Hi ,

 I am trying to make basic storage plugin for solr with drill. Is there 
 a way I could get the aggregate function information via expression 
 visitor in the plugin code so that I can optimize the Solr query as much as I 
 can.
 For example, for a count query I would just return the numFound from 
 solr response with rows =0.
 Source code : https://github.com/apache/drill/pull/100

 Could someone please help me on this?

 Thanks,
 Sudip Mukherjee




 ***Legal Disclaimer***
 This communication may contain confidential and privileged material 
 for the sole use of the intended recipient. Any unauthorized review, 
 use or distribution by others is strictly prohibited. If you have 
 received the message by mistake, please advise the sender by reply 
 email and delete the message. Thank you.
 **



***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**

RE: Lucene Format Plugin

[Sudip Mukherjee]

Hi Rahul,

Thanks for sharing your code. I was trying to get plugin for solr engine. But I 
thought of using solr's rest api to do the queries ,get schema metadata info 
etc. 
The goal for me is to expose a solr engine to tools like Tableau or  MS Excel 
and user can do stuff there.

I am still very new to this and there is a learning curve. It would be great if 
you can comment/review whatever I've done so far.

https://github.com/sudipmukherjee/drill/tree/master/contrib/storage-solr

Thanks,
Sudip

-Original Message-
From: rahul challapalli [mailto:challapallira...@gmail.com] 
Sent: 10 August 2015 AM 05:21
To: dev@drill.apache.org
Subject: Re: Lucene Format Plugin

Below is the link to my branch which contains the changes related to the format 
plugin.

https://github.com/rchallapalli/drill/tree/lucene/contrib/format-lucene

Any thoughts on how to handle contributions like this which still have some 
work to be done?

- Rahul


On Mon, Aug 3, 2015 at 12:21 PM, rahul challapalli  
challapallira...@gmail.com wrote:

 Thanks Jason.

 I want to look at the solr plugin and see where we can collaborate or 
 if we already duplicated part of the effort.

 I still need to push a few commits. I will share the code once I get 
 these changes pushed.

 - Rahul



 On Mon, Aug 3, 2015 at 11:31 AM, Jason Altekruse 
 altekruseja...@gmail.com
  wrote:

 Hey Rahul,

 This is really cool! Thanks for all of the time you put into writing 
 this, I think we have a lot of available opportunities to reach new 
 communities with efforts like this.

 I noticed last week another contributor opened a JIRA for a solr 
 plugin, there might be a good opportunity for the two of you to join 
 efforts, as I believe he likely stated working on a lucene reader as 
 part of his solr work.

 Would you like to post a link to your work on Github or another 
 public host of your code?

 https://issues.apache.org/jira/browse/DRILL-3585

 On Mon, Aug 3, 2015 at 2:29 AM, Stefán Baxter 
 ste...@activitystream.com
 wrote:

  Hi,
 
  I'm pretty new around here but I just wanted to tell you how much 
  your
 work
  can benefit us. This is great!.
 
  Look forward to trying it out.
 
  Regards,
   -Stefán
 
  On Mon, Aug 3, 2015 at 8:38 AM, rahul challapalli  
  challapallira...@gmail.com wrote:
 
   Hello Drillers,
  
   I have been working on a lucene format plugin. In its current 
   state,
 the
   below sample query successfully searches a lucene index and 
   returns
 the
   results.
  
   select path from dfs_test.`/search-index` where
  contents='maxItemsPerBlock'
   and contents = 'BlockTreeTermsIndex'
  
  
  
   *High Level Overview of Current Implementation:*
  
   *Parallelization:* A lucene segment is the lowest level of 
   parrallelization.
   *Filter Pushdown:* Currently the format plugin is designed to 
   push the complete filter into the scan.
   *Filter Evaluation:* Each condition in the filter is treated as a
 lucene
   TermQuery
   
  
 
 http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Ter
 mQuery.html
   
   and multiple conditions are joined using a BooleanQuery 
  
 
 http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/search/Boo
 leanQuery.html
   .
   If we *do not* use a TermQuery, then we have to know the exact 
   type of Analyzer 
  
 
 https://lucene.apache.org/core/5_2_1/core/org/apache/lucene/analysis/
 Analyzer.html
   
   to use with each field in the query.
   Ex: 'contents' field might have been analyzed using a
  StandardAnalyzer
   
  
 
 https://lucene.apache.org/core/5_2_1/analyzers-common/org/apache/luce
 ne/analysis/standard/StandardAnalyzer.html
   
   and the 'path' field might not have been analyzed at all.
   If desired, support for raw lucene queries with a reserved word
 should be
   easy to add.
   Ex: select * from dfs.`search-index` where searchQuery = 
   +contents:maxItemsPerBlock
   +path:/home/file.txt;
   *Converting SqlFilter to Lucene Query:* Currently only = and !=
   operators are handled while converting a sql filter into a lucene
 query.
   For indexed fields this might be sufficient to handle a good 
   number of cases. For non-indexed fields operators like ,, like 
   etc need to
 be
   handled.
   *FileSystems:* Currently the format plugin only works on a local 
   filesystem.
  
  
   Though far from complete, I want to work with the community to 
   get
 some
   feedback and avoid any chance of duplication of work. Kindly let 
   me
 know
   your thoughts
  
   - Rahul
  
 






***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**

Aggregate queries in drill

[Sudip Mukherjee]

Hi ,

I am trying to make basic storage plugin for solr with drill. Is there a way I 
could get the aggregate function information via expression visitor in the 
plugin code so that I can optimize the Solr query as much as I can.
For example, for a count query I would just return the numFound from solr 
response with rows =0.
Source code : https://github.com/apache/drill/pull/100

Could someone please help me on this?

Thanks,
Sudip Mukherjee




***Legal Disclaimer***
This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you.
**

[jira] [Created] (DRILL-3585) Apache Solr as a storage plugin

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-3585:
--

 Summary: Apache Solr as a storage plugin
 Key: DRILL-3585
 URL: https://issues.apache.org/jira/browse/DRILL-3585
 Project: Apache Drill
  Issue Type: New Feature
  Components: Client - HTTP
Reporter: Sudip Mukherjee
Assignee: Jason Altekruse


A new storage plugin supporting Apache solr search engine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3528) Apache drill plugin for solr

[Sudip Mukherjee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudip Mukherjee resolved DRILL-3528.

Resolution: Fixed

 Apache drill plugin for solr
 

 Key: DRILL-3528
 URL: https://issues.apache.org/jira/browse/DRILL-3528
 Project: Apache Drill
  Issue Type: New Feature
  Components: Storage - Other
 Environment: windows, eclipse ide,java 8 ,maven 3x
Reporter: Sudip Mukherjee
Assignee: Jacques Nadeau
 Attachments: storage-solr.rar


 I am trying to explore apache drill project and able to set it in my local 
 enviroment with the help of documentation. However, i was looking for some 
 code flow document to get started with writing a custom plug-in for trial. I 
 am trying add a type of apache solr to have a sql like interface on top of 
 it. 
 when i added the project plugin under /jars folder and tried to create new 
 storage of type solr got an exception Error : Invalid JSON mapping.
 Could you please help me with some debugging points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3528) Apache drill plugin for solr

[Sudip Mukherjee (JIRA)]

Sudip Mukherjee created DRILL-3528:
--

 Summary: Apache drill plugin for solr
 Key: DRILL-3528
 URL: https://issues.apache.org/jira/browse/DRILL-3528
 Project: Apache Drill
  Issue Type: New Feature
  Components: Storage - Other
 Environment: windows, eclipse ide
Reporter: Sudip Mukherjee
Assignee: Jacques Nadeau


I am trying to explore apache drill project and able to set it in my local 
enviroment with the help of documentation. However, i was looking for some code 
flow document to get started with writing a custom plug-in for trial. I am 
trying add a type of apache solr to have a sql like interface on top of it. 

when i added the project plugin under /jars folder and tried to create new 
storage of type solr got an exception Error : Invalid JSON mapping.

Could you please help me with some debugging points.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)