Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-09 Thread Zelaine Fong
r. Can you turn on the root logger to > > be at the debug level and retry the explain plan? > > > > > > Kunal Khatua > > > > > > > > From:

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-09 Thread Jinfeng Ni
y with explicit column names > > >> 3. Remove the extract header from your storage plugin configuration > and > > >> from your data files? Rewrite your query to use, > columns[0_based_index] > > >> instead of explicit column names > &

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-08 Thread Zelaine Fong
he debug level and retry the explain plan? > > > Kunal Khatua > > > > From: rahul challapalli <challapallira...@gmail.com> > Sent: Tuesday, March 7, 2017 5:24:43 AM > To: user > Subject:

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-07 Thread PROJJWAL SAHA
From: rahul challapalli <challapallira...@gmail.com> > Sent: Tuesday, March 7, 2017 5:24:43 AM > To: user > Subject: Re: Minimise query plan time for dfs plugin for local file system > on tsv file > > I did not get a chance to review the log file. > > However the nex

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-07 Thread Kunal Khatua
:43 AM To: user Subject: Re: Minimise query plan time for dfs plugin for local file system on tsv file I did not get a chance to review the log file. However the next thing I would try is to make your cluster a single node cluster first and then run the same explain plan query separately o

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-07 Thread rahul challapalli
so long for files which are local >> to >> > the drill machines. I have the same directory structure copied on all >> the 5 >> > nodes of the cluster. I am accessing the source files using out of the >> box >> > dfs storage plugin. >> > >> &g

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-06 Thread rahul challapalli
WAL SAHA <proj.s...@gmail.com> > Sent: Monday, March 6, 2017 1:45:38 AM > To: user@drill.apache.org > Subject: Fwd: Minimise query plan time for dfs plugin for local file > system on tsv file > > all, please help me in giving suggestions on what areas i can look into &g

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-06 Thread Gautam Parai
Can you please provide the drillbit.log file? Gautam From: PROJJWAL SAHA <proj.s...@gmail.com> Sent: Monday, March 6, 2017 1:45:38 AM To: user@drill.apache.org Subject: Fwd: Minimise query plan time for dfs plugin for local file system on tsv file all,

Fwd: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-06 Thread PROJJWAL SAHA
dfs storage plugin. Query planning time is approx 30 secs Query execution time is apprx 1.5 secs Regards, Projjwal -- Forwarded message -- From: PROJJWAL SAHA <proj.s...@gmail.com> Date: Fri, Mar 3, 2017 at 5:06 PM Subject: Minimise query plan time for dfs plugin for loca

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-05 Thread PROJJWAL SAHA
The files are copied to the same location in all the nodes of the cluster. And all the nodes have equal access to the files. The files are not located to a single shared file system. On Fri, Mar 3, 2017 at 7:12 PM, John Omernik wrote: > Can you help me understand what "local

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-03 Thread John Omernik
Can you help me understand what "local to the cluster" means in the context of a 5 node cluster? In the plan, the files are all file:// Are the files replicated to each node? is it a common shared filesystem? Do all 5 nodes have equal access to the 10 files? I wonder if using a local FS in a

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-03 Thread PROJJWAL SAHA
I did not change the default values used by drill. Are you talking of changing planner.memory_limit and planner.memory.max_query_memory_per_node ? If there are any other debug work that I can do, pls suggest Regards On Fri, Mar 3, 2017 at 5:14 PM, Nitin Pawar wrote: >

Re: Minimise query plan time for dfs plugin for local file system on tsv file

2017-03-03 Thread Nitin Pawar
how much memory have you set for planner ? On Fri, Mar 3, 2017 at 5:06 PM, PROJJWAL SAHA wrote: > Hello all, > > I am quering select * from dfs.xxx where yyy (filter condition) > > I am using dfs storage plugin that comes out of the box from drill on a > 1GB file, local to