Hi Francois,

More questions..

> + Can you share the query profile?
>   I will sum it up:
>  It is a select on 18 columns: 9 string, 9 integers.
>  Scan is done on 13862 parquet files spread  on 1000 fragments.
>  Fragments are spread accross 215 nodes.

So ~5 leaf fragments (or scanners) per Drillbit seems fine.

+ Does the query involve any aggregations or filters? Or is this a select query 
with only projections?
+ Any suspicious timings in the query profile?
+ Any suspicious warning messages in the logs around the time of failure on any 
of the drillbits? Specially on atsqa4-133.qa.lab? Specially this one (“..” are 
place holders):
  Message of mode .. of rpc type .. took longer than ..ms.  Actual duration was 
..ms.

Thank you,
Sudheesh

> On Sep 15, 2016, at 11:27 AM, François Méthot <fmetho...@gmail.com> wrote:
> 
> Hi Sudheesh,
> 
> + How many zookeeper servers in the quorum?
> The quorum has 3 servers, everything looks healthy
> 
> + What is the load on atsqa4-133.qa.lab when this happens? Any other
> applications running on that node? How many threads is the Drill process
> using?
> The load on the failing node(8 cores) is 14, when Drill is running. Which
> is nothing out of the ordinary according to our admin.
> HBase is also running.
> planner.width.max_per_node is set to 8
> 
> + When running the same query on 12 nodes, is the data size same?
> Yes
> 
> + Can you share the query profile?
>   I will sum it up:
>  It is a select on 18 columns: 9 string, 9 integers.
>  Scan is done on 13862 parquet files spread  on 1000 fragments.
>  Fragments are spread accross 215 nodes.
> 
> 
> We are in process of increasing our Zookeeper session timeout config to see
> if it helps.
> 
> thanks
> 
> Francois
> 
> 
> 
> 
> 
> 
> 
> On Wed, Sep 14, 2016 at 4:40 PM, Sudheesh Katkam <skat...@maprtech.com>
> wrote:
> 
>> Hi Francois,
>> 
>> Few questions:
>> + How many zookeeper servers in the quorum?
>> + What is the load on atsqa4-133.qa.lab when this happens? Any other
>> applications running on that node? How many threads is the Drill process
>> using?
>> + When running the same query on 12 nodes, is the data size same?
>> + Can you share the query profile?
>> 
>> This may not be the right thing to do, but for now, If the cluster is
>> heavily loaded, increase the zk timeout.
>> 
>> Thank you,
>> Sudheesh
>> 
>>> On Sep 14, 2016, at 11:53 AM, François Méthot <fmetho...@gmail.com>
>> wrote:
>>> 
>>> We are running 1.7.
>>> The log were taken from the jira tickets.
>>> 
>>> We will try out 1.8 soon.
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Sep 14, 2016 at 2:52 PM, Chun Chang <cch...@maprtech.com> wrote:
>>> 
>>>> Looks like you are running 1.5. I believe there are some work done in
>> that
>>>> area and the newer release should behave better.
>>>> 
>>>> On Wed, Sep 14, 2016 at 11:43 AM, François Méthot <fmetho...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> We are trying to find a solution/workaround to issue:
>>>>> 
>>>>> 2016-01-28 16:36:14,367 [Curator-ServiceCache-0] ERROR
>>>>> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: ForemanException:
>>>>> One more more nodes lost connectivity during query.  Identified nodes
>>>>> were [atsqa4-133.qa.lab:31010].
>>>>> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
>>>>> ForemanException: One more more nodes lost connectivity during query.
>>>>> Identified nodes were [atsqa4-133.qa.lab:31010].
>>>>>       at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
>>>>> close(Foreman.java:746)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> processEvent(Foreman.java:858)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> processEvent(Foreman.java:790)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> moveToState(Foreman.java:792)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman.moveToState(
>>>>> Foreman.java:909)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman.access$2700(
>>>>> Foreman.java:110)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>       at org.apache.drill.exec.work.foreman.Foreman$StateListener.
>>>>> moveToState(Foreman.java:1183)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>> 
>>>>> 
>>>>> DRILL-4325  <https://issues.apache.org/jira/browse/DRILL-4325>
>>>>> ForemanException:
>>>>> One or more nodes lost connectivity during query
>>>>> 
>>>>> 
>>>>> 
>>>>> Any one experienced this issue ?
>>>>> 
>>>>> It happens when running query involving many parquet files on a cluster
>>>> of
>>>>> 200 nodes. Same query on a smaller cluster of 12 nodes runs fine.
>>>>> 
>>>>> It is not caused by garbage collection, (checked on both ZK node and
>> the
>>>>> involved drill bit).
>>>>> 
>>>>> Negotiated max session timeout is 40 seconds.
>>>>> 
>>>>> The sequence seems:
>>>>> - Drill Query begins, using an existing ZK session.
>>>>> - Drill Zk session timeouts
>>>>>     - perhaps it was writing something that took too long
>>>>> - Drill attempts to renew session
>>>>>      - drill believes that the write operation failed, so it attempts
>>>> to
>>>>> re-create the zk node, which trigger another exception.
>>>>> 
>>>>> We are open to any suggestion. We will report any finding.
>>>>> 
>>>>> Thanks
>>>>> Francois
>>>>> 
>>>> 
>> 
>> 

Reply via email to