[jira] [Created] (DRILL-5374) Parquet filter pushdown does not prune partition with nulls when predicate uses float column

2017-03-21 Thread Robert Hou (JIRA) 
Robert Hou created DRILL-5374:
-

 Summary: Parquet filter pushdown does not prune partition with 
nulls when predicate uses float column
 Key: DRILL-5374
 URL: https://issues.apache.org/jira/browse/DRILL-5374
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Affects Versions: 1.9.0
Reporter: Robert Hou
Assignee: Jinfeng Ni


Drill does not prune enough partitions for this query when filter pushdown is 
used with metadata caching. The float column is being compared with a double 
value.

{code}
0: jdbc:drill:zk=10.10.100.186:5181/drill/rho> select count(*) from 
orders_parts_metadata where float_id < 1100.0;
{code}

To reproduce the problem, put the attached files into a directory. Then 

{code}
create the metadata:
refresh table metadata dfs.`path_to_directory`;
{code}

For example, if you put the files in 
/drill/testdata/filter/orders_parts_metadata, then run this sql command

{code}
refresh table metadata dfs.`/drill/testdata/filter/orders_parts_metadata`;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

2017-03-21 Thread Jinfeng Ni 
Very interesting findings, Francois. Thanks for sharing them with the community.

The change of max_per_node and affinity_factor seems to reduce the
possibility of one drillbit was hitting overload issue because of
either CPU or Network contention. In our in-house testing, we also
noticed that reducing max_per_node setting sometimes actually lead to
better performance.

The change you made in Foreman QueryManager probably is interesting. I
feel we probably want to come up with some similar ideas to address
the issue of temporary loss of connection between one drillbit and ZK.
Supposedly drillbitUnRegister should  trigger query cancellation only
when we know for sure (?) drillbit is dead.


On Tue, Mar 21, 2017 at 7:45 AM, François Méthot  wrote:
> Hi,
>
>  We have been having client-foreman connection and ZkConnection issue few
> months ago. It went from annoying to a show stopper when we moved from a 12
> nodes cluster to a 220 nodes cluster.
>
> Nodes specs
> - 8 cores total (2 x E5620)
> - 72 GB  RAM Total
> - Other applications share the same hardware.
>
> ~ 100 TB parquet data on hdfs.
>
>
>
>
> Based on our observation we have done few months ago, we ended up with
> those setting/guideline/changes:
>
> - Memory Setting
>   DRILL_MAX_DIRECT_MEMORY="20G"
>   DRILL_HEAP="8G"
>
>   Remaining RAM is for other applications
>
>
> - Threading
>   planner.width.max_per_node = 4
>
>   We think that higher number of threads will generate network traffic or
> more context switch on each node, leading to more chances of getting Zk
> disconnection.
>   But we observed that even with max_per_node of 1, we would still get
> disconnection. We had no clear indication from Cloudera Manager that
> Mem/CPU/Network is overloaded on faulty node. Although on very rare
> occasion we would get no stats  data at all from certain node.
>
> - Affinity Factor
>   We change the affinity factor from default to a big value.
>   planner.affinity_factor = 1000.0
>
>   This improved issue with some drillbit of our cluster scanning data
> stored on remote nodes. It somehow maximizes the chances of a drillbit
> reading local data. When drillbits only scan local data, it reduces the
> amount of network traffic, It accelerate queries and reduce the chance of
> ZkDisconnect.
>
> - If using hdfs, make sure each data file is stored on 1 block
>
> - Try more recent 1.8 JVM or switch to JVM 1.7
>   We have had CLIENT to  FOREMAN disconnection issue with certain version
> of JVM (linux, windows, mac). (we sent an email about this to the dev
> mailing list in the past)
>
> - Query Pattern
>   The more fields are getting selected (select * vs select few specific
> field) the more chance we will get the error. More data selected means more
> cpu/network activity leading to more chances of Zookeeper skipping a
> heartbeat.
>
>
> - Foreman QueryManager Resilience Hack
> When a query would fail, our log indicated that a drillbit was getting
> unregistered and then get registed again a short time after (few ms to few
> seconds), but the foreman QueryManager would catch the
> "drillbitUnregistered" event and fail the queries right away. As a test, we
> changed the QueryManager to not fail queries when a drillbit is getting
> unregistered. We have put this change in place in 1.8 and our log now
> indicates Zk Disconnect-Reconnect while query keeps running, so we kept
> that test code in. A query will now fail only if the drillbit lose
> connection with other drillbit (through the RPC bus) at some point. We have
> since move to 1.9 with that change as well. I haven't had chance to try
> back without the hack in 1.9.
>
> org.apache.drill.exec.work.foreman
>QueryManager.java
> private void drillbitUnregistered(.)
> 
> if (atLeastOneFailure)
>-> just log the error, do not cancel query.
>
> our query success rate went from <50% to >95% with all the changes above.
> We hope to get rid of the hack when an official fix is available.
>
>
>
> To cover the missing 5% error (any other type of errors), we advise users
> to try again. We also have built-in retry strategy implemented in our
> hourly python scripts that aggregates data.
>
> Hope it helps
>
> Francois
>
>
>
>
>
>
>
>
>
> On Thu, Mar 9, 2017 at 2:31 PM, Anup Tiwari 
> wrote:
>
>> Hi John,
>>
>> First of all sorry for delayed response and thanks for your suggestion,
>> reducing value of "planner.width.max_per_node" helped me a lot, above issue
>> which was coming 8 out of 10 times earlier now it is coming only 2 out of
>> 10 times.
>>
>> As mentioned above occurrences of connection error came down considerably,
>> but now sometimes i get "Heap Space Error" for few queries and due to this
>> sometimes drill-bits on some/all nodes gets killed. Let me know if any
>> other variable i can check for this(As of now, i have 8GB of Heap and 20GB
>> of Direct memory) :
>>
>> *Error Log :*
>>
>> ERROR o.a.drill.common.CatastrophicFailure

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

2017-03-21 Thread Jinfeng Ni 
Very interesting findings, Francois. Thanks for sharing them with the community.

The change of max_per_node and affinity_factor seems to reduce the
possibility of one drillbit was hitting overload issue because of
either CPU or Network contention. In our in-house testing, we also
noticed that reducing max_per_node setting sometimes actually lead to
better performance.

The change you made in Foreman QueryManager probably is interesting. I
feel we probably want to come up with some similar ideas to address
the issue of temporary loss of connection between one drillbit and ZK.
Supposedly drillbitUnRegister should  trigger query cancellation only
when we know for sure (?) drillbit is dead.


On Tue, Mar 21, 2017 at 7:45 AM, François Méthot  wrote:
> Hi,
>
>  We have been having client-foreman connection and ZkConnection issue few
> months ago. It went from annoying to a show stopper when we moved from a 12
> nodes cluster to a 220 nodes cluster.
>
> Nodes specs
> - 8 cores total (2 x E5620)
> - 72 GB  RAM Total
> - Other applications share the same hardware.
>
> ~ 100 TB parquet data on hdfs.
>
>
>
>
> Based on our observation we have done few months ago, we ended up with
> those setting/guideline/changes:
>
> - Memory Setting
>   DRILL_MAX_DIRECT_MEMORY="20G"
>   DRILL_HEAP="8G"
>
>   Remaining RAM is for other applications
>
>
> - Threading
>   planner.width.max_per_node = 4
>
>   We think that higher number of threads will generate network traffic or
> more context switch on each node, leading to more chances of getting Zk
> disconnection.
>   But we observed that even with max_per_node of 1, we would still get
> disconnection. We had no clear indication from Cloudera Manager that
> Mem/CPU/Network is overloaded on faulty node. Although on very rare
> occasion we would get no stats  data at all from certain node.
>
> - Affinity Factor
>   We change the affinity factor from default to a big value.
>   planner.affinity_factor = 1000.0
>
>   This improved issue with some drillbit of our cluster scanning data
> stored on remote nodes. It somehow maximizes the chances of a drillbit
> reading local data. When drillbits only scan local data, it reduces the
> amount of network traffic, It accelerate queries and reduce the chance of
> ZkDisconnect.
>
> - If using hdfs, make sure each data file is stored on 1 block
>
> - Try more recent 1.8 JVM or switch to JVM 1.7
>   We have had CLIENT to  FOREMAN disconnection issue with certain version
> of JVM (linux, windows, mac). (we sent an email about this to the dev
> mailing list in the past)
>
> - Query Pattern
>   The more fields are getting selected (select * vs select few specific
> field) the more chance we will get the error. More data selected means more
> cpu/network activity leading to more chances of Zookeeper skipping a
> heartbeat.
>
>
> - Foreman QueryManager Resilience Hack
> When a query would fail, our log indicated that a drillbit was getting
> unregistered and then get registed again a short time after (few ms to few
> seconds), but the foreman QueryManager would catch the
> "drillbitUnregistered" event and fail the queries right away. As a test, we
> changed the QueryManager to not fail queries when a drillbit is getting
> unregistered. We have put this change in place in 1.8 and our log now
> indicates Zk Disconnect-Reconnect while query keeps running, so we kept
> that test code in. A query will now fail only if the drillbit lose
> connection with other drillbit (through the RPC bus) at some point. We have
> since move to 1.9 with that change as well. I haven't had chance to try
> back without the hack in 1.9.
>
> org.apache.drill.exec.work.foreman
>QueryManager.java
> private void drillbitUnregistered(.)
> 
> if (atLeastOneFailure)
>-> just log the error, do not cancel query.
>
> our query success rate went from <50% to >95% with all the changes above.
> We hope to get rid of the hack when an official fix is available.
>
>
>
> To cover the missing 5% error (any other type of errors), we advise users
> to try again. We also have built-in retry strategy implemented in our
> hourly python scripts that aggregates data.
>
> Hope it helps
>
> Francois
>
>
>
>
>
>
>
>
>
> On Thu, Mar 9, 2017 at 2:31 PM, Anup Tiwari 
> wrote:
>
>> Hi John,
>>
>> First of all sorry for delayed response and thanks for your suggestion,
>> reducing value of "planner.width.max_per_node" helped me a lot, above issue
>> which was coming 8 out of 10 times earlier now it is coming only 2 out of
>> 10 times.
>>
>> As mentioned above occurrences of connection error came down considerably,
>> but now sometimes i get "Heap Space Error" for few queries and due to this
>> sometimes drill-bits on some/all nodes gets killed. Let me know if any
>> other variable i can check for this(As of now, i have 8GB of Heap and 20GB
>> of Direct memory) :
>>
>> *Error Log :*
>>
>> ERROR o.a.drill.common.CatastrophicFailure

[jira] [Created] (DRILL-5373) Drill JDBC error in the process of connection via SQuirrel: java.lang.NoClassDefFoundError: javax/validation/constraints/NotNull

2017-03-21 Thread Vitalii Diravka (JIRA) 
Vitalii Diravka created DRILL-5373:
--

 Summary: Drill JDBC error in the process of connection via 
SQuirrel: java.lang.NoClassDefFoundError: javax/validation/constraints/NotNull
 Key: DRILL-5373
 URL: https://issues.apache.org/jira/browse/DRILL-5373
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 1.10.0
 Environment: SQuirrel SQL Client Version 3.7
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
 Fix For: 1.11.0


When I tried to connect to drillbit by SQuirrel via jdbc I've got an error:
{code}
java.lang.NoClassDefFoundError: javax/validation/constraints/NotNull
at 
org.apache.drill.jdbc.impl.DrillMetaImpl.drillFieldMetaData(DrillMetaImpl.java:263)
at 
org.apache.drill.jdbc.impl.DrillMetaImpl$MetadataAdapter.getMeta(DrillMetaImpl.java:322)
at 
org.apache.drill.jdbc.impl.DrillMetaImpl.serverGetCatalogs(DrillMetaImpl.java:1034)
at 
org.apache.drill.jdbc.impl.DrillMetaImpl.getCatalogs(DrillMetaImpl.java:1077)
at 
oadd.org.apache.calcite.avatica.AvaticaDatabaseMetaData.getCatalogs(AvaticaDatabaseMetaData.java:609)
at 
org.apache.drill.jdbc.impl.DrillDatabaseMetaDataImpl.getCatalogs(DrillDatabaseMetaDataImpl.java:1168)
at 
net.sourceforge.squirrel_sql.fw.sql.SQLDatabaseMetaData.getCatalogs(SQLDatabaseMetaData.java:456)
at 
net.sourceforge.squirrel_sql.client.gui.session.CatalogsPanel.initInBackground(CatalogsPanel.java:93)
at 
net.sourceforge.squirrel_sql.client.gui.session.CatalogsPanel.(CatalogsPanel.java:56)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionPanel$MyToolBar.createGUI(SessionPanel.java:556)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionPanel$MyToolBar.(SessionPanel.java:525)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionPanel.propertiesHaveChanged(SessionPanel.java:435)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionPanel.initialize(SessionPanel.java:117)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionInternalFrame.createGUI(SessionInternalFrame.java:123)
at 
net.sourceforge.squirrel_sql.client.gui.session.SessionInternalFrame.(SessionInternalFrame.java:53)
at 
net.sourceforge.squirrel_sql.client.gui.WindowManager.createInternalFrame(WindowManager.java:405)
at 
net.sourceforge.squirrel_sql.client.mainframe.action.ConnectToAliasCommand$Runner.run(ConnectToAliasCommand.java:394)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:745)
at java.awt.EventQueue.access$300(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:706)
at java.awt.EventQueue$3.run(EventQueue.java:704)
at java.security.AccessController.doPrivileged(Native Method)
at 
java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:715)
at 
net.sourceforge.squirrel_sql.client.Main$1.dispatchEvent(Main.java:99)
at 
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at 
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at 
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

{code}

This annotation was added in DRILL-5301. But it requires a new dependency for 
the jdbc module.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

3/21 Hangout starts now

2017-03-21 Thread Chunhui Shi 
Hi,

I don't have topic for now. If you have anything want to raise for discussion 
please reply to the email or join the hangout at 
https://hangouts.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc


Thanks,

Chunhui

[GitHub] drill pull request #704: DRILL-5125: Provide option to use generic code for ...

2017-03-21 Thread ppadma 
Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/704#discussion_r107219761
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/svremover/TestSVRemover.java
 ---
@@ -34,4 +38,33 @@ public void testSVRWithNoFilter() throws Exception {
 int numOutputRecords = 
testPhysical(getFile("remover/sv_with_no_filter.json"));
 assertEquals(100, numOutputRecords);
   }
+
+  /**
+   * Test the generic version of the selection vector remover copier
+   * class. The code uses the traditional generated version by default.
+   * This test sets the option to use the generic version, then runs
+   * a query that exercises that version.
+   * 
+   * Note that the tests here exercise only the SV2 version of the
+   * selection remover; no tests exist for the SV4 version.
+   */
+
+  // TODO: Add an SV4 test once the improved mock data generator
+  // is available.
+
+  @Test
+  public void testGenericCopier() throws Exception {
+// TODO: replace this with new setup once revised test framework
+// is available.
+Properties config = new Properties( );
+config.put(ExecConstants.SYS_STORE_PROVIDER_LOCAL_ENABLE_WRITE, 
"false");
+config.put(ExecConstants.HTTP_ENABLE, "false");
+config.put(ExecConstants.REMOVER_ENABLE_GENERIC_COPIER, "true");
+updateTestCluster(1, DrillConfig.create(config));
+
+int numOutputRecords = testPhysical(getFile("remover/test1.json"));
+assertEquals(50, numOutputRecords);
+numOutputRecords = 
testPhysical(getFile("remover/sv_with_no_filter.json"));
+assertEquals(100, numOutputRecords);
+  }
 }
--- End diff --

It would be good to have the tests that cover all the vector types.  But,  
since it is off by default and you are exercising the code through other unit 
tests,  this is fine. 
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Hangouts: 03/21/2017

2017-03-21 Thread Arina Yelchiyeva 
Will hangouts be held today?


Kind regards
Arina

[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin

2017-03-21 Thread dmitriyHavrilovich 
Github user dmitriyHavrilovich commented on the issue:

https://github.com/apache/drill/pull/774
  
@arina-ielchiieva we done major refactor of code to follow your guideline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #520: DRILL-3510: Add ANSI_QUOTES option so that Drill's ...

2017-03-21 Thread vdiravka 
Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/520#discussion_r107198885
  
--- Diff: 
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillDatabaseMetaDataImpl.java
 ---
@@ -184,6 +184,18 @@ private ServerMeta getServerMeta() throws SQLException 
{
 return serverMeta;
   }
 
+  /**
+   * The same as {@link DrillDatabaseMetaDataImpl#getServerMeta()} but 
updates server metadata for the every call.
+   * Can be used for meta data that can be changed over the session
+   * @return server meta information.
+   * @throws SQLException for error when getting server meta
+   */
+  private ServerMeta getUpdatedServerMeta() throws SQLException {
--- End diff --

Deleted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #520: DRILL-3510: Add ANSI_QUOTES option so that Drill's ...

2017-03-21 Thread vdiravka 
Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/520#discussion_r107198532
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 ---
@@ -262,6 +266,22 @@ public long 
getParquetRowGroupFilterPushDownThreshold() {
 return 
options.getOption(PARQUET_ROWGROUP_FILTER_PUSHDOWN_PLANNING_THRESHOLD);
   }
 
+  /**
+   * @return Quoting enum for current quoting identifiers character
+   */
+  public Quoting getQuotingIdentifiers() {
--- End diff --

Done. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #520: DRILL-3510: Add ANSI_QUOTES option so that Drill's ...

2017-03-21 Thread vdiravka 
Github user vdiravka commented on a diff in the pull request:

https://github.com/apache/drill/pull/520#discussion_r107198625
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java ---
@@ -125,6 +123,15 @@ public Builder setSupportComplexTypes(boolean 
supportComplexTypes) {
 }
 
 public UserSession build() {
+  if (userSession.properties != null && 
userSession.properties.containsKey(DrillProperties.QUOTING_IDENTIFIERS)) {
--- End diff --

Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

2017-03-21 Thread François Méthot 
Hi,

 We have been having client-foreman connection and ZkConnection issue few
months ago. It went from annoying to a show stopper when we moved from a 12
nodes cluster to a 220 nodes cluster.

Nodes specs
- 8 cores total (2 x E5620)
- 72 GB  RAM Total
- Other applications share the same hardware.

~ 100 TB parquet data on hdfs.




Based on our observation we have done few months ago, we ended up with
those setting/guideline/changes:

- Memory Setting
  DRILL_MAX_DIRECT_MEMORY="20G"
  DRILL_HEAP="8G"

  Remaining RAM is for other applications


- Threading
  planner.width.max_per_node = 4

  We think that higher number of threads will generate network traffic or
more context switch on each node, leading to more chances of getting Zk
disconnection.
  But we observed that even with max_per_node of 1, we would still get
disconnection. We had no clear indication from Cloudera Manager that
Mem/CPU/Network is overloaded on faulty node. Although on very rare
occasion we would get no stats  data at all from certain node.

- Affinity Factor
  We change the affinity factor from default to a big value.
  planner.affinity_factor = 1000.0

  This improved issue with some drillbit of our cluster scanning data
stored on remote nodes. It somehow maximizes the chances of a drillbit
reading local data. When drillbits only scan local data, it reduces the
amount of network traffic, It accelerate queries and reduce the chance of
ZkDisconnect.

- If using hdfs, make sure each data file is stored on 1 block

- Try more recent 1.8 JVM or switch to JVM 1.7
  We have had CLIENT to  FOREMAN disconnection issue with certain version
of JVM (linux, windows, mac). (we sent an email about this to the dev
mailing list in the past)

- Query Pattern
  The more fields are getting selected (select * vs select few specific
field) the more chance we will get the error. More data selected means more
cpu/network activity leading to more chances of Zookeeper skipping a
heartbeat.


- Foreman QueryManager Resilience Hack
When a query would fail, our log indicated that a drillbit was getting
unregistered and then get registed again a short time after (few ms to few
seconds), but the foreman QueryManager would catch the
"drillbitUnregistered" event and fail the queries right away. As a test, we
changed the QueryManager to not fail queries when a drillbit is getting
unregistered. We have put this change in place in 1.8 and our log now
indicates Zk Disconnect-Reconnect while query keeps running, so we kept
that test code in. A query will now fail only if the drillbit lose
connection with other drillbit (through the RPC bus) at some point. We have
since move to 1.9 with that change as well. I haven't had chance to try
back without the hack in 1.9.

org.apache.drill.exec.work.foreman
   QueryManager.java
private void drillbitUnregistered(.)

if (atLeastOneFailure)
   -> just log the error, do not cancel query.

our query success rate went from <50% to >95% with all the changes above.
We hope to get rid of the hack when an official fix is available.



To cover the missing 5% error (any other type of errors), we advise users
to try again. We also have built-in retry strategy implemented in our
hourly python scripts that aggregates data.

Hope it helps

Francois









On Thu, Mar 9, 2017 at 2:31 PM, Anup Tiwari 
wrote:

> Hi John,
>
> First of all sorry for delayed response and thanks for your suggestion,
> reducing value of "planner.width.max_per_node" helped me a lot, above issue
> which was coming 8 out of 10 times earlier now it is coming only 2 out of
> 10 times.
>
> As mentioned above occurrences of connection error came down considerably,
> but now sometimes i get "Heap Space Error" for few queries and due to this
> sometimes drill-bits on some/all nodes gets killed. Let me know if any
> other variable i can check for this(As of now, i have 8GB of Heap and 20GB
> of Direct memory) :
>
> *Error Log :*
>
> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
> Occurred,
> exiting. Information message: Unable to handle out of memory condition in
> FragmentExecutor.
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.xerces.dom.DeferredDocumentImpl.
> getNodeObject(Unknown
> Source) ~[xercesImpl-2.11.0.jar:na]
> at
> org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Unknown
> Source) ~[xercesImpl-2.11.0.jar:na]
> at
> org.apache.xerces.dom.DeferredElementImpl.synchronizeChildren(Unknown
> Source) ~[xercesImpl-2.11.0.jar:na]
> at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source)
> ~[xercesImpl-2.11.0.jar:na]
> at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source)
> ~[xercesImpl-2.11.0.jar:na]
> at org.apache.xerces.dom.ElementImpl.normalize(Unknown Source)
> ~[xercesImpl-2.11.0.jar:na]
> at com.games24x7.device.NewDeviceData.setup(NewDeviceData.java:94)
>

[GitHub] drill issue #790: DRILL-5368: Fix memory leak issue in DrillClientImpl::proc...

2017-03-21 Thread sudheeshkatkam 
Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/790
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #791: DRILL-5369: Add initializer for ServerMetaContext

2017-03-21 Thread sudheeshkatkam 
Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/791
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #771: DRILL-5315: Address small typo in the comment

2017-03-21 Thread sudheeshkatkam 
Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/771
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---