[jira] [Created] (DRILL-7685) Case statement marking column as required in parquet metadata
Nitin Pawar created DRILL-7685: -- Summary: Case statement marking column as required in parquet metadata Key: DRILL-7685 URL: https://issues.apache.org/jira/browse/DRILL-7685 Project: Apache Drill Issue Type: Improvement Components: Storage - Parquet Affects Versions: 1.16.0 Reporter: Nitin Pawar We use apache drill for multi step processing. In one of the steps we have query as below ~create table dfs.tmp.`/t2` as select employee_id, case when department_id is not null then 1 else 2 end as case_output from cp.`employee.json`;~ This provides output as employee_id: OPTIONAL INT64 R:0 D:1 case_output: REQUIRED INT32 R:0 D:0 If we remove the end statement from case it does mark the column as optional. We feed this output to covariance function and because of this we get an error like below Error: Missing function implementation: [covariance(BIGINT-OPTIONAL, INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION-- -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7674) Ability to enfore quote chars for all non numeric fields. Also if we can a setting which can completely disables the quoting to retain backward compatibility that would h
[ https://issues.apache.org/jira/browse/DRILL-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Pawar updated DRILL-7674: --- Description: Ability to enfore quote chars for all non numeric fields. Also if we can a setting which can completely disables the quoting to retain backward compatibility that would help us as well. > Ability to enfore quote chars for all non numeric fields. Also if we can a > setting which can completely disables the quoting to retain backward > compatibility that would help us as well. > - > > Key: DRILL-7674 > URL: https://issues.apache.org/jira/browse/DRILL-7674 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text CSV >Affects Versions: 1.17.0 >Reporter: Nitin Pawar >Priority: Minor > > Ability to enfore quote chars for all non numeric fields. Also if we can a > setting which can completely disables the quoting to retain backward > compatibility that would help us as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7674) Ability to enfore quote chars for all non numeric fields. Also if we can a setting which can completely disables the quoting to retain backward compatibility that would h
Nitin Pawar created DRILL-7674: -- Summary: Ability to enfore quote chars for all non numeric fields. Also if we can a setting which can completely disables the quoting to retain backward compatibility that would help us as well. Key: DRILL-7674 URL: https://issues.apache.org/jira/browse/DRILL-7674 Project: Apache Drill Issue Type: Improvement Components: Storage - Text CSV Affects Versions: 1.17.0 Reporter: Nitin Pawar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012778#comment-17012778 ] Nitin Pawar commented on DRILL-7517: i meant the dump files are deleted by our ops guys already when they shutdown the machines. We have moved to drill 1.13 now > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > [jersey-container-servlet-core-2.25.1.jar:na] > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) > [jersey-container-servlet-core-2.25.1.jar:na] > at >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011621#comment-17011621 ] Nitin Pawar commented on DRILL-7517: I searched for dump file was unable to find any > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > [jersey-container-servlet-core-2.25.1.jar:na] > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010961#comment-17010961 ] Nitin Pawar commented on DRILL-7517: [~volodymyr], I am afraid I can not download the data files to local as that's customer data. These queries run perfectly fine on Drill-1.13 version which we are using currently. Issue related to OOM comes only for 1.16 Current guideline of having HEAP of 4 to 8 GB from the documentation, I kept it to 12GB. For now I have shutdown the 1.16 instances of drill as it had major impact on our production infra. Our current load of queries will be maximum 100 queries on drill cluster parallel at any given point of time. There might be other queries such as "explain plan extended" to know more details of the queries but I am not sure if that causes lots of issues as they are not processing any data as the plan we always run against cp.`employee.json` I will try to get heapdump and share it with you tomorrow > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010770#comment-17010770 ] Nitin Pawar commented on DRILL-7517: [~volodymyr], we did look at free memory available on machine it did have more than 30GB when the OOM came. > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > [jersey-container-servlet-core-2.25.1.jar:na] > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) > [jersey-container-servlet-core-2.25.1.jar:na] > at >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010768#comment-17010768 ] Nitin Pawar commented on DRILL-7517: [~volodymyr], we do not have an option to upgrade to 1.17 as it has some CSV fixes which breaks our current code. (eg: drill now adds double quote to csv by itself and we do the same so it gets doubled and data gets messed up). If there is a way where we could provide quotechar and escapechar to drill, we could move forward to 1.17 > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > [jersey-container-servlet-core-2.25.1.jar:na] > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010766#comment-17010766 ] Nitin Pawar commented on DRILL-7517: Here is the OOM log 2020-01-07 09:30:37,254 [21ebb195-58af-7ac6-e1d1-a7cebb1de4c8:foreman] ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory condition in Foreman. java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_222] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_222] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_222] at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_222] at org.apache.drill.common.expression.ExpressionStringBuilder.visitUnknown(ExpressionStringBuilder.java:351) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitUnknown(ExpressionStringBuilder.java:45) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.visitors.AbstractExprVisitor.visitTypedFieldExpr(AbstractExprVisitor.java:195) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.TypedFieldExpr.accept(TypedFieldExpr.java:38) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitFunctionHolderExpression(ExpressionStringBuilder.java:97) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitFunctionHolderExpression(ExpressionStringBuilder.java:45) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitFunctionCall(ExpressionStringBuilder.java:77) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitBooleanOperator(ExpressionStringBuilder.java:85) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitBooleanOperator(ExpressionStringBuilder.java:45) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.BooleanOperator.accept(BooleanOperator.java:35) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitFunctionCall(ExpressionStringBuilder.java:77) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitBooleanOperator(ExpressionStringBuilder.java:85) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.visitBooleanOperator(ExpressionStringBuilder.java:45) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.BooleanOperator.accept(BooleanOperator.java:35) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.common.expression.ExpressionStringBuilder.toString(ExpressionStringBuilder.java:52) ~[drill-logical-1.16.0.jar:1.16.0] at org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata.getFilterPredicate(AbstractGroupScanWithMetadata.java:314) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.applyFilter(AbstractParquetGroupScan.java:216) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.parquet.ParquetPushDownFilter.doOnMatch(ParquetPushDownFilter.java:175) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.parquet.ParquetPushDownFilter$2.onMatch(ParquetPushDownFilter.java:103) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:319) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:561) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:257) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:216) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:203) ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:419) ~[drill-java-exec-1.16.0.jar:1.16.0] > Drill 1.16.0 shuts down frequently > -- > >
[jira] [Commented] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009996#comment-17009996 ] Nitin Pawar commented on DRILL-7517: Questions from mail thread: Number of parallel queries running: Not more than 60 at a time Number of machines in cluster : 4 Config of each machine : 64GB RAM with 8 cores Heap configured: 12GB Direct memory: 40GB MAX memory: 56GB > Drill 1.16.0 shuts down frequently > -- > > Key: DRILL-7517 > URL: https://issues.apache.org/jira/browse/DRILL-7517 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Nitin Pawar >Priority: Critical > > We see following exception every few hours > Our drillbit cluster queries data from S3. The only queries we make to web > interface are for explain plan and no actual query goes via WEB UI. > here is the full exception > 2020-01-07 16:34:02,922 [qtp80683229-962] INFO > o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not > enough heap memory to run this query using the web interface. > org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is > not enough heap memory to run this query using the web interface. > Please try a query with fewer columns or with a filter or limit condition to > limit the data returned. > You can also try an ODBC/JDBC client. > [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) > ~[drill-common-1.16.0.jar:1.16.0] > at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) > [drill-java-exec-1.16.0.jar:1.16.0] > at > org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) > [drill-java-exec-1.16.0.jar:1.16.0] > at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_222] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > [jersey-server-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [jersey-common-2.25.1.jar:na] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > [jersey-common-2.25.1.jar:na] > at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > [jersey-server-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > [jersey-container-servlet-core-2.25.1.jar:na] > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > [jersey-container-servlet-core-2.25.1.jar:na] > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) > [jersey-container-servlet-core-2.25.1.jar:na] > at >
[jira] [Updated] (DRILL-7517) Drill 1.16.0 shuts down frequently
[ https://issues.apache.org/jira/browse/DRILL-7517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Pawar updated DRILL-7517: --- Description: We see following exception every few hours Our drillbit cluster queries data from S3. The only queries we make to web interface are for explain plan and no actual query goes via WEB UI. here is the full exception 2020-01-07 16:34:02,922 [qtp80683229-962] INFO o.a.d.exec.server.rest.QueryWrapper - User Error Occurred: There is not enough heap memory to run this query using the web interface. org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is not enough heap memory to run this query using the web interface. Please try a query with fewer columns or with a filter or limit condition to limit the data returned. You can also try an ODBC/JDBC client. [Error Id: 7ad61839-a2e8-4fdd-a600-e662fc0f03e0 ] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) ~[drill-common-1.16.0.jar:1.16.0] at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:115) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:74) [drill-java-exec-1.16.0.jar:1.16.0] at sun.reflect.GeneratedMethodAccessor212.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na] at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na] at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [jetty-servlet-9.3.25.v20180904.jar:9.3.25.v20180904] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.3.25.v20180904.jar:9.3.25.v20180904] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.25.v20180904.jar:9.3.25.v20180904] at
[jira] [Created] (DRILL-7517) Drill 1.16.0 shuts down frequently
Nitin Pawar created DRILL-7517: -- Summary: Drill 1.16.0 shuts down frequently Key: DRILL-7517 URL: https://issues.apache.org/jira/browse/DRILL-7517 Project: Apache Drill Issue Type: Bug Affects Versions: 1.16.0 Reporter: Nitin Pawar We see following exception every few hours Our drillbit cluster queries data from S3. [Error Id: 1f14f398-fab2-40cd-b14c-389d91c002e8 on 10.0.160.5:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Connection pool shut down Fragment 1:0 Please, refer to logs for more information. [Error Id: 1f14f398-fab2-40cd-b14c-389d91c002e8 on 10.0.160.5:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:630) ~[drill-common-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:363) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:329) [drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.16.0.jar:1.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222] Caused by: java.lang.IllegalStateException: Connection pool shut down at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:167) ~[httpcore-4.2.4.jar:4.2.4] at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:199) ~[httpcore-4.2.4.jar:4.2.4] at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:188) ~[httpclient-4.2.5.jar:4.2.5] at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222] at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:72) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.http.conn.$Proxy76.requestConnection(Unknown Source) ~[na:na] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:448) ~[httpclient-4.2.5.jar:4.2.5] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[httpclient-4.2.5.jar:4.2.5] at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) ~[httpclient-4.2.5.jar:4.2.5] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976) ~[aws-java-sdk-1.7.4.jar:na] at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956) ~[aws-java-sdk-1.7.4.jar:na] at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:892) ~[hadoop-aws-2.7.4.jar:na] at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77) ~[hadoop-aws-2.7.4.jar:na] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) ~[hadoop-common-2.7.4.jar:na] at org.apache.drill.exec.store.StorageStrategy.getNonExistentLocations(StorageStrategy.java:200) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.StorageStrategy.createFileAndApply(StorageStrategy.java:134) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.text.DrillTextRecordWriter.startNewSchema(DrillTextRecordWriter.java:92) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.StringOutputRecordWriter.updateSchema(StringOutputRecordWriter.java:44) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:160) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:108) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) ~[drill-java-exec-1.16.0.jar:1.16.0] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.16.0.jar:1.16.0] at
[jira] [Created] (DRILL-6551) Concat function results in SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer.
Nitin Pawar created DRILL-6551: -- Summary: Concat function results in SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer. Key: DRILL-6551 URL: https://issues.apache.org/jira/browse/DRILL-6551 Project: Apache Drill Issue Type: Bug Affects Versions: 1.13.0 Reporter: Nitin Pawar I get following error when doing concat(string1, string2) SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer. Fragment 0:0 [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on ip-10-0-103-63.ec2.internal:31010] Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer. Fragment 0:0 [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on ip-10-0-103-63.ec2.internal:31010] here in the function string1 ends with " - " and string2 is pretty large string. The error goes away when i do concat(string1, ' ', string2) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6105) SYSTEM ERROR: NullPointerException
Nitin Pawar created DRILL-6105: -- Summary: SYSTEM ERROR: NullPointerException Key: DRILL-6105 URL: https://issues.apache.org/jira/browse/DRILL-6105 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.12.0 Reporter: Nitin Pawar We just upgraded drill from 1.8.0 to 1.12.0 One of the issue which left us behind was whenever we upgraded Drill we were hit will null pointer exception in scenario which we could not spend time debugging. Then we rolled back to 1.8.0 This time we have spent some time and understood the problem data set: 3 million records 5 columns, 2 of which are date columns (Let us call these date columns as A and B) In column B majority of the values are nulls (roughly 99.5%) our query is select * from table where A = B above query fails with null pointer exception where the number of underlying parquet files are more than 1. If I merge the underlying parquet files into a single large file then we do not see this issue. We also were able to remove the error if we changed the where clause with coalesce(A,null) = coalesce(B,null) This somehow points us that the error is coming while reading the parquet files with the given data types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5099) support OFFSET values greater than one, for LEAD & LAG window functions
[
https://issues.apache.org/jira/browse/DRILL-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943019#comment-15943019
]
Nitin Pawar commented on DRILL-5099:
[~khfaraaz] I am almost done with changes for this JIRA
I am currently stuck at the point when copyNext has argument from next
partition.
Can any dev from MapR spare 15 minutes to help me resolve the issue.
> support OFFSET values greater than one, for LEAD & LAG window functions
> ---
>
> Key: DRILL-5099
> URL: https://issues.apache.org/jira/browse/DRILL-5099
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Priority: Minor
> Labels: window_function
>
> Provide support for OFFSET values greater than one, for LEAD & LAG window
> functions.
> Adding [~adeneche] comments from the dev list here
> {noformat}
> Things that need to be done to make Lag (or Lead) support offsets other
> than 1:
> - WindowFunction.Lead should extract the offset value from its FunctionCall
> argument, you can look at WindowFunctionNtile.numTilesFromExpression() for
> and example on how to do that.
> - make sure calls to copyNext() and copyPrev() in NoFrameSupportTemplate
> use the offset and not the hard coded value (you already figured that out)
> - finally make sure you update UnsupportedOperatorsVisitor to no longer
> throw an exception when we pass an offset value other than 1 to Lead or
> Lag. Just search for DRILL-3596 in that class and you will find the if
> block that need to be removed
> I think this should be enough to get it to work in the general case.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (DRILL-4469) SUM window query returns incorrect results over integer data
[
https://issues.apache.org/jira/browse/DRILL-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818022#comment-15818022
]
Nitin Pawar commented on DRILL-4469:
[~khfaraaz] would it be a good idea to disable this feature if it is giving
wrong results?
> SUM window query returns incorrect results over integer data
>
>
> Key: DRILL-4469
> URL: https://issues.apache.org/jira/browse/DRILL-4469
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.6.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>Priority: Critical
> Labels: window_function
> Attachments: t_alltype.csv, t_alltype.parquet
>
>
> SUM window query returns incorrect results as compared to Postgres, with or
> without the frame clause in the window definition. Note that there is a sub
> query involved and data in column c1 is sorted integer data with no nulls.
> Drill 1.6.0 commit ID: 6d5f4983
> Results from Drill 1.6.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT SUM(c1) OVER w FROM (select * from
> dfs.tmp.`t_alltype`) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1 RANGE
> BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING);
> +-+
> | EXPR$0 |
> +-+
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> ...
> | 10585 |
> | 10585 |
> | 10585 |
> ++
> 145 rows selected (0.257 seconds)
> {noformat}
> results from Postgres 9.3
> {noformat}
> postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW
> w AS (PARTITION BY c8 ORDER BY c1 RANGE BETWEEN UNBOUNDED PRECEDING AND
> UNBOUNDED FOLLOWING);
> sum
> --
> 4499
> 4499
> 4499
> 4499
> 4499
> 4499
> ...
> 5613
> 5613
> 5613
> 473
> 473
> 473
> 473
> 473
> (145 rows)
> {noformat}
> Removing the frame clause from window definition, still results in completely
> different results on Postgres vs Drill
> Results from Drill 1.6.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp>SELECT SUM(c1) OVER w FROM (select * from
> t_alltype) subQry WINDOW w AS (PARTITION BY c8 ORDER BY c1);
> +-+
> | EXPR$0 |
> +-+
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> ...
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> | 10585 |
> ++
> 145 rows selected (0.28 seconds)
> {noformat}
> Results from Postgres
> {noformat}
> postgres=# SELECT SUM(c1) OVER w FROM (select * from t_alltype) subQry WINDOW
> w AS (PARTITION BY c8 ORDER BY c1);
> sum
> --
> 5
>12
>21
>33
>47
>62
>78
>96
> 115
> 135
> 158
> 182
> 207
> 233
> 260
> 289
> ...
> 4914
> 5051
> 5189
> 5328
> 5470
> 5613
> 8
>70
> 198
> 332
> 473
> (145 rows)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4566) Add TDigest functions for computing median and quantile
[ https://issues.apache.org/jira/browse/DRILL-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427897#comment-15427897 ] Nitin Pawar commented on DRILL-4566: Is there any timeline or version when this will be merged in ? > Add TDigest functions for computing median and quantile > --- > > Key: DRILL-4566 > URL: https://issues.apache.org/jira/browse/DRILL-4566 > Project: Apache Drill > Issue Type: New Feature >Reporter: Steven Phillips >Assignee: Steven Phillips > > The tdigest library can be used by Drill to compute approximate value and > percentiles with using too much memory or spilling to disk, which would be > required to compute exactly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
