[jira] [Commented] (DRILL-7364) Timeout reading from Kafka topic using Kafka plugin
[
https://issues.apache.org/jira/browse/DRILL-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923940#comment-16923940
]
Abhishek Ravi commented on DRILL-7364:
--
In my experience, the issue of "Failed to fetch messages" on MapR Streams is
hit when "default stream" has not been configured in plugin config.
To answer your additional questions
1) Yes, with 20 partitions the query will run with 20 minor fragments.
2) Each minor fragment is equivalent to a Kafka Consumer and will be part of a
group.
> Timeout reading from Kafka topic using Kafka plugin
> ---
>
> Key: DRILL-7364
> URL: https://issues.apache.org/jira/browse/DRILL-7364
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Kafka
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Aditya Allamraju
>Priority: Major
>
> When we try to query Mapr-streams(similar to Apache Kafka) topic using Kafka
> plugin, we see the below timeout being thrown.
> {code:java}
> 0: jdbc:drill:drillbit=10.10.75.158:31010>
> 0: jdbc:drill:drillbit=10.10.75.158:31010> select count(*) from
> `/sample-stream:fast-messages` where k<100;
> Error: DATA_READ ERROR: Failed to fetch messages within 200 milliseconds.
> Consider increasing the value of the property : store.kafka.poll.timeout
> DATA_READ ERROR: Failed to fetch messages within 200 milliseconds. Consider
> increasing the value of the property : store.kafka.poll.timeout
> [Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 ]
> Fragment 0:0
> [Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 on
> vm75-158.support.mapr.com:31010] (state=,code=0)
> 0: jdbc:drill:drillbit=10.10.75.158:31010> ALTER SYSTEM SET
> `store.kafka.poll.timeout` = 5;
> +---++
> | ok | summary |
> +---++
> | true | store.kafka.poll.timeout updated. |
> +---++
> 1 row selected (0.148 seconds)
> 0: jdbc:drill:drillbit=10.10.75.158:31010>
> {code}
> The other interesting behavior is that:
> 1) Even after increasing the timeout value to 50secs, the drill query failed
> after the execution time crossed 50 secs(~51 secs).
> This pattern continued to whatever value we increased. For ex, after
> increasing the timeout to 100secs, query failed with above error after 101
> secs of execution time.
> The user is using Drill 1.15.
> I tried to reproduce this on my test cluster. But this was not consistently
> reproducing in the test cluster. Whereas in the client's cluster, we were
> able to reproduce this behavior consistently. We collected the logs. But they
> have very little info on what's happening.
> I believe it is now essential to know how(and why) the timeout parameter
> "store.kafka.poll.timeout" is related to the Query execution time to
> understand this bug.
> I also have few more questions where i couldn't find much documentation.
> _1) Is there a one-to-one mapping between number of partitions of a topic to
> minor fragments of a query? For ex, If a given topic(t) has say 20
> partitions, then will the query most likely have 20 minor fragments, other
> parameters being fairly sized._
> _2) Does each drillbit in the cluster equivalent to a Kafka-Consumer and all
> such drillbits of a cluster treated as part of a consumer group?_
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7140) RM: Drillbits fail with "No enum constant org.apache.drill.exec.resourcemgr.config.selectionpolicy.QueueSelectionPolicy.SelectionPolicy.bestfit"
Abhishek Ravi created DRILL-7140:
Summary: RM: Drillbits fail with "No enum constant
org.apache.drill.exec.resourcemgr.config.selectionpolicy.QueueSelectionPolicy.SelectionPolicy.bestfit"
Key: DRILL-7140
URL: https://issues.apache.org/jira/browse/DRILL-7140
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 2.0.0
Environment: master + changes to enable RM
Reporter: Abhishek Ravi
Assignee: Sorabh Hamirwasia
Fix For: 2.0.0
A sample configuration for RM with value *{{bestfit}}* for
*{{queue_selection_policy}}* fails with
{noformat}
at
org.apache.drill.exec.resourcemgr.config.ResourcePoolTreeImpl.(ResourcePoolTreeImpl.java:82)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.resourcemgr.config.ResourcePoolTreeImpl.(ResourcePoolTreeImpl.java:63)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.rm.DistributedResourceManager.(DistributedResourceManager.java:46)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
... 6 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant
org.apache.drill.exec.resourcemgr.config.selectionpolicy.QueueSelectionPolicy.SelectionPolicy.bestfit
at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_181]
at
org.apache.drill.exec.resourcemgr.config.selectionpolicy.QueueSelectionPolicy$SelectionPolicy.valueOf(QueueSelectionPolicy.java:32)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.resourcemgr.config.ResourcePoolTreeImpl.(ResourcePoolTreeImpl.java:74)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
... 8 common frames omitted
{noformat}
The issue here seems to be the case mismatch between *{{bestfit}}* and enum
constant *{{BESTFIT}}*. Hence {{SelectionPolicy.valueOf}} does not find
*{{bestfit}}*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6849) Runtime filter queries with nested broadcast returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi resolved DRILL-6849.
--
Resolution: Duplicate
> Runtime filter queries with nested broadcast returns wrong results
> --
>
> Key: DRILL-6849
> URL: https://issues.apache.org/jira/browse/DRILL-6849
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
>
> Running few queries on TPC-H SF100 data with latest changes in [PR
> #1504|[https://github.com/apache/drill/pull/1504].]
> I noticed that for a couple of queries, there are vast discrepancies in the
> results returned by queries with *Runtime Filter enabled* and without.
> Below are a couple of failing queries.
> h3. Query 1
> {code:sql}
> select
> p.p_mfgr,
> p.p_type,
> count(*) as num_parts
> from
> supplier s,
> part p,
> partsupp ps,
> nation n
> where
> n.n_nationkey = 15
> and n.n_nationkey = s.s_nationkey
> and s.s_suppkey = ps.ps_suppkey
> and ps.ps_partkey = p.p_partkey
> and p.p_type like '%STEEL%'
> group by
> p.p_mfgr,
> p.p_type
> order by
> p.p_mfgr,
> p.p_type;
> {code}
>
> h4. Expected Result (without Runtime filter)
> {noformat}
> +-+---++
> | p_mfgr | p_type | num_parts |
> +-+---++
> | Manufacturer#1 | ECONOMY ANODIZED STEEL | 4448 |
> | Manufacturer#1 | ECONOMY BRUSHED STEEL | 4308 |
> | Manufacturer#1 | ECONOMY BURNISHED STEEL | 4268 |
> | Manufacturer#1 | ECONOMY PLATED STEEL | 4266 |
> | Manufacturer#1 | ECONOMY POLISHED STEEL | 4302 |
> | Manufacturer#1 | LARGE ANODIZED STEEL | 4389 |
> | Manufacturer#1 | LARGE BRUSHED STEEL | 4254 |
> | Manufacturer#1 | LARGE BURNISHED STEEL | 4276 |
> | Manufacturer#1 | LARGE PLATED STEEL | 4351 |
> | Manufacturer#1 | LARGE POLISHED STEEL | 4281 |
> | Manufacturer#1 | MEDIUM ANODIZED STEEL | 4230 |
> | Manufacturer#1 | MEDIUM BRUSHED STEEL | 4247 |
> | Manufacturer#1 | MEDIUM BURNISHED STEEL | 4286 |
> | Manufacturer#1 | MEDIUM PLATED STEEL | 4269 |
> | Manufacturer#1 | MEDIUM POLISHED STEEL | 4274 |
> | Manufacturer#1 | PROMO ANODIZED STEEL | 4283 |
> | Manufacturer#1 | PROMO BRUSHED STEEL | 4221 |
> | Manufacturer#1 | PROMO BURNISHED STEEL | 4315 |
> | Manufacturer#1 | PROMO PLATED STEEL | 4361 |
> | Manufacturer#1 | PROMO POLISHED STEEL | 4213 |
> | Manufacturer#1 | SMALL ANODIZED STEEL | 4354 |
> | Manufacturer#1 | SMALL BRUSHED STEEL | 4159 |
> | Manufacturer#1 | SMALL BURNISHED STEEL | 4222 |
> ...
> ...
> 150 rows
> {noformat}
>
> h4. Actual Result
> {noformat}
> +-+---++
> | p_mfgr | p_type | num_parts |
> +-+---++
> | Manufacturer#1 | ECONOMY ANODIZED STEEL | 63 |
> | Manufacturer#1 | ECONOMY BRUSHED STEEL | 64 |
> | Manufacturer#1 | ECONOMY BURNISHED STEEL | 58 |
> | Manufacturer#1 | ECONOMY PLATED STEEL | 73 |
> | Manufacturer#1 | ECONOMY POLISHED STEEL | 59 |
> | Manufacturer#1 | LARGE ANODIZED STEEL | 60 |
> | Manufacturer#1 | LARGE BRUSHED STEEL | 62 |
> | Manufacturer#1 | LARGE BURNISHED STEEL | 47 |
> | Manufacturer#1 | LARGE PLATED STEEL | 51 |
> | Manufacturer#1 | LARGE POLISHED STEEL | 60 |
> | Manufacturer#1 | MEDIUM ANODIZED STEEL | 54 |
> | Manufacturer#1 | MEDIUM BRUSHED STEEL | 60 |
> | Manufacturer#1 | MEDIUM BURNISHED STEEL | 53 |
> | Manufacturer#1 | MEDIUM PLATED STEEL | 73 |
> | Manufacturer#1 | MEDIUM POLISHED STEEL | 65 |
> | Manufacturer#1 | PROMO ANODIZED STEEL | 71 |
> | Manufacturer#1 | PROMO BRUSHED STEEL | 67 |
> | Manufacturer#1 | PROMO BURNISHED STEEL | 64 |
> | Manufacturer#1 | PROMO PLATED STEEL | 47 |
> | Manufacturer#1 | PROMO POLISHED STEEL | 51 |
> | Manufacturer#1 | SMALL ANODIZED STEEL | 48 |
> | Manufacturer#1 | SMALL BRUSHED STEEL | 71 |
> | Manufacturer#1 | SMALL BURNISHED STEEL | 33 |
> ...
> ...
> 150 rows
> {noformat}
>
> h3. Query 2 (TPC-H 7 query)
> {code:sql}
> select
> supp_nation,
> cust_nation,
> l_year,
> sum(volume) as revenue
> from
> (
> select
> n1.n_name as supp_nation,
> n2.n_name as cust_nation,
> extract(year from l.l_shipdate) as l_year,
> l.l_extendedprice * (1 - l.l_discount) as volume
> from
> supplier s,
> lineitem l,
> orders o,
> customer c,
> nation n1,
> nation n2
> where
> s.s_suppkey = l.l_suppkey
> and o.o_orderkey = l.l_orderkey
> and c.c_custkey = o.o_custkey
> and s.s_nationkey = n1.n_nationkey
> and c.c_nationkey = n2.n_nationkey
> and (
> (n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
> or (n1.n_name = 'UNITED STATES' and n2.n_name = 'EGYPT')
> )
> and l.l_shipdate between date
[jira] [Commented] (DRILL-6849) Runtime filter queries with nested broadcast returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779639#comment-16779639
]
Abhishek Ravi commented on DRILL-6849:
--
This issue can be closed as well. Changes for DRILL-7016 fixed the issue.
> Runtime filter queries with nested broadcast returns wrong results
> --
>
> Key: DRILL-6849
> URL: https://issues.apache.org/jira/browse/DRILL-6849
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
>
> Running few queries on TPC-H SF100 data with latest changes in [PR
> #1504|[https://github.com/apache/drill/pull/1504].]
> I noticed that for a couple of queries, there are vast discrepancies in the
> results returned by queries with *Runtime Filter enabled* and without.
> Below are a couple of failing queries.
> h3. Query 1
> {code:sql}
> select
> p.p_mfgr,
> p.p_type,
> count(*) as num_parts
> from
> supplier s,
> part p,
> partsupp ps,
> nation n
> where
> n.n_nationkey = 15
> and n.n_nationkey = s.s_nationkey
> and s.s_suppkey = ps.ps_suppkey
> and ps.ps_partkey = p.p_partkey
> and p.p_type like '%STEEL%'
> group by
> p.p_mfgr,
> p.p_type
> order by
> p.p_mfgr,
> p.p_type;
> {code}
>
> h4. Expected Result (without Runtime filter)
> {noformat}
> +-+---++
> | p_mfgr | p_type | num_parts |
> +-+---++
> | Manufacturer#1 | ECONOMY ANODIZED STEEL | 4448 |
> | Manufacturer#1 | ECONOMY BRUSHED STEEL | 4308 |
> | Manufacturer#1 | ECONOMY BURNISHED STEEL | 4268 |
> | Manufacturer#1 | ECONOMY PLATED STEEL | 4266 |
> | Manufacturer#1 | ECONOMY POLISHED STEEL | 4302 |
> | Manufacturer#1 | LARGE ANODIZED STEEL | 4389 |
> | Manufacturer#1 | LARGE BRUSHED STEEL | 4254 |
> | Manufacturer#1 | LARGE BURNISHED STEEL | 4276 |
> | Manufacturer#1 | LARGE PLATED STEEL | 4351 |
> | Manufacturer#1 | LARGE POLISHED STEEL | 4281 |
> | Manufacturer#1 | MEDIUM ANODIZED STEEL | 4230 |
> | Manufacturer#1 | MEDIUM BRUSHED STEEL | 4247 |
> | Manufacturer#1 | MEDIUM BURNISHED STEEL | 4286 |
> | Manufacturer#1 | MEDIUM PLATED STEEL | 4269 |
> | Manufacturer#1 | MEDIUM POLISHED STEEL | 4274 |
> | Manufacturer#1 | PROMO ANODIZED STEEL | 4283 |
> | Manufacturer#1 | PROMO BRUSHED STEEL | 4221 |
> | Manufacturer#1 | PROMO BURNISHED STEEL | 4315 |
> | Manufacturer#1 | PROMO PLATED STEEL | 4361 |
> | Manufacturer#1 | PROMO POLISHED STEEL | 4213 |
> | Manufacturer#1 | SMALL ANODIZED STEEL | 4354 |
> | Manufacturer#1 | SMALL BRUSHED STEEL | 4159 |
> | Manufacturer#1 | SMALL BURNISHED STEEL | 4222 |
> ...
> ...
> 150 rows
> {noformat}
>
> h4. Actual Result
> {noformat}
> +-+---++
> | p_mfgr | p_type | num_parts |
> +-+---++
> | Manufacturer#1 | ECONOMY ANODIZED STEEL | 63 |
> | Manufacturer#1 | ECONOMY BRUSHED STEEL | 64 |
> | Manufacturer#1 | ECONOMY BURNISHED STEEL | 58 |
> | Manufacturer#1 | ECONOMY PLATED STEEL | 73 |
> | Manufacturer#1 | ECONOMY POLISHED STEEL | 59 |
> | Manufacturer#1 | LARGE ANODIZED STEEL | 60 |
> | Manufacturer#1 | LARGE BRUSHED STEEL | 62 |
> | Manufacturer#1 | LARGE BURNISHED STEEL | 47 |
> | Manufacturer#1 | LARGE PLATED STEEL | 51 |
> | Manufacturer#1 | LARGE POLISHED STEEL | 60 |
> | Manufacturer#1 | MEDIUM ANODIZED STEEL | 54 |
> | Manufacturer#1 | MEDIUM BRUSHED STEEL | 60 |
> | Manufacturer#1 | MEDIUM BURNISHED STEEL | 53 |
> | Manufacturer#1 | MEDIUM PLATED STEEL | 73 |
> | Manufacturer#1 | MEDIUM POLISHED STEEL | 65 |
> | Manufacturer#1 | PROMO ANODIZED STEEL | 71 |
> | Manufacturer#1 | PROMO BRUSHED STEEL | 67 |
> | Manufacturer#1 | PROMO BURNISHED STEEL | 64 |
> | Manufacturer#1 | PROMO PLATED STEEL | 47 |
> | Manufacturer#1 | PROMO POLISHED STEEL | 51 |
> | Manufacturer#1 | SMALL ANODIZED STEEL | 48 |
> | Manufacturer#1 | SMALL BRUSHED STEEL | 71 |
> | Manufacturer#1 | SMALL BURNISHED STEEL | 33 |
> ...
> ...
> 150 rows
> {noformat}
>
> h3. Query 2 (TPC-H 7 query)
> {code:sql}
> select
> supp_nation,
> cust_nation,
> l_year,
> sum(volume) as revenue
> from
> (
> select
> n1.n_name as supp_nation,
> n2.n_name as cust_nation,
> extract(year from l.l_shipdate) as l_year,
> l.l_extendedprice * (1 - l.l_discount) as volume
> from
> supplier s,
> lineitem l,
> orders o,
> customer c,
> nation n1,
> nation n2
> where
> s.s_suppkey = l.l_suppkey
> and o.o_orderkey = l.l_orderkey
> and c.c_custkey = o.o_custkey
> and s.s_nationkey = n1.n_nationkey
> and c.c_nationkey = n2.n_nationkey
> and (
> (n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
> or (n1.n_name =
[jira] [Commented] (DRILL-6871) Enabling runtime filter eliminates more incoming rows than it should.
[
https://issues.apache.org/jira/browse/DRILL-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778442#comment-16778442
]
Abhishek Ravi commented on DRILL-6871:
--
With the resolution of DRILL-7016, I do not see this issue anymore.
> Enabling runtime filter eliminates more incoming rows than it should.
> -
>
> Key: DRILL-6871
> URL: https://issues.apache.org/jira/browse/DRILL-6871
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Kunal Khatua
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 24036b35-7c41-12c9-8c08-9327c8d8fdf2.sys.drill,
> 24036bbf-ede2-690a-93b7-9b43c28eed52.sys.drill,
> 24036c15-54a9-7b5d-07be-4d6de4f63731.sys.drill,
> 24036c22-7f39-ae8b-f8f4-000ecc2572cc.sys.drill
>
>
> When testing with the following combination on TPC-H dataset (scale factor
> 100) using a 4 node setup...
> {code:java}
> exec.hashjoin.bloom_filter.fpp=0.2
> exec.hashjoin.enable.runtime_filter=true
> exec.hashjoin.runtime_filter.max.waiting.time=2
> {code}
> It was observed that the filter eliminates more rows than it should.
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405566 |
> +-+
> 1 row selected (10.565 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, supplier s where l.l_suppkey = s.s_suppkey and s.s_acctbal <1000);
> +-+
> | EXPR$0 |
> +-+
> | 405769 |
> +-+
> 1 row selected (9.845 seconds)
> {code}
> The expected row count for the above (broadcast-join) query should have been
> *109307880*
>
> {code:java}
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 37338355 |
> +---+
> 1 row selected (44.698 seconds)
> 0: jdbc:drill:schema=dfs.par100> select count(*) from (select * from lineitem
> l, orders o where o.o_orderkey = l.l_orderkey and o.o_totalprice < 10);
> +---+
> | EXPR$0 |
> +---+
> | 38044874 |
> +---+
> 1 row selected (44.871 seconds)
> {code}
> The expected row count for the above (hash partition-join) query should have
> been *96176495*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi reassigned DRILL-6855:
Assignee: Abhishek Ravi
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
[
https://issues.apache.org/jira/browse/DRILL-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6998:
-
Description:
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
The query does not return even though Fragment 0:0 reports a state change from
{{RUNNING}} -> {{FINISHED}}
Following is the jstack output of the {{Frag0:0}}.
{noformat}
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
- locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
>From the code, it seems that {{RuntimeFilterSink.close}} is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
logger.error("interrupted while sleeping to wait for the aggregating
worker thread to exit", e);
}
}
{code}
This is because {{AsyncAggregateWorker}} exits due to the following exception,
before it could set asyncAggregateWorker.over is set to *false*.
{noformat}
2019-01-22 16:01:18,773 [drill-executor-1301] ERROR
o.a.d.e.w.filter.RuntimeFilterSink - Failed to aggregate or route the RFW
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.drill.exec.work.filter.RuntimeFilterWritable.unwrap(RuntimeFilterWritable.java:67)
[jira] [Created] (DRILL-6998) Queries failing with "Failed to aggregate or route the RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
Abhishek Ravi created DRILL-6998:
Summary: Queries failing with "Failed to aggregate or route the
RFW" due to "java.lang.ArrayIndexOutOfBoundsException" do not complete
Key: DRILL-6998
URL: https://issues.apache.org/jira/browse/DRILL-6998
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.16.0
Reporter: Abhishek Ravi
Assignee: weijie.tong
Fix For: 1.16.0
Following query joins 2 tables on *two* (>1) fields.
{noformat}
select count(*) from lineitem l inner join partsupp p on l.l_partkey =
p.ps_partkey AND l.l_suppkey = p.ps_suppkey
{noformat}
The query does not return even though Fragment 0:0 reports a state change from
{{RUNNING}} -> {{FINISHED}}
Following is the jstack output of the {{Frag0:0}}.
{noformat}
"23b85137-b102-39a9-70d9-72381c5fb93b:frag:0:0" #16037 daemon prio=10 os_prio=0
tid=0x7f5f48d415d0 nid=0x1a61 waiting on condition [0x7f61b32b2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.drill.exec.work.filter.RuntimeFilterSink.close(RuntimeFilterSink.java:116)
at
org.apache.drill.exec.work.filter.RuntimeFilterRouter.waitForComplete(RuntimeFilterRouter.java:113)
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.wrapUpCompletion(QueryStateProcessor.java:315)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.running(QueryStateProcessor.java:276)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:92)
- locked <0x00055f9a7468> (a
org.apache.drill.exec.work.foreman.QueryStateProcessor)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:349)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.processEvent(QueryStateProcessor.java:342)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor$StateSwitch.addEvent(QueryStateProcessor.java:344)
at
org.apache.drill.exec.work.foreman.QueryStateProcessor.addToEventQueue(QueryStateProcessor.java:155)
at
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:213)
at
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:519)
at
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:483)
at
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
at
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:546)
at
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:63)
at
org.apache.drill.exec.work.batch.ControlMessageHandler.requestFragmentStatus(ControlMessageHandler.java:253)
at
org.apache.drill.exec.rpc.control.LocalControlConnectionManager.runCommand(LocalControlConnectionManager.java:130)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragmentStatus(ControlTunnel.java:89)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:122)
at
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:91)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:367)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:219)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:330)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
>From the code, it seems that {{RuntimeFilterSink.close}} is stuck at
{code:java}
while (!asyncAggregateWorker.over.get()) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
logger.error("interrupted while sleeping to wait for the aggregating
worker thread to exit", e);
}
}
{code}
This is because {{AsyncAggregateWorker}} exits due to the following exception,
before it could set
[jira] [Created] (DRILL-6967) TIMESTAMPDIFF returns incorrect value for SQL_TSI_QUARTER
Abhishek Ravi created DRILL-6967:
Summary: TIMESTAMPDIFF returns incorrect value for SQL_TSI_QUARTER
Key: DRILL-6967
URL: https://issues.apache.org/jira/browse/DRILL-6967
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Fix For: 1.16.0
When checking the fix for DRILL-3610, I noticed that the value returned by
{{TIMESTAMPDIFF}} for {{SQL_TSI_QUARTER}} is incorrect.
For example, consider the following queries on {{orders}} table with TPC-H
SF100 data
Let's get a row from orders table
{noformat}
0: jdbc:drill:drillbits=10.10.100.188> select * from orders limit 1;
+-+++---+--+--+--+-++
| o_orderkey | o_custkey | o_orderstatus | o_totalprice | o_orderdate |
o_orderpriority | o_clerk | o_shippriority |
o_comment |
+-+++---+--+--+--+-++
| 456460071 | 9573185| O | 234213.28 | 1998-03-09 |
4-NOT SPECIFIED | Clerk#65824 | 0 | r deposits. quickly
even ideas haggle flu |
+-+++---+--+--+--+-++
{noformat}
Now let's use {{TIMESTAMPADD}} to get the date 8 quarters / 2 years ago
{noformat}
0: jdbc:drill:drillbits=10.10.100.188> select
cast(TIMESTAMPADD(SQL_TSI_QUARTER,-8,o_orderdate) as DATE) AS quarterdate from
orders where o_orderkey = 456460071;
+--+
| quarterdate |
+--+
| 1996-03-09 |
+--+
{noformat}
So far, so good.
Now let's query the difference between the date in the row and the date
returned by TIMESTAMPADD (a date from 8 quarters ago)
{noformat}
0: jdbc:drill:drillbits=10.10.100.188> select
TIMESTAMPDIFF(SQL_TSI_QUARTER,TO_DATE('1996-03-09','-MM-dd'),o_orderdate)
AS quarterdiff from orders where o_orderkey = 456460071;
+--+
| quarterdiff |
+--+
| 6|
+--+
{noformat}
*6 is incorrect!*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6949) Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition the inner data any further" when Semi join is enabled
[
https://issues.apache.org/jira/browse/DRILL-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6949:
-
Attachment: 23cc1240-74ff-a0c0-8cd5-938fc136e4e2.sys.drill
> Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition
> the inner data any further" when Semi join is enabled
>
>
> Key: DRILL-6949
> URL: https://issues.apache.org/jira/browse/DRILL-6949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
> Attachments: 23cc1240-74ff-a0c0-8cd5-938fc136e4e2.sys.drill,
> 23cc1369-0812-63ce-1861-872636571437.sys.drill
>
>
> Following query fails when with *Error: UNSUPPORTED_OPERATION ERROR:
> Hash-Join can not partition the inner data any further (probably due to too
> many join-key duplicates)* on TPC-H SF100 data.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> select
> count(*)
> from
> lineitem l1
> where
> l1.l_discount IN (
> select
> distinct(cast(l2.l_discount as double))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> {code}
> The subquery contains *distinct* keyword and hence there should not be
> duplicate values.
> I suspect that the failure is caused by semijoin because the query succeeds
> when semijoin is disabled explicitly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6949) Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition the inner data any further" when Semi join is enabled
[
https://issues.apache.org/jira/browse/DRILL-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736526#comment-16736526
]
Abhishek Ravi commented on DRILL-6949:
--
[^23cc1240-74ff-a0c0-8cd5-938fc136e4e2.sys.drill] refers to the same query
when *planner.enable_semijoin* is explicitly set to *false*.
> Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition
> the inner data any further" when Semi join is enabled
>
>
> Key: DRILL-6949
> URL: https://issues.apache.org/jira/browse/DRILL-6949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
> Attachments: 23cc1240-74ff-a0c0-8cd5-938fc136e4e2.sys.drill,
> 23cc1369-0812-63ce-1861-872636571437.sys.drill
>
>
> Following query fails when with *Error: UNSUPPORTED_OPERATION ERROR:
> Hash-Join can not partition the inner data any further (probably due to too
> many join-key duplicates)* on TPC-H SF100 data.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> select
> count(*)
> from
> lineitem l1
> where
> l1.l_discount IN (
> select
> distinct(cast(l2.l_discount as double))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> {code}
> The subquery contains *distinct* keyword and hence there should not be
> duplicate values.
> I suspect that the failure is caused by semijoin because the query succeeds
> when semijoin is disabled explicitly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6949) Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition the inner data any further" when Semi join is enabled
[
https://issues.apache.org/jira/browse/DRILL-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736524#comment-16736524
]
Abhishek Ravi commented on DRILL-6949:
--
[^23cc1369-0812-63ce-1861-872636571437.sys.drill] Is the profile for the
failed query.
> Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition
> the inner data any further" when Semi join is enabled
>
>
> Key: DRILL-6949
> URL: https://issues.apache.org/jira/browse/DRILL-6949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
> Attachments: 23cc1369-0812-63ce-1861-872636571437.sys.drill
>
>
> Following query fails when with *Error: UNSUPPORTED_OPERATION ERROR:
> Hash-Join can not partition the inner data any further (probably due to too
> many join-key duplicates)* on TPC-H SF100 data.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> select
> count(*)
> from
> lineitem l1
> where
> l1.l_discount IN (
> select
> distinct(cast(l2.l_discount as double))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> {code}
> The subquery contains *distinct* keyword and hence there should not be
> duplicate values.
> I suspect that the failure is caused by semijoin because the query succeeds
> when semijoin is disabled explicitly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6949) Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition the inner data any further" when Semi join is enabled
[
https://issues.apache.org/jira/browse/DRILL-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6949:
-
Attachment: 23cc1369-0812-63ce-1861-872636571437.sys.drill
> Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition
> the inner data any further" when Semi join is enabled
>
>
> Key: DRILL-6949
> URL: https://issues.apache.org/jira/browse/DRILL-6949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
> Attachments: 23cc1369-0812-63ce-1861-872636571437.sys.drill
>
>
> Following query fails when with *Error: UNSUPPORTED_OPERATION ERROR:
> Hash-Join can not partition the inner data any further (probably due to too
> many join-key duplicates)* on TPC-H SF100 data.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> select
> count(*)
> from
> lineitem l1
> where
> l1.l_discount IN (
> select
> distinct(cast(l2.l_discount as double))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> {code}
> The subquery contains *distinct* keyword and hence there should not be
> duplicate values.
> I suspect that the failure is caused by semijoin because the query succeeds
> when semijoin is disabled explicitly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6949) Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join can not partition the inner data any further" when Semi join is enabled
Abhishek Ravi created DRILL-6949:
Summary: Query fails with "UNSUPPORTED_OPERATION ERROR: Hash-Join
can not partition the inner data any further" when Semi join is enabled
Key: DRILL-6949
URL: https://issues.apache.org/jira/browse/DRILL-6949
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Following query fails when with *Error: UNSUPPORTED_OPERATION ERROR: Hash-Join
can not partition the inner data any further (probably due to too many join-key
duplicates)* on TPC-H SF100 data.
{code:sql}
set `exec.hashjoin.enable.runtime_filter` = true;
set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
set `planner.enable_broadcast_join` = false;
select
count(*)
from
lineitem l1
where
l1.l_discount IN (
select
distinct(cast(l2.l_discount as double))
from
lineitem l2);
reset `exec.hashjoin.enable.runtime_filter`;
reset `exec.hashjoin.runtime_filter.max.waiting.time`;
reset `planner.enable_broadcast_join`;
{code}
The subquery contains *distinct* keyword and hence there should not be
duplicate values.
I suspect that the failure is caused by semijoin because the query succeeds
when semijoin is disabled explicitly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736508#comment-16736508
]
Abhishek Ravi commented on DRILL-6914:
--
This attachment refers to the query profile for a scenario where semijoin is
*disabled* and runtime filter is *enabled*. The query
succeeds.[^23cc1af3-0e8e-b2c9-a889-a96504988d6c.sys.drill]
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 23cc1af3-0e8e-b2c9-a889-a96504988d6c.sys.drill,
> 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
>
[jira] [Updated] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6914:
-
Attachment: 23cc1af3-0e8e-b2c9-a889-a96504988d6c.sys.drill
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 23cc1af3-0e8e-b2c9-a889-a96504988d6c.sys.drill,
> 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
>
[jira] [Commented] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736502#comment-16736502
]
Abhishek Ravi commented on DRILL-6914:
--
[~ben-zvi] - I have attached the query profile for the failure.
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
>
[jira] [Updated] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6914:
-
Attachment: 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
>
[jira] [Commented] (DRILL-6918) Querying empty topics fails with "NumberFormatException"
[
https://issues.apache.org/jira/browse/DRILL-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733940#comment-16733940
]
Abhishek Ravi commented on DRILL-6918:
--
The PR fixes the issue for empty topic partitions, however, as mentioned
earlier, \{{NumberFormatException}} still exists on non-empty topic partitions
for a \{{select *}} query where predicate field does not exist.
In this case, \{{ensureAtLeastOneField}} / \{{Scan}} does not add the column
but \{{Project}} does. Specifically, \{{ProjectRecordBatch.setupNewSchema}}
adds the column to the container as NullableInteger. The downstream Filter
operator sees this column and thus hits the \{{NumberFormatException}}.
Interestingly, when specific columns are projected (but the predicate still
contains non-existing field), the exception is not hit. First thing I noticed
is that the physical plan is different from \{{select *}} case. There is no
Project between Scan and Filter. Hence the non-existing column is not added to
the schema by the time Filter is processing the records. Filter considers the
field as Nullable VarBinary (this is different from Project), which is why the
issue is not seen.
h3. Physical plan for a select * query
{noformat}
select * from `topic2` where somethingelse <> 'abc'
{noformat}
{noformat}
00-00Screen : rowType = RecordType(DYNAMIC_STAR **): rowcount = 3.0,
cumulative cost = {27.3 rows, 69.3 cpu, 6144.0 io, 0.0 network, 0.0 memory}, id
= 422
00-01 Project(**=[$0]) : rowType = RecordType(DYNAMIC_STAR **): rowcount =
3.0, cumulative cost = {27.0 rows, 69.0 cpu, 6144.0 io, 0.0 network, 0.0
memory}, id = 421
00-02Project(T0¦¦**=[$0]) : rowType = RecordType(DYNAMIC_STAR T0¦¦**):
rowcount = 3.0, cumulative cost = {24.0 rows, 66.0 cpu, 6144.0 io, 0.0 network,
0.0 memory}, id = 420
00-03 SelectionVectorRemover : rowType = RecordType(DYNAMIC_STAR
T0¦¦**, ANY somethingelse): rowcount = 3.0, cumulative cost = {21.0 rows, 63.0
cpu, 6144.0 io, 0.0 network, 0.0 memory}, id = 419
00-04Filter(condition=[<>($1, 'abc')]) : rowType =
RecordType(DYNAMIC_STAR T0¦¦**, ANY somethingelse): rowcount = 3.0, cumulative
cost = {18.0 rows, 60.0 cpu, 6144.0 io, 0.0 network, 0.0 memory}, id = 418
00-05 Project(T0¦¦**=[$0], somethingelse=[$1]) : rowType =
RecordType(DYNAMIC_STAR T0¦¦**, ANY somethingelse): rowcount = 6.0, cumulative
cost = {12.0 rows, 24.0 cpu, 6144.0 io, 0.0 network, 0.0 memory}, id = 417
00-06Scan(table=[[kafka2, topic2]], groupscan=[KafkaGroupScan
[KafkaScanSpec=KafkaScanSpec [topicName=topic2], columns=[`**`,
`somethingelse`]]]) : rowType = RecordType(DYNAMIC_STAR **, ANY somethingelse):
rowcount = 6.0, cumulative cost = {6.0 rows, 12.0 cpu, 6144.0 io, 0.0 network,
0.0 memory}, id = 416
{noformat}
h2. Physical plan for select with specific fields.
{noformat}
select LastName from `topic2` where somethingelse <> 'abc'
{noformat}
{noformat}
00-00Screen : rowType = RecordType(ANY LastName): rowcount = 3.0,
cumulative cost = {21.3 rows, 57.3 cpu, 6144.0 io, 0.0 network, 0.0 memory}, id
= 225
00-01 Project(LastName=[$0]) : rowType = RecordType(ANY LastName):
rowcount = 3.0, cumulative cost = {21.0 rows, 57.0 cpu, 6144.0 io, 0.0 network,
0.0 memory}, id = 224
00-02Project(LastName=[$1]) : rowType = RecordType(ANY LastName):
rowcount = 3.0, cumulative cost = {18.0 rows, 54.0 cpu, 6144.0 io, 0.0 network,
0.0 memory}, id = 223
00-03 SelectionVectorRemover : rowType = RecordType(ANY somethingelse,
ANY LastName): rowcount = 3.0, cumulative cost = {15.0 rows, 51.0 cpu, 6144.0
io, 0.0 network, 0.0 memory}, id = 222
00-04Filter(condition=[<>($0, 'abc')]) : rowType = RecordType(ANY
somethingelse, ANY LastName): rowcount = 3.0, cumulative cost = {12.0 rows,
48.0 cpu, 6144.0 io, 0.0 network, 0.0 memory}, id = 221
00-05 Scan(table=[[kafka2, topic2]], groupscan=[KafkaGroupScan
[KafkaScanSpec=KafkaScanSpec [topicName=topic2], columns=[`somethingelse`,
`LastName`]]]) : rowType = RecordType(ANY somethingelse, ANY LastName):
rowcount = 6.0, cumulative cost = {6.0 rows, 12.0 cpu, 6144.0 io, 0.0 network,
0.0 memory}, id = 220
{noformat}
> Querying empty topics fails with "NumberFormatException"
>
>
> Key: DRILL-6918
> URL: https://issues.apache.org/jira/browse/DRILL-6918
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Kafka
>Affects Versions: 1.14.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Minor
> Fix For: 1.16.0
>
>
> Queries with filter conditions fail with {{NumberFormatException}} when
> querying empty topics.
> {noformat}
> 0: jdbc:drill:drillbit=10.10.100.189> select * from `topic2` where Field1 =
> 'abc';
> Error:
[jira] [Commented] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733544#comment-16733544
]
Abhishek Ravi commented on DRILL-6914:
--
[~ben-zvi] - When I refer to *either option true*, I mean *specifically setting
the other to false*. \{{runtime_filter}} does have some pending issues but this
specific issue is seen when both semijoin and runtime_filter features are
enabled.
Working on something else right now, will re-run and attach the profile for
failed query soon.
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
>
[jira] [Commented] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733347#comment-16733347
]
Abhishek Ravi commented on DRILL-6914:
--
Yes, as mentioned earlier, the issue is seen with SF 100 and ONLY when both
\{{planner.enable_semijoin}} and \{{exec.hashjoin.enable.runtime_filter}}
options are set to \{{true}}. The issue is not seen if either one is disabled.
> Query with RuntimeFilter and SemiJoin fails with IllegalStateException:
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
>
> Following query fails on TPC-H SF 100 dataset when
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
> count(*) as row_count
> from
> lineitem l1
> where
> l1.l_shipdate IN (
> select
> distinct(cast(l2.l_shipdate as date))
> from
> lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
>
[jira] [Commented] (DRILL-6918) Querying empty topics fails with "NumberFormatException"
[
https://issues.apache.org/jira/browse/DRILL-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729894#comment-16729894
]
Abhishek Ravi commented on DRILL-6918:
--
The issue here is that when a column in the predicate, say {{field1}} does not
exist, {{JsonReaderUtils.ensureAtLeastOneField}} would create the column as
Nullable integer (if allTextMode is disabled) and sets the iterOutcome as
{{OK_NEW_SCHEMA}}.
In the *Filter* operator, when the last known outcome is {{OK_NEW_SCHEMA}}, the
{{setupNewSchema}} will throw the exception since the value in the predicate is
not an integer.
One way to get around this issue is by setting {{store.kafka.all_text_mode}} to
{{true}}.
As a fix for the issue seen as a part of the bug, we could wrap the
{{messageReader.ensureAtLeastOneField()}} call in {{KafkaRecordReader.java}}
within {{currentMessageCount > 0}} check. This would handle the case when
topics are empty but we would still hit the NumberFormatException for queries
on non-empty topics where the column in the predicate does not exist.
> Querying empty topics fails with "NumberFormatException"
>
>
> Key: DRILL-6918
> URL: https://issues.apache.org/jira/browse/DRILL-6918
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Kafka
>Affects Versions: 1.14.0
>Reporter: Abhishek Ravi
>Assignee: Abhishek Ravi
>Priority: Minor
> Fix For: 1.16.0
>
>
> Queries with filter conditions fail with {{NumberFormatException}} when
> querying empty topics.
> {noformat}
> 0: jdbc:drill:drillbit=10.10.100.189> select * from `topic2` where Field1 =
> 'abc';
> Error: SYSTEM ERROR: NumberFormatException: abc
> Fragment 0:0
> Please, refer to logs for more information.
> [Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
> (state=,code=0)
> {noformat}
>
> *Logs:*
> {noformat}
> 2018-12-20 22:36:34,576 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23e3760d-7d23-5489-e2fb-6daf383053ee issued by root: select * from `topic2`
> where Field1 = 'abc'
> 2018-12-20 22:36:35,134 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
> o.a.d.e.s.k.KafkaPushDownFilterIntoScan - Partitions ScanSpec before
> pushdown: [KafkaPartitionScanSpec [topicName=topic2, partitionId=2,
> startOffset=0, endOffset=0], KafkaPartitionScanSpec [topicName=topic2,
> partitionId=1, startOffset=0, endOffset=0], KafkaPartitionScanSpec
> [topicName=topic2, partitionId=0, startOffset=0, endOffset=0]]
> 2018-12-20 22:36:35,170 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.k.KafkaScanBatchCreator - Number of record readers initialized : 3
> 2018-12-20 22:36:35,171 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-12-20 22:36:35,172 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State to report: RUNNING
> 2018-12-20 22:36:35,173 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.k.d.MessageReaderFactory - Initialized Message Reader :
> JsonMessageReader[jsonReader=null]
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.store.kafka.MessageIterator - Start offset of topic2:2 is - 0
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.kafka.KafkaRecordReader - Last offset processed for topic2:2 is - 0
> 2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.s.kafka.KafkaRecordReader - Total time to fetch messages from
> topic2:2 is - 0 milliseconds
> 2018-12-20 22:36:35,178 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] WARN
> o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
> `Field1`, returning null instance.
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0: State change requested RUNNING -->
> FAILED
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2
> failed batches
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
> o.a.d.e.p.i.s.RemovingRecordBatch -
> RemovingRecordBatch[container=org.apache.drill.exec.record.VectorContainer@3ce6a91e[recordCount
> = 0, schemaChanged = true, schema = null, wrappers = [], ...], state=FIRST,
> copier=null]
> 2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
>
[jira] [Updated] (DRILL-6918) Querying empty topics fails with "NumberFormatException"
[
https://issues.apache.org/jira/browse/DRILL-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6918:
-
Description:
Queries with filter conditions fail with {{NumberFormatException}} when
querying empty topics.
{noformat}
0: jdbc:drill:drillbit=10.10.100.189> select * from `topic2` where Field1 =
'abc';
Error: SYSTEM ERROR: NumberFormatException: abc
Fragment 0:0
Please, refer to logs for more information.
[Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
(state=,code=0)
{noformat}
*Logs:*
{noformat}
2018-12-20 22:36:34,576 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
23e3760d-7d23-5489-e2fb-6daf383053ee issued by root: select * from `topic2`
where Field1 = 'abc'
2018-12-20 22:36:35,134 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
o.a.d.e.s.k.KafkaPushDownFilterIntoScan - Partitions ScanSpec before pushdown:
[KafkaPartitionScanSpec [topicName=topic2, partitionId=2, startOffset=0,
endOffset=0], KafkaPartitionScanSpec [topicName=topic2, partitionId=1,
startOffset=0, endOffset=0], KafkaPartitionScanSpec [topicName=topic2,
partitionId=0, startOffset=0, endOffset=0]]
2018-12-20 22:36:35,170 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.k.KafkaScanBatchCreator - Number of record readers initialized : 3
2018-12-20 22:36:35,171 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2018-12-20 22:36:35,172 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State to report: RUNNING
2018-12-20 22:36:35,173 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.k.d.MessageReaderFactory - Initialized Message Reader :
JsonMessageReader[jsonReader=null]
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.store.kafka.MessageIterator - Start offset of topic2:2 is - 0
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.kafka.KafkaRecordReader - Last offset processed for topic2:2 is - 0
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.kafka.KafkaRecordReader - Total time to fetch messages from topic2:2
is - 0 milliseconds
2018-12-20 22:36:35,178 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] WARN
o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
`Field1`, returning null instance.
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested RUNNING --> FAILED
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed
batches
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.p.i.s.RemovingRecordBatch -
RemovingRecordBatch[container=org.apache.drill.exec.record.VectorContainer@3ce6a91e[recordCount
= 0, schemaChanged = true, schema = null, wrappers = [], ...], state=FIRST,
copier=null]
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.p.i.filter.FilterRecordBatch -
FilterRecordBatch[container=org.apache.drill.exec.record.VectorContainer@2057ff66[recordCount
= 0, schemaChanged = true, schema = null, wrappers =
[org.apache.drill.exec.vector.NullableIntVector@32edcdf2[field = [`T4¦¦**`
(INT:OPTIONAL)], ...],
org.apache.drill.exec.vector.NullableIntVector@3a5bf582[field = [`Field1`
(INT:OPTIONAL)], ...]], ...], selectionVector2=[SV2: recs=0 - ], filter=null,
popConfig=org.apache.drill.exec.physical.config.Filter@1d69df75]
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump completed.
2018-12-20 22:36:35,192 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested FAILED --> FINISHED
2018-12-20 22:36:35,194 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NumberFormatException: abc
Fragment 0:0
Please, refer to logs for more information.
[Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NumberFormatException: abc
Fragment 0:0
Please, refer to logs for more information.
[Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
[jira] [Created] (DRILL-6918) Querying empty topics fails with "NumberFormatException"
Abhishek Ravi created DRILL-6918:
Summary: Querying empty topics fails with "NumberFormatException"
Key: DRILL-6918
URL: https://issues.apache.org/jira/browse/DRILL-6918
Project: Apache Drill
Issue Type: Bug
Components: Storage - Kafka
Affects Versions: 1.14.0
Reporter: Abhishek Ravi
Assignee: Abhishek Ravi
Fix For: 1.16.0
Queries with filter conditions fail with {{NumberFormatException}} when
querying empty topics.
{noformat}
2018-12-20 22:36:34,576 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
23e3760d-7d23-5489-e2fb-6daf383053ee issued by root: select * from `topic2`
where Field1 = 'abc'
2018-12-20 22:36:35,134 [23e3760d-7d23-5489-e2fb-6daf383053ee:foreman] INFO
o.a.d.e.s.k.KafkaPushDownFilterIntoScan - Partitions ScanSpec before pushdown:
[KafkaPartitionScanSpec [topicName=topic2, partitionId=2, startOffset=0,
endOffset=0], KafkaPartitionScanSpec [topicName=topic2, partitionId=1,
startOffset=0, endOffset=0], KafkaPartitionScanSpec [topicName=topic2,
partitionId=0, startOffset=0, endOffset=0]]
2018-12-20 22:36:35,170 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.k.KafkaScanBatchCreator - Number of record readers initialized : 3
2018-12-20 22:36:35,171 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2018-12-20 22:36:35,172 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State to report: RUNNING
2018-12-20 22:36:35,173 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.k.d.MessageReaderFactory - Initialized Message Reader :
JsonMessageReader[jsonReader=null]
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.store.kafka.MessageIterator - Start offset of topic2:2 is - 0
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.kafka.KafkaRecordReader - Last offset processed for topic2:2 is - 0
2018-12-20 22:36:35,177 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.s.kafka.KafkaRecordReader - Total time to fetch messages from topic2:2
is - 0 milliseconds
2018-12-20 22:36:35,178 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] WARN
o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path
`Field1`, returning null instance.
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested RUNNING --> FAILED
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed
batches
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.p.i.s.RemovingRecordBatch -
RemovingRecordBatch[container=org.apache.drill.exec.record.VectorContainer@3ce6a91e[recordCount
= 0, schemaChanged = true, schema = null, wrappers = [], ...], state=FIRST,
copier=null]
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.p.i.filter.FilterRecordBatch -
FilterRecordBatch[container=org.apache.drill.exec.record.VectorContainer@2057ff66[recordCount
= 0, schemaChanged = true, schema = null, wrappers =
[org.apache.drill.exec.vector.NullableIntVector@32edcdf2[field = [`T4¦¦**`
(INT:OPTIONAL)], ...],
org.apache.drill.exec.vector.NullableIntVector@3a5bf582[field = [`Field1`
(INT:OPTIONAL)], ...]], ...], selectionVector2=[SV2: recs=0 - ], filter=null,
popConfig=org.apache.drill.exec.physical.config.Filter@1d69df75]
2018-12-20 22:36:35,191 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump completed.
2018-12-20 22:36:35,192 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 23e3760d-7d23-5489-e2fb-6daf383053ee:0:0:
State change requested FAILED --> FINISHED
2018-12-20 22:36:35,194 [23e3760d-7d23-5489-e2fb-6daf383053ee:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NumberFormatException: abc
Fragment 0:0
Please, refer to logs for more information.
[Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NumberFormatException: abc
Fragment 0:0
Please, refer to logs for more information.
[Error Id: a0718456-c053-4820-9bd8-69c683598344 on qa-node189.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
at
[jira] [Resolved] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi resolved DRILL-6838.
--
Resolution: Fixed
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi closed DRILL-6838.
Resolution: Fixed
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi reopened DRILL-6838:
--
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725494#comment-16725494
]
Abhishek Ravi commented on DRILL-6838:
--
The issue seems to have come back because of semi join changes as a part of
DRILL-6798. I have opened a separate bug DRILL-6914 since both RuntimeFilter
and SemiJoin have to be enabled to hit the memory leak. Closing this bug.
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query
Abhishek Ravi created DRILL-6914:
Summary: Query with RuntimeFilter and SemiJoin fails with
IllegalStateException: Memory was leaked by query
Key: DRILL-6914
URL: https://issues.apache.org/jira/browse/DRILL-6914
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Fix For: 1.16.0
Following query fails on TPC-H SF 100 dataset when
exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
Note that the query does not fail if any one of them or both are disabled.
{code:sql}
set `exec.hashjoin.enable.runtime_filter` = true;
set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
set `planner.enable_broadcast_join` = false;
set `planner.enable_semijoin` = true;
select
count(*) as row_count
from
lineitem l1
where
l1.l_shipdate IN (
select
distinct(cast(l2.l_shipdate as date))
from
lineitem l2);
reset `exec.hashjoin.enable.runtime_filter`;
reset `exec.hashjoin.runtime_filter.max.waiting.time`;
reset `planner.enable_broadcast_join`;
reset `planner.enable_semijoin`;
{code}
{noformat}
Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory
leaked: (134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
(state=,code=0)
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at sqlline.BufferedRows.(BufferedRows.java:37)
at sqlline.SqlLine.print(SqlLine.java:1716)
at sqlline.Commands.execute(Commands.java:949)
at sqlline.Commands.sql(Commands.java:882)
at sqlline.SqlLine.dispatch(SqlLine.java:725)
at sqlline.SqlLine.runCommands(SqlLine.java:1779)
at sqlline.Commands.run(Commands.java:1485)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:722)
at sqlline.SqlLine.initArgs(SqlLine.java:458)
at sqlline.SqlLine.begin(SqlLine.java:514)
at sqlline.SqlLine.start(SqlLine.java:264)
at sqlline.SqlLine.main(SqlLine.java:195)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(134217728)
Allocator(frag:1:0) 800/134217728/172453568/70126322567
(res/actual/peak/limit)
Fragment 1:0
Please, refer to logs for more information.
[Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
[jira] [Reopened] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi reopened DRILL-6838:
--
[~weijie] - I am hitting the memory leak again on the latest drill master.
The issue not only seen in queries with deeply nested probe side, but also seen
on queries that just have simple Hash Join.
Not only the 2 queries mentioned earlier fail but 2 additional failures.
- [Simple Hash Join on date type
field|https://github.com/mapr/drill-test-framework/blob/master/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/hash_join_date.sql]
- [Simple Hash Join on double type
field|https://github.com/mapr/drill-test-framework/blob/master/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/hash_join_double.sql]
I hit this issue on TPC-H SF 100 dataset
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692375#comment-16692375
]
Abhishek Ravi commented on DRILL-6855:
--
The {{IOException }}message is as follows
{noformat}
java.io.IOException: Error getting user info for current user,
{noformat}
I think the changes for this bug should be something very similar to
https://issues.apache.org/jira/browse/DRILL-5704, PR ->
https://github.com/apache/drill/pull/895/files
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692375#comment-16692375
]
Abhishek Ravi edited comment on DRILL-6855 at 11/19/18 10:38 PM:
-
The {{IOException}} message is as follows
{noformat}
java.io.IOException: Error getting user info for current user,
{noformat}
I think the changes for this bug should be something very similar to
https://issues.apache.org/jira/browse/DRILL-5704, PR ->
[https://github.com/apache/drill/pull/895/files]
was (Author: aravi5):
The {{IOException }}message is as follows
{noformat}
java.io.IOException: Error getting user info for current user,
{noformat}
I think the changes for this bug should be something very similar to
https://issues.apache.org/jira/browse/DRILL-5704, PR ->
https://github.com/apache/drill/pull/895/files
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687669#comment-16687669
]
Abhishek Ravi commented on DRILL-6855:
--
[~arina] , [~kkhatua] - would be great to get your thoughts on this.
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6855:
-
Description:
Query from a *proxy user* fails with following error when *impersonation* is
*enabled* but user does not exist. This behaviour was discovered when running
Drill on MapR.
{noformat}
Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
root schema or current default schema.
Current default schema: No default schema selected
{noformat}
The above error is very confusing and made it very hard to relate to proxy user
does not exist + impersonation issue.
The {{fs.access(wsPath, FsAction.READ)}} in {{WorkspaceSchemaFactory.accessible
fails with IOException,}} which is not handled in {{accessible}} but in
{{DynamicRootSchema.loadSchemaFactory}}. At this point none of the schemas are
registered and hence the root schema will be registered as default schema.
The query execution continues and fails much ahead at
{{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}} eventually
throws {{SchemaUtilites.throwSchemaNotFoundException}}.
One possible fix could be to handle {{IOException}} similar to
{{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
was:
Query from a *proxy user* fails with following error when *impersonation* is
*enabled* but user does not exist. This behaviour was discovered when running
Drill on MapR.
{noformat}
Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
root schema or current default schema.
Current default schema: No default schema selected
{noformat}
The above error is very confusing and made it very hard to relate to proxy user
does not exist + impersonation issue.
The {{fs.access(wsPath, FsAction.READ)}} in {{WorkspaceSchemaFactory.accessible
}}fails with {{IOException,}} which is not handled in {{accessible}} but in
{{DynamicRootSchema.loadSchemaFactory}}. At this point none of the schemas are
registered and hence the root schema will be registered as default schema.
The query execution continues and fails much ahead at
{{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}} eventually
throws {{SchemaUtilites.throwSchemaNotFoundException}}.
One possible fix could be to handle {{IOException}} similar to
{{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible fails with IOException,}} which is not
> handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}. At
> this point none of the schemas are registered and hence the root schema will
> be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
[
https://issues.apache.org/jira/browse/DRILL-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6855:
-
Description:
Query from a *proxy user* fails with following error when *impersonation* is
*enabled* but user does not exist. This behaviour was discovered when running
Drill on MapR.
{noformat}
Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
root schema or current default schema.
Current default schema: No default schema selected
{noformat}
The above error is very confusing and made it very hard to relate to proxy user
does not exist + impersonation issue.
The {{fs.access(wsPath, FsAction.READ)}} in {{WorkspaceSchemaFactory.accessible
}}fails with {{IOException,}} which is not handled in {{accessible}} but in
{{DynamicRootSchema.loadSchemaFactory}}. At this point none of the schemas are
registered and hence the root schema will be registered as default schema.
The query execution continues and fails much ahead at
{{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}} eventually
throws {{SchemaUtilites.throwSchemaNotFoundException}}.
One possible fix could be to handle {{IOException}} similar to
{{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
was:
Query from a *proxy user* fails with following error when *impersonation* is
*enabled* but user does not exist. This behaviour was discovered when running
Drill on MapR.
{noformat}
Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
root schema or current default schema.
Current default schema: No default schema selected
{noformat}
The above error is very confusing and made it very hard to relate to proxy user
does not exist + impersonation issue.
The {{fs.access(wsPath, FsAction.READ)}} in
{{WorkspaceSchemaFactory.accessible}}
fails with {{IOException,}} which is not handled in {{accessible}} but in
{{DynamicRootSchema.loadSchemaFactory}}. At this point none of the schemas are
registered and hence the root schema will be registered as default schema.
The query execution continues and fails much ahead at
{{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}} eventually
throws {{SchemaUtilites.throwSchemaNotFoundException}}.
One possible fix could be to handle {{IOException}} similar to
{{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
> Query from non-existent proxy user fails with "No default schema selected"
> when impersonation is enabled
>
>
> Key: DRILL-6855
> URL: https://issues.apache.org/jira/browse/DRILL-6855
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Priority: Major
>
> Query from a *proxy user* fails with following error when *impersonation* is
> *enabled* but user does not exist. This behaviour was discovered when running
> Drill on MapR.
> {noformat}
> Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
> root schema or current default schema.
> Current default schema: No default schema selected
> {noformat}
> The above error is very confusing and made it very hard to relate to proxy
> user does not exist + impersonation issue.
> The {{fs.access(wsPath, FsAction.READ)}} in
> {{WorkspaceSchemaFactory.accessible }}fails with {{IOException,}} which is
> not handled in {{accessible}} but in {{DynamicRootSchema.loadSchemaFactory}}.
> At this point none of the schemas are registered and hence the root schema
> will be registered as default schema.
> The query execution continues and fails much ahead at
> {{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}}
> eventually throws {{SchemaUtilites.throwSchemaNotFoundException}}.
> One possible fix could be to handle {{IOException}} similar to
> {{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6855) Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled
Abhishek Ravi created DRILL-6855:
Summary: Query from non-existent proxy user fails with "No default
schema selected" when impersonation is enabled
Key: DRILL-6855
URL: https://issues.apache.org/jira/browse/DRILL-6855
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Query from a *proxy user* fails with following error when *impersonation* is
*enabled* but user does not exist. This behaviour was discovered when running
Drill on MapR.
{noformat}
Error: VALIDATION ERROR: Schema [[dfs]] is not valid with respect to either
root schema or current default schema.
Current default schema: No default schema selected
{noformat}
The above error is very confusing and made it very hard to relate to proxy user
does not exist + impersonation issue.
The {{fs.access(wsPath, FsAction.READ)}} in
{{WorkspaceSchemaFactory.accessible}}
fails with {{IOException,}} which is not handled in {{accessible}} but in
{{DynamicRootSchema.loadSchemaFactory}}. At this point none of the schemas are
registered and hence the root schema will be registered as default schema.
The query execution continues and fails much ahead at
{{DrillSqlWorker.getQueryPlan}} where the {{SqlConverter.validate}} eventually
throws {{SchemaUtilites.throwSchemaNotFoundException}}.
One possible fix could be to handle {{IOException}} similar to
{{FileNotFoundException}} in {{WorkspaceSchemaFactory.accessible}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6849) Runtime filter queries with nested broadcast returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6849:
-
Description:
Running few queries on TPC-H SF100 data with latest changes in [PR
#1504|[https://github.com/apache/drill/pull/1504].]
I noticed that for a couple of queries, there are vast discrepancies in the
results returned by queries with *Runtime Filter enabled* and without.
Below are a couple of failing queries.
h3. Query 1
{code:sql}
select
p.p_mfgr,
p.p_type,
count(*) as num_parts
from
supplier s,
part p,
partsupp ps,
nation n
where
n.n_nationkey = 15
and n.n_nationkey = s.s_nationkey
and s.s_suppkey = ps.ps_suppkey
and ps.ps_partkey = p.p_partkey
and p.p_type like '%STEEL%'
group by
p.p_mfgr,
p.p_type
order by
p.p_mfgr,
p.p_type;
{code}
h4. Expected Result (without Runtime filter)
{noformat}
+-+---++
| p_mfgr | p_type | num_parts |
+-+---++
| Manufacturer#1 | ECONOMY ANODIZED STEEL | 4448 |
| Manufacturer#1 | ECONOMY BRUSHED STEEL | 4308 |
| Manufacturer#1 | ECONOMY BURNISHED STEEL | 4268 |
| Manufacturer#1 | ECONOMY PLATED STEEL | 4266 |
| Manufacturer#1 | ECONOMY POLISHED STEEL | 4302 |
| Manufacturer#1 | LARGE ANODIZED STEEL | 4389 |
| Manufacturer#1 | LARGE BRUSHED STEEL | 4254 |
| Manufacturer#1 | LARGE BURNISHED STEEL | 4276 |
| Manufacturer#1 | LARGE PLATED STEEL | 4351 |
| Manufacturer#1 | LARGE POLISHED STEEL | 4281 |
| Manufacturer#1 | MEDIUM ANODIZED STEEL | 4230 |
| Manufacturer#1 | MEDIUM BRUSHED STEEL | 4247 |
| Manufacturer#1 | MEDIUM BURNISHED STEEL | 4286 |
| Manufacturer#1 | MEDIUM PLATED STEEL | 4269 |
| Manufacturer#1 | MEDIUM POLISHED STEEL | 4274 |
| Manufacturer#1 | PROMO ANODIZED STEEL | 4283 |
| Manufacturer#1 | PROMO BRUSHED STEEL | 4221 |
| Manufacturer#1 | PROMO BURNISHED STEEL | 4315 |
| Manufacturer#1 | PROMO PLATED STEEL | 4361 |
| Manufacturer#1 | PROMO POLISHED STEEL | 4213 |
| Manufacturer#1 | SMALL ANODIZED STEEL | 4354 |
| Manufacturer#1 | SMALL BRUSHED STEEL | 4159 |
| Manufacturer#1 | SMALL BURNISHED STEEL | 4222 |
...
...
150 rows
{noformat}
h4. Actual Result
{noformat}
+-+---++
| p_mfgr | p_type | num_parts |
+-+---++
| Manufacturer#1 | ECONOMY ANODIZED STEEL | 63 |
| Manufacturer#1 | ECONOMY BRUSHED STEEL | 64 |
| Manufacturer#1 | ECONOMY BURNISHED STEEL | 58 |
| Manufacturer#1 | ECONOMY PLATED STEEL | 73 |
| Manufacturer#1 | ECONOMY POLISHED STEEL | 59 |
| Manufacturer#1 | LARGE ANODIZED STEEL | 60 |
| Manufacturer#1 | LARGE BRUSHED STEEL | 62 |
| Manufacturer#1 | LARGE BURNISHED STEEL | 47 |
| Manufacturer#1 | LARGE PLATED STEEL | 51 |
| Manufacturer#1 | LARGE POLISHED STEEL | 60 |
| Manufacturer#1 | MEDIUM ANODIZED STEEL | 54 |
| Manufacturer#1 | MEDIUM BRUSHED STEEL | 60 |
| Manufacturer#1 | MEDIUM BURNISHED STEEL | 53 |
| Manufacturer#1 | MEDIUM PLATED STEEL | 73 |
| Manufacturer#1 | MEDIUM POLISHED STEEL | 65 |
| Manufacturer#1 | PROMO ANODIZED STEEL | 71 |
| Manufacturer#1 | PROMO BRUSHED STEEL | 67 |
| Manufacturer#1 | PROMO BURNISHED STEEL | 64 |
| Manufacturer#1 | PROMO PLATED STEEL | 47 |
| Manufacturer#1 | PROMO POLISHED STEEL | 51 |
| Manufacturer#1 | SMALL ANODIZED STEEL | 48 |
| Manufacturer#1 | SMALL BRUSHED STEEL | 71 |
| Manufacturer#1 | SMALL BURNISHED STEEL | 33 |
...
...
150 rows
{noformat}
h3. Query 2 (TPC-H 7 query)
{code:sql}
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l.l_shipdate) as l_year,
l.l_extendedprice * (1 - l.l_discount) as volume
from
supplier s,
lineitem l,
orders o,
customer c,
nation n1,
nation n2
where
s.s_suppkey = l.l_suppkey
and o.o_orderkey = l.l_orderkey
and c.c_custkey = o.o_custkey
and s.s_nationkey = n1.n_nationkey
and c.c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
or (n1.n_name = 'UNITED STATES' and n2.n_name = 'EGYPT')
)
and l.l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year;
{code}
h4. Expected Result
{noformat}
EGYPT UNITED STATES 1996 5.2735809932832E9
UNITED STATES EGYPT 1996 5.320488357330402E9
EGYPT UNITED STATES 1995 5.282512709079098E9
UNITED STATES EGYPT 1995 5.3213732978949E9
{noformat}
h4. Actual Result
{noformat}
EGYPT UNITED STATES 1996 1.628296170457E7
UNITED STATES EGYPT 1996 1.39809230059E7
EGYPT UNITED STATES 1995 1.360689552253E7
UNITED STATES EGYPT 1995 1.524104447329E7
{noformat}
was:
Running few queries on TPC-H SF100 data with latest changes in [PR
#1504|[https://github.com/apache/drill/pull/1504].]
I noticed that for a couple of queries, there are vast
[jira] [Created] (DRILL-6849) Runtime filter queries with nested broadcast returns wrong results
Abhishek Ravi created DRILL-6849:
Summary: Runtime filter queries with nested broadcast returns
wrong results
Key: DRILL-6849
URL: https://issues.apache.org/jira/browse/DRILL-6849
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Abhishek Ravi
Assignee: weijie.tong
Fix For: 1.15.0
Running few queries on TPC-H SF100 data with latest changes in [PR
#1504|[https://github.com/apache/drill/pull/1504].]
I noticed that for a couple of queries, there are vast discrepancies in the
results returned by queries with *Runtime Filter enabled* and without.
Below are a couple of failing queries.
h3. h3. Query 1
{code:sql}
select
p.p_mfgr,
p.p_type,
count(*) as num_parts
from
supplier s,
part p,
partsupp ps,
nation n
where
n.n_nationkey = 15
and n.n_nationkey = s.s_nationkey
and s.s_suppkey = ps.ps_suppkey
and ps.ps_partkey = p.p_partkey
and p.p_type like '%STEEL%'
group by
p.p_mfgr,
p.p_type
order by
p.p_mfgr,
p.p_type;
{code}
h4. h4. Expected Result (without Runtime filter)
{noformat}
+-+---++
| p_mfgr | p_type | num_parts |
+-+---++
| Manufacturer#1 | ECONOMY ANODIZED STEEL | 4448 |
| Manufacturer#1 | ECONOMY BRUSHED STEEL | 4308 |
| Manufacturer#1 | ECONOMY BURNISHED STEEL | 4268 |
| Manufacturer#1 | ECONOMY PLATED STEEL | 4266 |
| Manufacturer#1 | ECONOMY POLISHED STEEL | 4302 |
| Manufacturer#1 | LARGE ANODIZED STEEL | 4389 |
| Manufacturer#1 | LARGE BRUSHED STEEL | 4254 |
| Manufacturer#1 | LARGE BURNISHED STEEL | 4276 |
| Manufacturer#1 | LARGE PLATED STEEL | 4351 |
| Manufacturer#1 | LARGE POLISHED STEEL | 4281 |
| Manufacturer#1 | MEDIUM ANODIZED STEEL | 4230 |
| Manufacturer#1 | MEDIUM BRUSHED STEEL | 4247 |
| Manufacturer#1 | MEDIUM BURNISHED STEEL | 4286 |
| Manufacturer#1 | MEDIUM PLATED STEEL | 4269 |
| Manufacturer#1 | MEDIUM POLISHED STEEL | 4274 |
| Manufacturer#1 | PROMO ANODIZED STEEL | 4283 |
| Manufacturer#1 | PROMO BRUSHED STEEL | 4221 |
| Manufacturer#1 | PROMO BURNISHED STEEL | 4315 |
| Manufacturer#1 | PROMO PLATED STEEL | 4361 |
| Manufacturer#1 | PROMO POLISHED STEEL | 4213 |
| Manufacturer#1 | SMALL ANODIZED STEEL | 4354 |
| Manufacturer#1 | SMALL BRUSHED STEEL | 4159 |
| Manufacturer#1 | SMALL BURNISHED STEEL | 4222 |
...
...
150 rows
{noformat}
h4. h4. Actual Result
{noformat}
+-+---++
| p_mfgr | p_type | num_parts |
+-+---++
| Manufacturer#1 | ECONOMY ANODIZED STEEL | 63 |
| Manufacturer#1 | ECONOMY BRUSHED STEEL | 64 |
| Manufacturer#1 | ECONOMY BURNISHED STEEL | 58 |
| Manufacturer#1 | ECONOMY PLATED STEEL | 73 |
| Manufacturer#1 | ECONOMY POLISHED STEEL | 59 |
| Manufacturer#1 | LARGE ANODIZED STEEL | 60 |
| Manufacturer#1 | LARGE BRUSHED STEEL | 62 |
| Manufacturer#1 | LARGE BURNISHED STEEL | 47 |
| Manufacturer#1 | LARGE PLATED STEEL | 51 |
| Manufacturer#1 | LARGE POLISHED STEEL | 60 |
| Manufacturer#1 | MEDIUM ANODIZED STEEL | 54 |
| Manufacturer#1 | MEDIUM BRUSHED STEEL | 60 |
| Manufacturer#1 | MEDIUM BURNISHED STEEL | 53 |
| Manufacturer#1 | MEDIUM PLATED STEEL | 73 |
| Manufacturer#1 | MEDIUM POLISHED STEEL | 65 |
| Manufacturer#1 | PROMO ANODIZED STEEL | 71 |
| Manufacturer#1 | PROMO BRUSHED STEEL | 67 |
| Manufacturer#1 | PROMO BURNISHED STEEL | 64 |
| Manufacturer#1 | PROMO PLATED STEEL | 47 |
| Manufacturer#1 | PROMO POLISHED STEEL | 51 |
| Manufacturer#1 | SMALL ANODIZED STEEL | 48 |
| Manufacturer#1 | SMALL BRUSHED STEEL | 71 |
| Manufacturer#1 | SMALL BURNISHED STEEL | 33 |
...
...
150 rows
{noformat}
h3. h3. Query 2 (TPC-H 7 query)
{code:sql}
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l.l_shipdate) as l_year,
l.l_extendedprice * (1 - l.l_discount) as volume
from
supplier s,
lineitem l,
orders o,
customer c,
nation n1,
nation n2
where
s.s_suppkey = l.l_suppkey
and o.o_orderkey = l.l_orderkey
and c.c_custkey = o.o_custkey
and s.s_nationkey = n1.n_nationkey
and c.c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
or (n1.n_name = 'UNITED STATES' and n2.n_name = 'EGYPT')
)
and l.l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year;
{code}
h4. h4. Expected Result
{noformat}
EGYPT UNITED STATES 1996 5.2735809932832E9
UNITED STATES EGYPT 1996 5.320488357330402E9
EGYPT UNITED STATES 1995 5.282512709079098E9
UNITED STATES EGYPT 1995 5.3213732978949E9
{noformat}
h4. h4. Actual Result
{noformat}
EGYPT UNITED STATES 1996 1.628296170457E7
UNITED STATES
[jira] [Comment Edited] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi edited comment on DRILL-6838 at 11/11/18 11:56 PM:
-
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi commented on DRILL-6838:
--
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h1. h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Comment Edited] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi edited comment on DRILL-6838 at 11/11/18 11:55 PM:
-
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Updated] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi updated DRILL-6838:
-
Component/s: (was: Execution - Relational Operators)
Execution - Flow
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Priority: Major
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
Abhishek Ravi created DRILL-6838:
Summary: Query with Runtime Filter fails with
IllegalStateException: Memory was leaked by query
Key: DRILL-6838
URL: https://issues.apache.org/jira/browse/DRILL-6838
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.15.0
Environment: Drill 1.15 + PR ->
[https://github.com/apache/drill/pull/1504]
Reporter: Abhishek Ravi
Attachments: RuntimeFilterMemoryLeak.log
When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
1.15.0 SNAPSHOT, I saw failures with following error:
{noformat}
Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory
leaked: (1048576)
Allocator(frag:5:38) 600/1048576/77833920/50062914560
(res/actual/peak/limit)
Fragment 5:38
Please, refer to logs for more information.
[Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
(state=,code=0)
{noformat}
The failure is observed when the query has nested joins. (TPC-H query 7 can
easily reproduce the issue). I have attached logs for a simpler sample query
that fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6733) Unit tests from KafkaFilterPushdownTest are failing in some environments.
[
https://issues.apache.org/jira/browse/DRILL-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Ravi reassigned DRILL-6733:
Assignee: Abhishek Ravi
> Unit tests from KafkaFilterPushdownTest are failing in some environments.
> -
>
> Key: DRILL-6733
> URL: https://issues.apache.org/jira/browse/DRILL-6733
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Anton Gozhiy
>Assignee: Abhishek Ravi
>Priority: Major
>
> *Steps:*
> # Build the Drill project without skipping the unit tests:
> {noformat}
> mvn clean install
> {noformat}
> Alternatively, if the project was already built, run tests for Kafka:
> {noformat}
> mvn test -pl contrib/storage-kafka
> {noformat}
> *Expected results:*
> All tests are passed.
> *Actual results:*
> Tests from KafkaFilterPushdownTest are failing:
> {noformat}
> ---
> T E S T S
> ---
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
> -1,283,514.348 sec - in org.apache.drill.exec.store.kafka.MessageIteratorTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
> -1,283,513.783 sec - in org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: -1,283,512.35
> sec - in org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
> Running org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec -
> in org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Running org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.2 sec -
> in org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Running org.apache.drill.exec.store.kafka.MessageIteratorTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.036 sec -
> in org.apache.drill.exec.store.kafka.MessageIteratorTest
> Running org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.611 sec -
> in org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Running org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
> 13:09:29.511 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 213
> B(139.3 KiB), h: 20.0 MiB(719.0 MiB), nh: 794.5 KiB(120.1 MiB)):
> testPushdownWithOr(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
> java.lang.AssertionError: expected:<26> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.runKafkaSQLVerifyCount(KafkaTestBase.java:69)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest.testPushdownWithOr(KafkaFilterPushdownTest.java:259)
> ~[test-classes/:na]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> 13:09:33.307 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 377
> B(139.7 KiB), h: 18.5 MiB(743.2 MiB), nh: 699.5 KiB(120.9 MiB)):
> testPushdownWithAndOrCombo2(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
>
> java.lang.AssertionError: expected:<4> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.runKafkaSQLVerifyCount(KafkaTestBase.java:69)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest.testPushdownWithAndOrCombo2(KafkaFilterPushdownTest.java:316)
> ~[test-classes/:na]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> 13:09:44.424 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(139.7 KiB), h: 11.7 MiB(774.6 MiB), nh: 537.1 KiB(122.3 MiB)):
> testPushdownOnTimestamp(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
>
> java.lang.AssertionError: expected:<20> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.runKafkaSQLVerifyCount(KafkaTestBase.java:69)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest.testPushdownOnTimestamp(KafkaFilterPushdownTest.java:92)
> ~[test-classes/:na]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> 13:09:48.162 [main] ERROR org.apache.drill.TestReporter -
[jira] [Commented] (DRILL-6733) Unit tests from KafkaFilterPushdownTest are failing in some environments.
[
https://issues.apache.org/jira/browse/DRILL-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608596#comment-16608596
]
Abhishek Ravi commented on DRILL-6733:
--
{{EmbeddedKafkaCluster}} is initialized in {{TestKafkaSuit}}. It could be that
the messages expired by the time query was executed. I was able to reproduce
the issue by adding a delay in {{KafkaFilterPushdownTest.setup.}}
To test queries with timestamp, topic is created with {{CREATE_TIME.}}
Solution is to set \{{retention.ms}} to -1 for the topic. This will ensure that
the messages produced will stay forever.
> Unit tests from KafkaFilterPushdownTest are failing in some environments.
> -
>
> Key: DRILL-6733
> URL: https://issues.apache.org/jira/browse/DRILL-6733
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Anton Gozhiy
>Priority: Major
>
> *Steps:*
> # Build the Drill project without skipping the unit tests:
> {noformat}
> mvn clean install
> {noformat}
> Alternatively, if the project was already built, run tests for Kafka:
> {noformat}
> mvn test -pl contrib/storage-kafka
> {noformat}
> *Expected results:*
> All tests are passed.
> *Actual results:*
> Tests from KafkaFilterPushdownTest are failing:
> {noformat}
> ---
> T E S T S
> ---
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
> -1,283,514.348 sec - in org.apache.drill.exec.store.kafka.MessageIteratorTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
> -1,283,513.783 sec - in org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: -1,283,512.35
> sec - in org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
> Running org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec -
> in org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Running org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.2 sec -
> in org.apache.drill.exec.store.kafka.KafkaQueriesTest
> Running org.apache.drill.exec.store.kafka.MessageIteratorTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.036 sec -
> in org.apache.drill.exec.store.kafka.MessageIteratorTest
> Running org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.611 sec -
> in org.apache.drill.exec.store.kafka.decoders.MessageReaderFactoryTest
> Running org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
> 13:09:29.511 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 213
> B(139.3 KiB), h: 20.0 MiB(719.0 MiB), nh: 794.5 KiB(120.1 MiB)):
> testPushdownWithOr(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
> java.lang.AssertionError: expected:<26> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.runKafkaSQLVerifyCount(KafkaTestBase.java:69)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest.testPushdownWithOr(KafkaFilterPushdownTest.java:259)
> ~[test-classes/:na]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> 13:09:33.307 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 377
> B(139.7 KiB), h: 18.5 MiB(743.2 MiB), nh: 699.5 KiB(120.9 MiB)):
> testPushdownWithAndOrCombo2(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
>
> java.lang.AssertionError: expected:<4> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.runKafkaSQLVerifyCount(KafkaTestBase.java:69)
> ~[test-classes/:na]
> at
> org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest.testPushdownWithAndOrCombo2(KafkaFilterPushdownTest.java:316)
> ~[test-classes/:na]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> 13:09:44.424 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0
> B(139.7 KiB), h: 11.7 MiB(774.6 MiB), nh: 537.1 KiB(122.3 MiB)):
> testPushdownOnTimestamp(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
>
> java.lang.AssertionError: expected:<20> but was:<0>
> at
> org.apache.drill.exec.store.kafka.KafkaTestBase.logResultAndVerifyRowCount(KafkaTestBase.java:76)
> ~[test-classes/:na]
> at
>
[jira] [Comment Edited] (DRILL-6625) Intermittent failures in Kafka unit tests
[
https://issues.apache.org/jira/browse/DRILL-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604682#comment-16604682
]
Abhishek Ravi edited comment on DRILL-6625 at 9/5/18 5:11 PM:
--
Had some time to dig further into this issue. [~ben-zvi] - mentioned that the
issues are seen when running on Mac and they seem to be intermittent failures.
h3. h3. Issue 1 - "{{The server disconnected before a response was received"}}
In {{KafkaMessageGenerator}}, the following properties are set when creating a
producer.
{code:java}
producerProperties.put(ProducerConfig.RETRIES_CONFIG, 0);
producerProperties.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
producerProperties.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
{code}
Consider slower systems or heavily loaded system, where the acknowledgement for
the message produced did not arrive in 1000ms - since {{RETRIES_CONFIG}} is set
to 0, produce will fail with {{org.apache.kafka.common.errors.NetworkException:
The server disconnected before a response was received.}}
Although, the unit tests never failed in my environment, I was able to
reproduce this error by reducing the value for {{REQUEST_TIMEOUT_MS_CONFIG}} to
as low as *50 ms*.
{noformat}
23:19:55.136 [main] ERROR o.a.d.e.s.k.KafkaMessageGenerator -
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.drill.exec.store.kafka.KafkaMessageGenerator.populateJsonMsgIntoKafka(KafkaMessageGenerator.java:126)
~[test-classes/:na]
at
org.apache.drill.exec.store.kafka.TestKafkaSuit.initKafka(TestKafkaSuit.java:88)
[test-classes/:na]
{noformat}
h3. h3. ISSUE 2 - {{Failed to fetch messages within 200 milliseconds}}
Again, this issue may occur in slower systems or systems under heavy load where
a consumer poll did not return messages within {{Poll Timeout}} set.
For unit tests, {{KAFKA_POLL_TIMEOUT}} is set to *200 ms*.
{code:java}
testNoResult(String.format("alter session set `%s` = %d",
ExecConstants.KAFKA_POLL_TIMEOUT, 200));
{code}
If a poll does not return a message within this time then following exception
is thrown.
{{org.apache.drill.exec.rpc.RpcException:
org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: Failed
to fetch messages within 10 milliseconds. Consider increasing the value of the
property : store.kafka.poll.timeout}}
I was able to reproduce this issue by reducing {{KAFKA_POLL_TIMEOUT}} value to
as low as *50 ms.*
h3. h3. Solution
* The value for producer {{REQUEST_TIMEOUT_MS_CONFIG}} should be increased (to
say 10s) and similarly the value for consumer {{KAFKA_POLL_TIMEOUT}} should
also be increased.
* We should increase producer {{RETRIES_CONFIG}} to allow retries. We can
eliminate the possibility of having duplicate messages by using *{{Idempotent
Producer}}*{{.}}
[~akumarb2010], [~kam_iitkgp], [~kkhatua] - any other suggestions?
was (Author: aravi5):
Had some time to dig further into this issue. [~ben-zvi] - mentioned that the
issues are seen when running on Mac and they seem to be intermittent failures.
h3. h3. Issue 1 - "{{The server disconnected before a response was received"}}
In {{KafkaMessageGenerator}}, the following properties are set when creating a
producer.
{code}
producerProperties.put(ProducerConfig.RETRIES_CONFIG, 0);
producerProperties.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
producerProperties.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
{code}
Consider slower systems or heavily loaded system, where the acknowledgement for
the message produced did not arrive in 1000ms - since {{RETRIES_CONFIG}} is set
to 0, produce will fail with {{org.apache.kafka.common.errors.NetworkException:
The server disconnected before a response was received.}}
Although, the unit tests never failed in my environment, I was able to
reproduce this error by reducing the value for {{REQUEST_TIMEOUT_MS_CONFIG}} to
as low as *50 ms*.
{noformat}
23:19:55.136 [main] ERROR o.a.d.e.s.k.KafkaMessageGenerator -
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
at
[jira] [Commented] (DRILL-6625) Intermittent failures in Kafka unit tests
[
https://issues.apache.org/jira/browse/DRILL-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604682#comment-16604682
]
Abhishek Ravi commented on DRILL-6625:
--
Had some time to dig further into this issue. [~ben-zvi] - mentioned that the
issues are seen when running on Mac and they seem to be intermittent failures.
h3. h3. Issue 1 - "{{The server disconnected before a response was received"}}
In {{KafkaMessageGenerator}}, the following properties are set when creating a
producer.
{code}
producerProperties.put(ProducerConfig.RETRIES_CONFIG, 0);
producerProperties.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
producerProperties.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
{code}
Consider slower systems or heavily loaded system, where the acknowledgement for
the message produced did not arrive in 1000ms - since {{RETRIES_CONFIG}} is set
to 0, produce will fail with {{org.apache.kafka.common.errors.NetworkException:
The server disconnected before a response was received.}}
Although, the unit tests never failed in my environment, I was able to
reproduce this error by reducing the value for {{REQUEST_TIMEOUT_MS_CONFIG}} to
as low as *50 ms*.
{noformat}
23:19:55.136 [main] ERROR o.a.d.e.s.k.KafkaMessageGenerator -
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NetworkException: The server disconnected before
a response was received.
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
~[kafka-clients-0.11.0.1.jar:na]
at
org.apache.drill.exec.store.kafka.KafkaMessageGenerator.populateJsonMsgIntoKafka(KafkaMessageGenerator.java:126)
~[test-classes/:na]
at
org.apache.drill.exec.store.kafka.TestKafkaSuit.initKafka(TestKafkaSuit.java:88)
[test-classes/:na]
{noformat}
h3. h3. ISSUE 2 - {{Failed to fetch messages within 200 milliseconds}}
Again, this issue may occur in slower systems or systems under heavy load where
a consumer poll did not return messages within {{Poll Timeout}} set.
For unit tests, {{KAFKA_POLL_TIMEOUT}} is set to *200 ms*.
{code}
testNoResult(String.format("alter session set `%s` = %d",
ExecConstants.KAFKA_POLL_TIMEOUT, 200));
{code}
If a poll does not return a message within this time then following exception
is thrown.
{{org.apache.drill.exec.rpc.RpcException:
org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR: Failed
to fetch messages within 10 milliseconds. Consider increasing the value of the
property : store.kafka.poll.timeout}}
I was able to reproduce this issue by reducing {{KAFKA_POLL_TIMEOUT}} value to
as low as *50 ms.*
h3. h3. Solution
* The value for producer {{REQUEST_TIMEOUT_MS_CONFIG}} should be increased (to
say 10s) and similarly the value for consumer {{KAFKA_POLL_TIMEOUT}} should
also be increased.
* We should increase producer {{RETRIES_CONFIG}} to allow retries. We can
eliminate the possibility of having duplicate messages by using *{{Idempotent
Producer}}*{{.}}
[~akumarb2010], [~kkhatua] - any thoughts?
> Intermittent failures in Kafka unit tests
> -
>
> Key: DRILL-6625
> URL: https://issues.apache.org/jira/browse/DRILL-6625
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.13.0
>Reporter: Boaz Ben-Zvi
>Assignee: Abhishek Ravi
>Priority: Major
> Fix For: 1.15.0
>
>
> The following failures have been seen (consistently on my Mac, or
> occasionally on Jenkins) when running the unit tests, in the Kafka test suit.
> After the failure, maven hangs for a long time.
> Cost was 0.0 (instead of 26.0) :
> {code:java}
> Running org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
> 16:46:57.748 [main] ERROR org.apache.drill.TestReporter - Test Failed (d:
> -65.3 KiB(73.6 KiB), h: -573.5 MiB(379.5 MiB), nh: 1.2 MiB(117.1 MiB)):
> testPushdownWithOr(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
> java.lang.AssertionError: Unable to find expected string "kafkaScanSpec"
> : {
> "topicName" : "drill-pushdown-topic"
> },
> "cost" : 26.0 in plan: {
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ {
> "kind" : "STRING",
> "accessibleScopes" : "ALL",
> "name" :
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450191#comment-16450191
]
Abhishek Ravi commented on DRILL-5977:
--
I haven't got a chance to actively work on it in the last couple of weeks.
Planning to send it out for review before 05/15.
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Abhishek Ravi
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421951#comment-16421951
]
Abhishek Ravi commented on DRILL-5977:
--
Thank you for review [~akumarb2010]. Yes, you are absolutely right. As an
initial approach to tackle this problem I plan to do the following after
obtaining *top-level predicates* in an expression.
# Check if condition on {{kafkaMsgTimestamp}} / {{kafkaMsgOffset exists.}}
# Check if there is no {{OR}} joining top-level predicates.
Do filter pushdown only when both checks succeed. Does this sound good?
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Bhallamudi Venkata Siva Kamesh
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416857#comment-16416857
]
Abhishek Ravi commented on DRILL-5977:
--
[~kam_iitkgp] [~akumarb2010] - I am planning to take a dig at implementation of
filter pushdown. The following document captures some high level implementation
details for this feature.
https://docs.google.com/document/d/1SZ4wO4Ii4nAHwgWbY6JJynPOseCha0DBBZJ2ig1RO0M/edit#
Please review the document and let me know what you think.
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Bhallamudi Venkata Siva Kamesh
>Priority: Major
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
