[jira] [Created] (DRILL-8442) NPE on DeltaRowGroupScan
Matt Keranen created DRILL-8442: --- Summary: NPE on DeltaRowGroupScan Key: DRILL-8442 URL: https://issues.apache.org/jira/browse/DRILL-8442 Project: Apache Drill Issue Type: Bug Components: Storage - Other Affects Versions: 1.21.1 Environment: pyspark 3.4.0 delta-spark 2.4.0 Ubuntu 22.04.2 LTS Reporter: Matt Keranen SELECT * on Delta table (Parquet) throws null pointer exception: {noformat} 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET `exec.query.max_rows`=10002023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested AWAITING_ALLOCATION --> RUNNING2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING --> FINISHED2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))limit 52023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested AWAITING_ALLOCATION --> FAILED2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested AWAITING_ALLOCATION --> FAILED2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED --> FINISHED2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED --> FINISHED2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested AWAITING_ALLOCATION --> FAILED2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException Fragment: 1:1 Please, refer to logs for more information. [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException Fragment: 1:1 Please, refer to logs for more information. [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688) at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem: `java.lang.NullPointerException` at [Source: (String)"{ "pop" : "single-sender", "@id" : 0, "receiver-major-fragment" : 0, "receiver-minor-fragment" : 0, "child" : { "pop" : "selection-vector-remover", "@id" : 1, "child" : { "pop" : "limit", "@id" : 2, "child" : { "pop" : "delta-row-group-scan", "@id" : 3, "userName" : "mattk", "formatPluginConfig" : { "type" : "delta", "version" : null, "timestamp" : null }, "rowGroupReadEntries"[truncated 18683 chars]; line: 467, column: 7] (through reference chain:
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735427#comment-17735427 ] ASF GitHub Bot commented on DRILL-8353: --- kmatt commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1599386040 @vvysotskyi https://issues.apache.org/jira/browse/DRILL-8442 Should this be a GitHub issue, or is Jira the correct place for it? > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.21.0 > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735428#comment-17735428 ] ASF GitHub Bot commented on DRILL-8353: --- cgivre commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1599387786 @kmatt A github issue is good! Please be sure to tag @vvysotskyi in it as he was the original developer of this plugin. > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.21.0 > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8442) NPE on DeltaRowGroupScan
[ https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Keranen updated DRILL-8442: Description: SELECT * on Delta table (Parquet) throws null pointer exception: {noformat} 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET `exec.query.max_rows`=1000 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING --> FINISHED 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select * from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta')) limit 5 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested AWAITING_ALLOCATION --> FAILED 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested AWAITING_ALLOCATION --> FAILED 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED --> FINISHED 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED --> FINISHED 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO o.a.d.e.w.fragment.FragmentExecutor - 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested AWAITING_ALLOCATION --> FAILED 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException Fragment: 1:1 Please, refer to logs for more information. [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException Fragment: 1:1 Please, refer to logs for more information. [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688) at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392) at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem: `java.lang.NullPointerException` at [Source: (String)"{ "pop" : "single-sender", "@id" : 0, "receiver-major-fragment" : 0, "receiver-minor-fragment" : 0, "child" : { "pop" : "selection-vector-remover", "@id" : 1, "child" : { "pop" : "limit", "@id" : 2, "child" : { "pop" : "delta-row-group-scan", "@id" : 3, "userName" : "mattk", "formatPluginConfig" : { "type" : "delta", "version" : null, "timestamp" : null }, "rowGroupReadEntries"[truncated 18683 chars]; line: 467, column: 7] (through reference chain: org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"]) at