[jira] [Closed] (DRILL-8489) Sender memory leak when rpc encode exception
[
https://issues.apache.org/jira/browse/DRILL-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8489.
---
Resolution: Fixed
> Sender memory leak when rpc encode exception
>
>
> Key: DRILL-8489
> URL: https://issues.apache.org/jira/browse/DRILL-8489
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
>
> When encode throw Exception, if encode msg instanceof ReferenceCounted, netty
> can release msg, but drill convert msg to OutboundRpcMessage, so netty can
> not release msg. this causes sender memory leaks
> exception info
> {code:java}
> 2024-04-16 16:25:57,998 [DataClient-7] ERROR
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
> Connection: /10.32.112.138:47924 <--> /10.32.112.138:31012 (data client).
> Closing connection.
> io.netty.handler.codec.EncoderException:
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
> buffer of size 4096 due to memory limit (9223372036854775807). Current
> allocation: 0
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247)
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
> allocate buffer of size 4096 due to memory limit (9223372036854775807).
> Current allocation: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:245)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:55)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:50)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(safeRelease.java:87)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(RpcEncoder.java:38)
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:90){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8488) HashJoinPOP memory leak is caused by OutOfMemoryException
[ https://issues.apache.org/jira/browse/DRILL-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8488. --- Resolution: Fixed > HashJoinPOP memory leak is caused by OutOfMemoryException > -- > > Key: DRILL-8488 > URL: https://issues.apache.org/jira/browse/DRILL-8488 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.21.1 >Reporter: shihuafeng >Priority: Major > Fix For: 1.21.2 > > > [DRILL-8485|[DRILL-8485] HashJoinPOP memory leak is caused by an oom > exception when read data from InputStream - ASF JIRA (apache.org)] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8489) Sender memory leak when rpc encode exception
[
https://issues.apache.org/jira/browse/DRILL-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8489:
Fix Version/s: 1.21.2
(was: 1.22.0)
> Sender memory leak when rpc encode exception
>
>
> Key: DRILL-8489
> URL: https://issues.apache.org/jira/browse/DRILL-8489
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
>
> When encode throw Exception, if encode msg instanceof ReferenceCounted, netty
> can release msg, but drill convert msg to OutboundRpcMessage, so netty can
> not release msg. this causes sender memory leaks
> exception info
> {code:java}
> 2024-04-16 16:25:57,998 [DataClient-7] ERROR
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
> Connection: /10.32.112.138:47924 <--> /10.32.112.138:31012 (data client).
> Closing connection.
> io.netty.handler.codec.EncoderException:
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
> buffer of size 4096 due to memory limit (9223372036854775807). Current
> allocation: 0
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247)
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
> allocate buffer of size 4096 due to memory limit (9223372036854775807).
> Current allocation: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:245)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:55)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:50)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(safeRelease.java:87)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(RpcEncoder.java:38)
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:90){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8488) HashJoinPOP memory leak is caused by OutOfMemoryException
[ https://issues.apache.org/jira/browse/DRILL-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8488: Fix Version/s: 1.21.2 (was: 1.22.0) > HashJoinPOP memory leak is caused by OutOfMemoryException > -- > > Key: DRILL-8488 > URL: https://issues.apache.org/jira/browse/DRILL-8488 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.21.1 >Reporter: shihuafeng >Priority: Major > Fix For: 1.21.2 > > > [DRILL-8485|[DRILL-8485] HashJoinPOP memory leak is caused by an oom > exception when read data from InputStream - ASF JIRA (apache.org)] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8480) Cleanup before finished. 0 out of 1 streams have finished
[
https://issues.apache.org/jira/browse/DRILL-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8480.
---
> Cleanup before finished. 0 out of 1 streams have finished
> -
>
> Key: DRILL-8480
> URL: https://issues.apache.org/jira/browse/DRILL-8480
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Fix For: 1.21.2
>
> Attachments: 1a349ff1-d1f9-62bf-ed8c-26346c548005.sys.drill,
> tableWithNumber2.parquet
>
>
> Drill fails to execute a query with the following exception:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Cleanup before finished. 0 out of 1 streams have
> finished
> Fragment: 1:0
> Please, refer to logs for more information.
> [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on
> compute7.vmcluster.com:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of
> 1 streams have finished
> at
> org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71)
> at
> org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240)
> ... 5 common frames omitted
> Suppressed: java.lang.IllegalStateException: Cleanup before finished.
> 0 out of 1 streams have finished
> ... 15 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (32768)
> Allocator(op:1:0:8:UnorderedReceiver) 100/32768/32768/100
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159)
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571)
> ... 7 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (1016640)
> Allocator(frag:1:0) 3000/1016640/30016640/90715827882
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574)
> ... 7 common frames omitted {code}
> Steps to reproduce:
> 1.Enable unequal join:
> {code:java}
> alter session set `planner.enable_nljoin_for_scalar_only`=false; {code}
> 2. Disable join optimization to prevent Drill from flipping sides of
> join that may break the query execution because the NestedLoopJoin operator
> that executes unequal joins supports only the left join.
> {code:java}
> alter session set
[jira] [Updated] (DRILL-8480) Cleanup before finished. 0 out of 1 streams have finished
[
https://issues.apache.org/jira/browse/DRILL-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8480:
Fix Version/s: 1.21.2
> Cleanup before finished. 0 out of 1 streams have finished
> -
>
> Key: DRILL-8480
> URL: https://issues.apache.org/jira/browse/DRILL-8480
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Fix For: 1.21.2
>
> Attachments: 1a349ff1-d1f9-62bf-ed8c-26346c548005.sys.drill,
> tableWithNumber2.parquet
>
>
> Drill fails to execute a query with the following exception:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Cleanup before finished. 0 out of 1 streams have
> finished
> Fragment: 1:0
> Please, refer to logs for more information.
> [Error Id: 270da8f4-0bb6-4985-bf4f-34853138881c on
> compute7.vmcluster.com:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:657)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:395)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:245)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:362)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.IllegalStateException: Cleanup before finished. 0 out of
> 1 streams have finished
> at
> org.apache.drill.exec.work.batch.BaseRawBatchBuffer.close(BaseRawBatchBuffer.java:111)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:71)
> at
> org.apache.drill.exec.work.batch.AbstractDataCollector.close(AbstractDataCollector.java:121)
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:91)
> at
> org.apache.drill.exec.work.batch.IncomingBuffers.close(IncomingBuffers.java:144)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:567)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:417)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:240)
> ... 5 common frames omitted
> Suppressed: java.lang.IllegalStateException: Cleanup before finished.
> 0 out of 1 streams have finished
> ... 15 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (32768)
> Allocator(op:1:0:8:UnorderedReceiver) 100/32768/32768/100
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.BaseOperatorContext.close(BaseOperatorContext.java:159)
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:77)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:571)
> ... 7 common frames omitted
> Suppressed: java.lang.IllegalStateException: Memory was leaked by
> query. Memory leaked: (1016640)
> Allocator(frag:1:0) 3000/1016640/30016640/90715827882
> (res/actual/peak/limit)
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose(FragmentContextImpl.java:581)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.close(FragmentContextImpl.java:574)
> ... 7 common frames omitted {code}
> Steps to reproduce:
> 1.Enable unequal join:
> {code:java}
> alter session set `planner.enable_nljoin_for_scalar_only`=false; {code}
> 2. Disable join optimization to prevent Drill from flipping sides of
> join that may break the query execution because the NestedLoopJoin operator
> that executes unequal joins supports only the left join.
> {code:java}
> alter session set
[jira] [Closed] (DRILL-8487) HTTP Caching
[ https://issues.apache.org/jira/browse/DRILL-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8487. --- Resolution: Duplicate > HTTP Caching > > > Key: DRILL-8487 > URL: https://issues.apache.org/jira/browse/DRILL-8487 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Sena >Priority: Major > > I am using http storage plugin and I want to activate the caching. In the > documentation it says that this requires adding cacheResults. So I added this > to my config. When I test using an older version(1.20.1), I can see the query > result files under the tmp/http-cache directory, but when I test using a > newer version(1.21.1), there are no query result files in that directory, it > only contains the journal. This PR > [https://github.com/apache/drill/pull/2669] may have caused the issue. > Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8487) HTTP Caching
[ https://issues.apache.org/jira/browse/DRILL-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8487: - > HTTP Caching > > > Key: DRILL-8487 > URL: https://issues.apache.org/jira/browse/DRILL-8487 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Sena >Priority: Major > > I am using http storage plugin and I want to activate the caching. In the > documentation it says that this requires adding cacheResults. So I added this > to my config. When I test using an older version(1.20.1), I can see the query > result files under the tmp/http-cache directory, but when I test using a > newer version(1.21.1), there are no query result files in that directory, it > only contains the journal. This PR > [https://github.com/apache/drill/pull/2669] may have caused the issue. > Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8487) HTTP Caching
[ https://issues.apache.org/jira/browse/DRILL-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8487. --- Resolution: Fixed > HTTP Caching > > > Key: DRILL-8487 > URL: https://issues.apache.org/jira/browse/DRILL-8487 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Sena >Priority: Major > > I am using http storage plugin and I want to activate the caching. In the > documentation it says that this requires adding cacheResults. So I added this > to my config. When I test using an older version(1.20.1), I can see the query > result files under the tmp/http-cache directory, but when I test using a > newer version(1.21.1), there are no query result files in that directory, it > only contains the journal. This PR > [https://github.com/apache/drill/pull/2669] may have caused the issue. > Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8487) HTTP Caching
[ https://issues.apache.org/jira/browse/DRILL-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8487: Fix Version/s: (was: 1.20.1) > HTTP Caching > > > Key: DRILL-8487 > URL: https://issues.apache.org/jira/browse/DRILL-8487 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Sena >Priority: Major > > I am using http storage plugin and I want to activate the caching. In the > documentation it says that this requires adding cacheResults. So I added this > to my config. When I test using an older version(1.20.1), I can see the query > result files under the tmp/http-cache directory, but when I test using a > newer version(1.21.1), there are no query result files in that directory, it > only contains the journal. This PR > [https://github.com/apache/drill/pull/2669] may have caused the issue. > Also, is it possible to implement maximum cache size? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8494) HTTP Caching Not Saving Pages
[ https://issues.apache.org/jira/browse/DRILL-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8494. --- Resolution: Fixed > HTTP Caching Not Saving Pages > - > > Key: DRILL-8494 > URL: https://issues.apache.org/jira/browse/DRILL-8494 > Project: Apache Drill > Issue Type: Bug > Components: Storage - HTTP >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > A minor bugfix, but the HTTP storage plugin was not actually caching results > even when caching was set to true. This bug was introduced in DRILL-8329. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8495) Tried to remove unmanaged buffer
[
https://issues.apache.org/jira/browse/DRILL-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8495:
Fix Version/s: 1.21.2
> Tried to remove unmanaged buffer
>
>
> Key: DRILL-8495
> URL: https://issues.apache.org/jira/browse/DRILL-8495
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Fix For: 1.21.2
>
>
>
> Drill throws an exception on Hive table:
> {code:java}
> (java.lang.IllegalStateException) Tried to remove unmanaged buffer.
> org.apache.drill.exec.ops.BufferManagerImpl.replace():51
> io.netty.buffer.DrillBuf.reallocIfNeeded():101
>
> org.apache.drill.exec.store.hive.writers.primitive.HiveStringWriter.write():38
>
> org.apache.drill.exec.store.hive.readers.HiveDefaultRecordReader.readHiveRecordAndInsertIntoRecordBatch():416
>
> org.apache.drill.exec.store.hive.readers.HiveDefaultRecordReader.next():402
> org.apache.drill.exec.physical.impl.ScanBatch.internalNext():235
> org.apache.drill.exec.physical.impl.ScanBatch.next():299
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():237
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractRecordBatch.next():101
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():160
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():237
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
>
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
> org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0():321
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1899
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():310
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748 {code}
>
>
> Reproduce:
> # Create Hive table:
> {code:java}
> create table if NOT EXISTS students(id int, name string, surname string)
> stored as parquet;{code}
> # Insert a new row with 2 string values of size > 256 bytes:
> {code:java}
> insert into students values (1,
> 'Veeery
> long name',
> 'bg
> surname');{code}
> # Execute Drill query:
> {code:java}
> select * from hive.`students` {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8493) Drill Unable to Read XML Files with Namespaces
[ https://issues.apache.org/jira/browse/DRILL-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8493. --- Resolution: Fixed > Drill Unable to Read XML Files with Namespaces > -- > > Key: DRILL-8493 > URL: https://issues.apache.org/jira/browse/DRILL-8493 > Project: Apache Drill > Issue Type: Bug > Components: Format - XML >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > This is a bug fix whereby Drill ignores all data when an XML file has a > namespace. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8446) Incorrect use of OperatingSystemMXBean
[ https://issues.apache.org/jira/browse/DRILL-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8446. --- Resolution: Fixed > Incorrect use of OperatingSystemMXBean > -- > > Key: DRILL-8446 > URL: https://issues.apache.org/jira/browse/DRILL-8446 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Mahmoud Ouali Alami >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > Attachments: image-2023-07-04-15-36-42-905.png, > image-2023-07-04-16-24-59-662.png > > > *Context :* > In Drill "CpuGaugeSet" class, we use an internal class instead of a public > class : com.sun.management.OperatingSystemMXBean; > !image-2023-07-04-15-36-42-905.png|width=387,height=257! > This can result to a NoClassDefFoundError: > !image-2023-07-04-16-24-59-662.png|width=845,height=108! > *To do :* > Replace the private class "com.sun.managemenet.OperatingSystemMXBean" with > "java.lang.management.OperatingSystemMXBean", > > Kind regards, > Mahmoud > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8479) Merge Join Memory Leak Depleting Incoming Batches Throw Exception
[
https://issues.apache.org/jira/browse/DRILL-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8479.
---
Resolution: Fixed
> Merge Join Memory Leak Depleting Incoming Batches Throw Exception
> -
>
> Key: DRILL-8479
> URL: https://issues.apache.org/jira/browse/DRILL-8479
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Critical
> Fix For: 1.21.2
>
> Attachments: 0001-mergejoin-leak.patch
>
>
> *Describe the bug*
> megerjoin leak when RecordIterator allocate memory exception with
> OutOfMemoryException{*}{*}
> {*}Steps to reproduce the behavior{*}:
> # prepare data for tpch 1s
> # set direct memory 5g
> # set planner.enable_hashjoin =false to ensure use mergejoin operator。
> # set drill.memory.debug.allocator =true (Check for memory leaks )
> # 20 concurrent for tpch sql8
> # when it had OutOfMemoryException or null EXCEPTION , stopped all sql.
> # finding memory leak
> *Expected behavior*
> when all sql sop , we should find direct memory is 0 AND could not
> find leak log like following.
> {code:java}
> Allocator(op:2:0:11:MergeJoinPOP) 100/73728/4874240/100
> (res/actual/peak/limit){code}
> *Error detail, log output or screenshots*
> {code:java}
> Unable to allocate buffer of size XX (rounded from XX) due to memory limit
> (). Current allocation: xx{code}
> [^0001-mergejoin-leak.patch]
> sql
> {code:java}
> // code placeholder
> select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
> sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
> o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
> hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
> hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
> hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
> s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey
> and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name
> = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
> '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
> all_nations group by o_year order by o_year
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8479) Merge Join Memory Leak Depleting Incoming Batches Throw Exception
[
https://issues.apache.org/jira/browse/DRILL-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8479:
Summary: Merge Join Memory Leak Depleting Incoming Batches Throw Exception
(was: mergejion memory leak when exception)
> Merge Join Memory Leak Depleting Incoming Batches Throw Exception
> -
>
> Key: DRILL-8479
> URL: https://issues.apache.org/jira/browse/DRILL-8479
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Critical
> Fix For: 1.21.2
>
> Attachments: 0001-mergejoin-leak.patch
>
>
> *Describe the bug*
> megerjoin leak when RecordIterator allocate memory exception with
> OutOfMemoryException{*}{*}
> {*}Steps to reproduce the behavior{*}:
> # prepare data for tpch 1s
> # set direct memory 5g
> # set planner.enable_hashjoin =false to ensure use mergejoin operator。
> # set drill.memory.debug.allocator =true (Check for memory leaks )
> # 20 concurrent for tpch sql8
> # when it had OutOfMemoryException or null EXCEPTION , stopped all sql.
> # finding memory leak
> *Expected behavior*
> when all sql sop , we should find direct memory is 0 AND could not
> find leak log like following.
> {code:java}
> Allocator(op:2:0:11:MergeJoinPOP) 100/73728/4874240/100
> (res/actual/peak/limit){code}
> *Error detail, log output or screenshots*
> {code:java}
> Unable to allocate buffer of size XX (rounded from XX) due to memory limit
> (). Current allocation: xx{code}
> [^0001-mergejoin-leak.patch]
> sql
> {code:java}
> // code placeholder
> select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
> sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
> o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
> hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
> hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
> hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
> s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey
> and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name
> = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
> '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
> all_nations group by o_year order by o_year
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8415) Upgrade Jackson 2.14.3 → 2.16.1
[ https://issues.apache.org/jira/browse/DRILL-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8415. --- Resolution: Fixed > Upgrade Jackson 2.14.3 → 2.16.1 > --- > > Key: DRILL-8415 > URL: https://issues.apache.org/jira/browse/DRILL-8415 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > I'm not advocating for an upgrade to [Jackson > 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. > 2.15.0-rc1 has just been released and 2.15.0 should be out soon. > There are some security focused enhancements including a new class called > StreamReadConstraints. The defaults on > [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] > are pretty high but it is not inconceivable that some Drill users might need > to relax them. Parsing large strings as numbers is sub-quadratic, thus the > default limit of 1000 chars or bytes (depending on input context). > When the Drill team consider upgrading to Jackson 2.15 or above, you might > also want to consider adding some way for users to configure the > StreamReadConstraints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8456. --- Resolution: Fixed > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8486) ParquetDecodingException: could not read bytes at offset
[
https://issues.apache.org/jira/browse/DRILL-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8486:
Fix Version/s: 1.21.2
> ParquetDecodingException: could not read bytes at offset
> -
>
> Key: DRILL-8486
> URL: https://issues.apache.org/jira/browse/DRILL-8486
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Fix For: 1.21.2
>
> Attachments: test.parquet
>
>
> Drill fails to read a parquet file with the following exception:
>
> {code:java}
> Caused by: org.apache.parquet.io.ParquetDecodingException: could not read
> bytes at offset 591804
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:42)
> at
> org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput$ValuesReaderWrapper.getNextEntry(VarLenColumnBulkInput.java:754)
> ... 43 common frames omitted
> Caused by: java.io.EOFException: null
> at
> org.apache.parquet.bytes.SingleBufferInputStream.read(SingleBufferInputStream.java:52)
> at
> org.apache.parquet.bytes.BytesUtils.readIntLittleEndian(BytesUtils.java:83)
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:39)
> ... 44 common frames omitted {code}
>
>
> This issue only affects queries with {{store.parquet.flat.reader.bulk}} set
> to {{{}true{}}}(by default).
> Attaching the parquet file for the reproduce: [^test.parquet].
> Query: {{select log, app_name from dfs.tmp.`test.parquet`}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8486) ParquetDecodingException: could not read bytes at offset
[
https://issues.apache.org/jira/browse/DRILL-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8486.
---
> ParquetDecodingException: could not read bytes at offset
> -
>
> Key: DRILL-8486
> URL: https://issues.apache.org/jira/browse/DRILL-8486
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Maksym Rymar
>Assignee: Maksym Rymar
>Priority: Major
> Fix For: 1.21.2
>
> Attachments: test.parquet
>
>
> Drill fails to read a parquet file with the following exception:
>
> {code:java}
> Caused by: org.apache.parquet.io.ParquetDecodingException: could not read
> bytes at offset 591804
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:42)
> at
> org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput$ValuesReaderWrapper.getNextEntry(VarLenColumnBulkInput.java:754)
> ... 43 common frames omitted
> Caused by: java.io.EOFException: null
> at
> org.apache.parquet.bytes.SingleBufferInputStream.read(SingleBufferInputStream.java:52)
> at
> org.apache.parquet.bytes.BytesUtils.readIntLittleEndian(BytesUtils.java:83)
> at
> org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:39)
> ... 44 common frames omitted {code}
>
>
> This issue only affects queries with {{store.parquet.flat.reader.bulk}} set
> to {{{}true{}}}(by default).
> Attaching the parquet file for the reproduce: [^test.parquet].
> Query: {{select log, app_name from dfs.tmp.`test.parquet`}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8415) Upgrade Jackson 2.14.3 → 2.16.1
[ https://issues.apache.org/jira/browse/DRILL-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8415: Fix Version/s: 1.21.2 (was: 1.22.0) > Upgrade Jackson 2.14.3 → 2.16.1 > --- > > Key: DRILL-8415 > URL: https://issues.apache.org/jira/browse/DRILL-8415 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > I'm not advocating for an upgrade to [Jackson > 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. > 2.15.0-rc1 has just been released and 2.15.0 should be out soon. > There are some security focused enhancements including a new class called > StreamReadConstraints. The defaults on > [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] > are pretty high but it is not inconceivable that some Drill users might need > to relax them. Parsing large strings as numbers is sub-quadratic, thus the > default limit of 1000 chars or bytes (depending on input context). > When the Drill team consider upgrading to Jackson 2.15 or above, you might > also want to consider adding some way for users to configure the > StreamReadConstraints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8445) Upgrade Janino 3.1.8 → 3.1.11
[ https://issues.apache.org/jira/browse/DRILL-8445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8445: Fix Version/s: 1.21.2 > Upgrade Janino 3.1.8 → 3.1.11 > - > > Key: DRILL-8445 > URL: https://issues.apache.org/jira/browse/DRILL-8445 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > I'm not familar with exactly how janino is used inside Drill. > There is a new 3.1.10 release today to fix > [https://github.com/janino-compiler/janino/issues/201] > This may be an issue if Janino is used to parse input that may not be > entirely trustworthy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8445) Upgrade Janino 3.1.8 → 3.1.11
[ https://issues.apache.org/jira/browse/DRILL-8445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8445. --- Resolution: Fixed > Upgrade Janino 3.1.8 → 3.1.11 > - > > Key: DRILL-8445 > URL: https://issues.apache.org/jira/browse/DRILL-8445 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > I'm not familar with exactly how janino is used inside Drill. > There is a new 3.1.10 release today to fix > [https://github.com/janino-compiler/janino/issues/201] > This may be an issue if Janino is used to parse input that may not be > entirely trustworthy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8446) Incorrect use of OperatingSystemMXBean
[ https://issues.apache.org/jira/browse/DRILL-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8446: Fix Version/s: 1.21.2 (was: 1.22.0) > Incorrect use of OperatingSystemMXBean > -- > > Key: DRILL-8446 > URL: https://issues.apache.org/jira/browse/DRILL-8446 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Mahmoud Ouali Alami >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > Attachments: image-2023-07-04-15-36-42-905.png, > image-2023-07-04-16-24-59-662.png > > > *Context :* > In Drill "CpuGaugeSet" class, we use an internal class instead of a public > class : com.sun.management.OperatingSystemMXBean; > !image-2023-07-04-15-36-42-905.png|width=387,height=257! > This can result to a NoClassDefFoundError: > !image-2023-07-04-16-24-59-662.png|width=845,height=108! > *To do :* > Replace the private class "com.sun.managemenet.OperatingSystemMXBean" with > "java.lang.management.OperatingSystemMXBean", > > Kind regards, > Mahmoud > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8446) Incorrect use of OperatingSystemMXBean
[ https://issues.apache.org/jira/browse/DRILL-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8446: - > Incorrect use of OperatingSystemMXBean > -- > > Key: DRILL-8446 > URL: https://issues.apache.org/jira/browse/DRILL-8446 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Mahmoud Ouali Alami >Assignee: James Turton >Priority: Major > Fix For: 1.22.0 > > Attachments: image-2023-07-04-15-36-42-905.png, > image-2023-07-04-16-24-59-662.png > > > *Context :* > In Drill "CpuGaugeSet" class, we use an internal class instead of a public > class : com.sun.management.OperatingSystemMXBean; > !image-2023-07-04-15-36-42-905.png|width=387,height=257! > This can result to a NoClassDefFoundError: > !image-2023-07-04-16-24-59-662.png|width=845,height=108! > *To do :* > Replace the private class "com.sun.managemenet.OperatingSystemMXBean" with > "java.lang.management.OperatingSystemMXBean", > > Kind regards, > Mahmoud > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton reopened DRILL-8458:
-
> Reading Parquet v2 data page with repetition levels larger than column data
> throws IllegalArgumentException
> ---
>
> Key: DRILL-8458
> URL: https://issues.apache.org/jira/browse/DRILL-8458
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Peter Franzen
>Assignee: James Turton
>Priority: Major
> Fix For: 1.22.0
>
>
> When the size of the repetition level bytes in a Parquet v2 data page is
> larger than the size of the column data bytes,
> {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}}
> throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set
> the limit of a ByteBuffer to a value large than its capacity.
>
> The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}:
>
> {code:java}
> 217 int pageBufOffset = 0;
> 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 219 BytesInput repLevelBytes = BytesInput.from(
> 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize)
> 221 );
> 222 pageBufOffset += repLevelSize;
> 223
> 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 225 final BytesInput defLevelBytes = BytesInput.from(
> 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize)
> 227 );
> 228 pageBufOffset += defLevelSize; {code}
>
> The buffer {{pageBuf}} contains the repetition level bytes followed by the
> definition level bytes followed by the column data bytes.
>
> The code at lines 217-221 reads the repetition level bytes, and then updates
> the position of the {{pageBuf}} buffer to the start of the definition level
> bytes (lines 222 and 224).
>
> The code at lines 225-227 reads the definition level bytes, and when creating
> a slice of the \{{pageBuf }}buffer containing the definition level bytes, the
> slice's limit is set as if the position was at the beginning of the
> repetition level bytes (line 226), i.e as if it not had been updated.
>
> This means that if the capacity of the pageBuf buffer (which is the size of
> the repetition level bytes + the size of the definition level bytes + the
> size of the column data bytes) is less than (repLevelSize + repLevelSize +
> defLevelSize), the call to limit() will throw.
>
> The fix is to change line 226 to
> {code:java}
> (ByteBuffer) bb.slice().limit(defLevelSize){code}
>
> For symmetry, line 220 could also be changed to
> {code:java}
> (ByteBuffer) bb.slice().limit(repLevelSize){code}
>
> although {{pageBufOffset}} is always 0 there and will not cause the limit to
> exceed the capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8458.
---
Resolution: Fixed
> Reading Parquet v2 data page with repetition levels larger than column data
> throws IllegalArgumentException
> ---
>
> Key: DRILL-8458
> URL: https://issues.apache.org/jira/browse/DRILL-8458
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Peter Franzen
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.2
>
>
> When the size of the repetition level bytes in a Parquet v2 data page is
> larger than the size of the column data bytes,
> {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}}
> throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set
> the limit of a ByteBuffer to a value large than its capacity.
>
> The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}:
>
> {code:java}
> 217 int pageBufOffset = 0;
> 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 219 BytesInput repLevelBytes = BytesInput.from(
> 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize)
> 221 );
> 222 pageBufOffset += repLevelSize;
> 223
> 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 225 final BytesInput defLevelBytes = BytesInput.from(
> 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize)
> 227 );
> 228 pageBufOffset += defLevelSize; {code}
>
> The buffer {{pageBuf}} contains the repetition level bytes followed by the
> definition level bytes followed by the column data bytes.
>
> The code at lines 217-221 reads the repetition level bytes, and then updates
> the position of the {{pageBuf}} buffer to the start of the definition level
> bytes (lines 222 and 224).
>
> The code at lines 225-227 reads the definition level bytes, and when creating
> a slice of the \{{pageBuf }}buffer containing the definition level bytes, the
> slice's limit is set as if the position was at the beginning of the
> repetition level bytes (line 226), i.e as if it not had been updated.
>
> This means that if the capacity of the pageBuf buffer (which is the size of
> the repetition level bytes + the size of the definition level bytes + the
> size of the column data bytes) is less than (repLevelSize + repLevelSize +
> defLevelSize), the call to limit() will throw.
>
> The fix is to change line 226 to
> {code:java}
> (ByteBuffer) bb.slice().limit(defLevelSize){code}
>
> For symmetry, line 220 could also be changed to
> {code:java}
> (ByteBuffer) bb.slice().limit(repLevelSize){code}
>
> although {{pageBufOffset}} is always 0 there and will not cause the limit to
> exceed the capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8459. --- Resolution: Fixed > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8459: - > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8459) Bump avro to 1.11.3 due to CVE
[ https://issues.apache.org/jira/browse/DRILL-8459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8459: Fix Version/s: 1.21.2 (was: 1.22.0) > Bump avro to 1.11.3 due to CVE > -- > > Key: DRILL-8459 > URL: https://issues.apache.org/jira/browse/DRILL-8459 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > https://github.com/apache/drill/security/dependabot/49 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8458) Reading Parquet v2 data page with repetition levels larger than column data throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/DRILL-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8458:
Fix Version/s: 1.21.2
(was: 1.22.0)
> Reading Parquet v2 data page with repetition levels larger than column data
> throws IllegalArgumentException
> ---
>
> Key: DRILL-8458
> URL: https://issues.apache.org/jira/browse/DRILL-8458
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.21.1
>Reporter: Peter Franzen
>Assignee: James Turton
>Priority: Major
> Fix For: 1.21.2
>
>
> When the size of the repetition level bytes in a Parquet v2 data page is
> larger than the size of the column data bytes,
> {{org.apache.parquet.hadoop.ColumnChunkIncReadStore$ColumnChunkIncPageReader::readPage}}
> throws an {{{}IllegalArgumentException{}}}. This is caused by trying to set
> the limit of a ByteBuffer to a value large than its capacity.
>
> The offending code is at line 226 in {{{}ColumnChunkIncReadStore.java{}}}:
>
> {code:java}
> 217 int pageBufOffset = 0;
> 218 ByteBuffer bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 219 BytesInput repLevelBytes = BytesInput.from(
> 220 (ByteBuffer) bb.slice().limit(pageBufOffset + repLevelSize)
> 221 );
> 222 pageBufOffset += repLevelSize;
> 223
> 224 bb = (ByteBuffer) pageBuf.position(pageBufOffset);
> 225 final BytesInput defLevelBytes = BytesInput.from(
> 226 (ByteBuffer) bb.slice().limit(pageBufOffset + defLevelSize)
> 227 );
> 228 pageBufOffset += defLevelSize; {code}
>
> The buffer {{pageBuf}} contains the repetition level bytes followed by the
> definition level bytes followed by the column data bytes.
>
> The code at lines 217-221 reads the repetition level bytes, and then updates
> the position of the {{pageBuf}} buffer to the start of the definition level
> bytes (lines 222 and 224).
>
> The code at lines 225-227 reads the definition level bytes, and when creating
> a slice of the \{{pageBuf }}buffer containing the definition level bytes, the
> slice's limit is set as if the position was at the beginning of the
> repetition level bytes (line 226), i.e as if it not had been updated.
>
> This means that if the capacity of the pageBuf buffer (which is the size of
> the repetition level bytes + the size of the definition level bytes + the
> size of the column data bytes) is less than (repLevelSize + repLevelSize +
> defLevelSize), the call to limit() will throw.
>
> The fix is to change line 226 to
> {code:java}
> (ByteBuffer) bb.slice().limit(defLevelSize){code}
>
> For symmetry, line 220 could also be changed to
> {code:java}
> (ByteBuffer) bb.slice().limit(repLevelSize){code}
>
> although {{pageBufOffset}} is always 0 there and will not cause the limit to
> exceed the capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8456: - > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8452: Fix Version/s: 1.21.2 (was: 1.22.0) > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8452: - > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.22.0 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8452) Library upgrades
[ https://issues.apache.org/jira/browse/DRILL-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8452. --- Resolution: Fixed > Library upgrades > > > Key: DRILL-8452 > URL: https://issues.apache.org/jira/browse/DRILL-8452 > Project: Apache Drill > Issue Type: Improvement > Components: library >Affects Versions: 1.21.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 1.21.2 > > > * aircompressor.version -> 0.25 > * antlr.version -> -4.13.0- 4.9.3 > * asm.version -> 9.5 > * avro.version -> 1.11.2 > * commons.compress.version -> 1.23.0 > * commons.validator.version -> 1.7 > * hbase.version -> 2.5.5 (Hadoop 2 profile) > * hbase.version -> 2.5.5-hadoop3 > * -hikari.version -> 5.0.1- > * httpclient.version -> 4.5.14 > * httpdlog-parser.version -> 5.10.0 > * jersey.version -> 2.40 > * jetty -> 9.4.51.v20230217 > * jna.version -> 5.13.0 > * joda.version -> 2.12.5 > * libthrift.version -> 0.18.1 > * log4j.version -> 2.20.0 > * -maven.version -> 3.9.4- > * metrics.version -> 4.2.19 > * protostuff.version -> 1.8.0 > * snakeyaml.version -> 2.1 > * surefire.version -> 3.1.2 > * testcontainers.version -> 1.18.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8456) Upgrade POI to 5.2.4
[ https://issues.apache.org/jira/browse/DRILL-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8456: Fix Version/s: 1.21.2 (was: 1.22.0) > Upgrade POI to 5.2.4 > > > Key: DRILL-8456 > URL: https://issues.apache.org/jira/browse/DRILL-8456 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > latest release with some transitive dependencies having security patches -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8442) NPE on DeltaRowGroupScan
[
https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8442:
Fix Version/s: 1.21.2
> NPE on DeltaRowGroupScan
>
>
> Key: DRILL-8442
> URL: https://issues.apache.org/jira/browse/DRILL-8442
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.21.1
> Environment: pyspark 3.4.0
> delta-spark 2.4.0
> Ubuntu 22.04.2 LTS
>
>Reporter: Matt Keranen
>Assignee: Vova Vysotskyi
>Priority: Minor
> Fix For: 1.21.2
>
>
> SELECT * on Delta table (Parquet) throws null pointer exception:
>
> {noformat}
> 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET
> `exec.query.max_rows`=1000
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING -->
> FINISHED
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED
> 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *
> from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))
> limit 5
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException:
> Cannot construct instance of
> `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem:
> `java.lang.NullPointerException`
> at [Source: (String)"{
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
>
[jira] [Closed] (DRILL-8442) NPE on DeltaRowGroupScan
[
https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8442.
---
Resolution: Fixed
> NPE on DeltaRowGroupScan
>
>
> Key: DRILL-8442
> URL: https://issues.apache.org/jira/browse/DRILL-8442
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.21.1
> Environment: pyspark 3.4.0
> delta-spark 2.4.0
> Ubuntu 22.04.2 LTS
>
>Reporter: Matt Keranen
>Assignee: Vova Vysotskyi
>Priority: Minor
> Fix For: 1.21.2
>
>
> SELECT * on Delta table (Parquet) throws null pointer exception:
>
> {noformat}
> 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET
> `exec.query.max_rows`=1000
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING -->
> FINISHED
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED
> 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *
> from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))
> limit 5
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException:
> Cannot construct instance of
> `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem:
> `java.lang.NullPointerException`
> at [Source: (String)"{
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child"
[jira] [Closed] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8438. --- Resolution: Fixed > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > Fix For: 1.21.2 > > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DRILL-8442) NPE on DeltaRowGroupScan
[
https://issues.apache.org/jira/browse/DRILL-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton reopened DRILL-8442:
-
> NPE on DeltaRowGroupScan
>
>
> Key: DRILL-8442
> URL: https://issues.apache.org/jira/browse/DRILL-8442
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Affects Versions: 1.21.1
> Environment: pyspark 3.4.0
> delta-spark 2.4.0
> Ubuntu 22.04.2 LTS
>
>Reporter: Matt Keranen
>Assignee: Vova Vysotskyi
>Priority: Minor
>
> SELECT * on Delta table (Parquet) throws null pointer exception:
>
> {noformat}
> 2023-06-20 18:58:19,058 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2 issued by mattk: ALTER SESSION SET
> `exec.query.max_rows`=1000
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2023-06-20 18:58:19,068 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: RUNNING
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State change requested RUNNING -->
> FINISHED
> 2023-06-20 18:58:19,118 [1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 1b6e0933-dd1c-f16b-f6af-dd466d5d94f2:0:0: State to report: FINISHED
> 2023-06-20 18:58:19,137 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7 issued by mattk: select *
> from table(delta.root.`Warehouse/dbo/DeltaTestTable` (type => 'delta'))
> limit 5
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:1: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:0: State change requested FAILED -->
> FINISHED
> 2023-06-20 18:58:23,038 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 1b6e0933-c599-8d17-8971-5b0c2ecefac7:1:3: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2023-06-20 18:58:23,037 [1b6e0933-c599-8d17-8971-5b0c2ecefac7:frag:1:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment: 1:1
> Please, refer to logs for more information.
> [Error Id: c6b09027-199a-46e1-abb8-f37576c50382 on vm-etl-01:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:392)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:244)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:359)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException:
> Cannot construct instance of
> `org.apache.drill.exec.store.delta.DeltaRowGroupScan`, problem:
> `java.lang.NullPointerException`
> at [Source: (String)"{
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child" : {
> "pop" : "limit",
> "@id" : 2,
>
[jira] [Reopened] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reopened DRILL-8438: - > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8438) Bump YAUAA to 7.19.2
[ https://issues.apache.org/jira/browse/DRILL-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8438: Fix Version/s: 1.21.2 > Bump YAUAA to 7.19.2 > > > Key: DRILL-8438 > URL: https://issues.apache.org/jira/browse/DRILL-8438 > Project: Apache Drill > Issue Type: Task > Components: Functions - Drill >Reporter: Charles Givre >Assignee: Niels Basjes >Priority: Minor > Fix For: 1.21.2 > > > Bump YAUAA to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8430) Add factory method for creating Jackson ObjectMappers
[ https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8430: Fix Version/s: 1.21.2 > Add factory method for creating Jackson ObjectMappers > - > > Key: DRILL-8430 > URL: https://issues.apache.org/jira/browse/DRILL-8430 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > See https://issues.apache.org/jira/browse/DRILL-8415 > It's useful to keep any customisation of the ObjectMapper creation in 1 place -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8430) Add factory method for creating Jackson ObjectMappers
[ https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8430. --- Resolution: Fixed > Add factory method for creating Jackson ObjectMappers > - > > Key: DRILL-8430 > URL: https://issues.apache.org/jira/browse/DRILL-8430 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > See https://issues.apache.org/jira/browse/DRILL-8415 > It's useful to keep any customisation of the ObjectMapper creation in 1 place -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8430) Add factory method for creating Jackson ObjectMappers
[ https://issues.apache.org/jira/browse/DRILL-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8430: Summary: Add factory method for creating Jackson ObjectMappers (was: add factory method for creating Jackson ObjectMappers) > Add factory method for creating Jackson ObjectMappers > - > > Key: DRILL-8430 > URL: https://issues.apache.org/jira/browse/DRILL-8430 > Project: Apache Drill > Issue Type: Task > Components: Server >Reporter: PJ Fanning >Priority: Major > > See https://issues.apache.org/jira/browse/DRILL-8415 > It's useful to keep any customisation of the ObjectMapper creation in 1 place -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8475) Update the binary distributions LICENSE
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8475. --- > Update the binary distributions LICENSE > --- > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: dependencies.txt, drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DRILL-8475) Update the binary distributions LICENSE
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton resolved DRILL-8475. - Resolution: Fixed > Update the binary distributions LICENSE > --- > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: dependencies.txt, drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8478) HashPartition memory leak when exception
[
https://issues.apache.org/jira/browse/DRILL-8478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8478.
---
Resolution: Fixed
> HashPartition memory leak when exception
> -
>
> Key: DRILL-8478
> URL: https://issues.apache.org/jira/browse/DRILL-8478
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
> Attachments:
> 0001-DRILL-8478.-HashPartition-memory-leak-when-it-alloca.patch
>
>
> *Describe the bug*
> hashpartition leak when allocate memory exception with OutOfMemoryException
> *To Reproduce*
> Steps to reproduce the behavior:
> # prepare data for tpch 1s
> # 20 concurrent for tpch sql8
> # set direct memory 5g
> # when it had OutOfMemoryException , stopped all sql.
> # finding memory leak
> *Expected behavior*
> (1)i set \{DRILL_MAX_DIRECT_MEMORY:-"5G"}
> (2) i run sql8 (sql detail as Additional context) with 20 concurrent
> (3) it had OutOfMemoryException when create hashPartion
> *Error detail, log output or screenshots*
> Unable to allocate buffer of size 262144 (rounded from 262140) due to memory
> limit (41943040). Current allocation: 20447232
>
> sql
> {code:java}
> // code placeholder
> select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
> sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
> o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
> hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
> hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
> hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
> s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey
> and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name
> = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
> '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
> all_nations group by o_year order by o_year
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8479) mergejion memory leak when exception
[
https://issues.apache.org/jira/browse/DRILL-8479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8479:
Fix Version/s: 1.21.2
> mergejion memory leak when exception
> -
>
> Key: DRILL-8479
> URL: https://issues.apache.org/jira/browse/DRILL-8479
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Critical
> Fix For: 1.21.2
>
> Attachments: 0001-mergejoin-leak.patch
>
>
> *Describe the bug*
> megerjoin leak when RecordIterator allocate memory exception with
> OutOfMemoryException{*}{*}
> {*}Steps to reproduce the behavior{*}:
> # prepare data for tpch 1s
> # set direct memory 5g
> # set planner.enable_hashjoin =false to ensure use mergejoin operator。
> # set drill.memory.debug.allocator =true (Check for memory leaks )
> # 20 concurrent for tpch sql8
> # when it had OutOfMemoryException or null EXCEPTION , stopped all sql.
> # finding memory leak
> *Expected behavior*
> when all sql sop , we should find direct memory is 0 AND could not
> find leak log like following.
> {code:java}
> Allocator(op:2:0:11:MergeJoinPOP) 100/73728/4874240/100
> (res/actual/peak/limit){code}
> *Error detail, log output or screenshots*
> {code:java}
> Unable to allocate buffer of size XX (rounded from XX) due to memory limit
> (). Current allocation: xx{code}
> [^0001-mergejoin-leak.patch]
> sql
> {code:java}
> // code placeholder
> select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
> sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
> o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
> hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
> hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
> hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
> s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey
> and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name
> = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
> '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
> all_nations group by o_year order by o_year
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8484) HashJoinPOP memory leak is caused by an oom exception when read data from Stream with container
[
https://issues.apache.org/jira/browse/DRILL-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8484:
Fix Version/s: 1.21.2
(was: 1.22.0)
> HashJoinPOP memory leak is caused by an oom exception when read data from
> Stream with container
> -
>
> Key: DRILL-8484
> URL: https://issues.apache.org/jira/browse/DRILL-8484
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
>
> *Describe the bug*
> An oom exception occurred When read data from Stream with container
> ,resulting in hashJoinPOP memory leak
> *To Reproduce*
> prepare data for tpch 1s
> # 30 concurrent for tpch sql8
> # set direct memory 5g
> # when it had OutOfMemoryException , stopped all sql.
> # finding memory leak
> *leak info*
> {code:java}
> Allocator(frag:5:0) 500/100/31067136/40041943040
> (res/actual/peak/limit)
> child allocators: 1
> Allocator(op:5:0:1:HashJoinPOP) 100/16384/22822912/41943040
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[1882757] allocator: op:5:0:1:HashJoinPOP), isOwning: true,
> size: 8192, references: 2, life: 16936270178816167..0, allocatorManager:
> [1703465, life: 16936270178813617..0] holds 4 buffers.
> DrillBuf[2041995], udle: [1703441 0..957]{code}
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8482) Assign region throw exception when some region is deployed on affinity node and some on non-affinity node
[
https://issues.apache.org/jira/browse/DRILL-8482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8482:
Fix Version/s: 1.21.2
(was: 1.22.0)
> Assign region throw exception when some region is deployed on affinity node
> and some on non-affinity node
> -
>
> Key: DRILL-8482
> URL: https://issues.apache.org/jira/browse/DRILL-8482
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
> Attachments:
> 0001-DRILL-8482-Assign-region-throw-exception-when-some-r.patch
>
>
> *[^0001-DRILL-8482-Assign-region-throw-exception-when-some-r.patch]Describe
> the bug*
> Assign region throw exception when some region is deployed on affinity
> node and some on non-affinity node。
> *To Reproduce*
> Steps to reproduce the behavior:
> #
> {code:java}
> NavigableMap regionsToScan = Maps.newTreeMap();
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[0], splits[1]),
> SERVER_A);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[1], splits[2]),
> SERVER_A);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[2], splits[3]),
> SERVER_B);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[3], splits[4]),
> SERVER_B);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[6], splits[7]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[7], splits[8]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[8], splits[9]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[9], splits[10]),
> SERVER_D);
> final List endpoints = Lists.newArrayList();
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_A).setControlPort(1234).build());
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_B).setControlPort(1234).build());
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_C).setControlPort(1234).build());
> HBaseGroupScan scan = new HBaseGroupScan();
> scan.setRegionsToScan(regionsToScan);
> scan.setHBaseScanSpec(new HBaseScanSpec(TABLE_NAME_STR, splits[0], splits[0],
> null));
> scan.applyAssignments(endpoints);{code}
> *Expected behavior*
> A has 3 regions
> B has 2 regions
> C has 3 regions
> *Error detail, log output or screenshots*
> {code:java}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.store.hbase.HBaseGroupScan.applyAssignments(HBaseGroupScan.java:283){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8482) Assign region throw exception when some region is deployed on affinity node and some on non-affinity node
[
https://issues.apache.org/jira/browse/DRILL-8482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8482.
---
Resolution: Fixed
> Assign region throw exception when some region is deployed on affinity node
> and some on non-affinity node
> -
>
> Key: DRILL-8482
> URL: https://issues.apache.org/jira/browse/DRILL-8482
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
> Attachments:
> 0001-DRILL-8482-Assign-region-throw-exception-when-some-r.patch
>
>
> *[^0001-DRILL-8482-Assign-region-throw-exception-when-some-r.patch]Describe
> the bug*
> Assign region throw exception when some region is deployed on affinity
> node and some on non-affinity node。
> *To Reproduce*
> Steps to reproduce the behavior:
> #
> {code:java}
> NavigableMap regionsToScan = Maps.newTreeMap();
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[0], splits[1]),
> SERVER_A);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[1], splits[2]),
> SERVER_A);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[2], splits[3]),
> SERVER_B);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[3], splits[4]),
> SERVER_B);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[6], splits[7]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[7], splits[8]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[8], splits[9]),
> SERVER_D);
> regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[9], splits[10]),
> SERVER_D);
> final List endpoints = Lists.newArrayList();
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_A).setControlPort(1234).build());
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_B).setControlPort(1234).build());
> endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_C).setControlPort(1234).build());
> HBaseGroupScan scan = new HBaseGroupScan();
> scan.setRegionsToScan(regionsToScan);
> scan.setHBaseScanSpec(new HBaseScanSpec(TABLE_NAME_STR, splits[0], splits[0],
> null));
> scan.applyAssignments(endpoints);{code}
> *Expected behavior*
> A has 3 regions
> B has 2 regions
> C has 3 regions
> *Error detail, log output or screenshots*
> {code:java}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.drill.exec.store.hbase.HBaseGroupScan.applyAssignments(HBaseGroupScan.java:283){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8481) Ability to query XML root attributes
[
https://issues.apache.org/jira/browse/DRILL-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8481.
---
> Ability to query XML root attributes
>
>
> Key: DRILL-8481
> URL: https://issues.apache.org/jira/browse/DRILL-8481
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - XML
>Affects Versions: 1.21.1
>Reporter: benj
>Assignee: Charles Givre
>Priority: Major
> Fix For: 1.21.2
>
>
> Hi,
> It is possible to retrieve the field attributes except those of the root
> It would be interesting to be able to retrieve the attributes found in the
> root node of XML files.
> In my common use cases, I have many XML files each containing a single XML
> frame with often one or more attributes in the root tag.
> To recover this value, I am currently forced to preprocess the files to
> "copy" this attribute into the fields of the XML record.
> Even with multiple xml records under the root, it would be useful to consider
> that the root attributes are accessible for each record
> Example (fichier aaa.xml):
> {noformat}
>
>
> blue
>
> {noformat}
> With request :
> {code:sql}
> SELECT * FROM(SELECT filename, * FROM TABLE(dfs.test.`/aaa.xml`(type=>'xml',
> dataLevel=>1)) as xml) AS x;
> {code}
> I can access to :
> * P1_SubVersion
> * P1_MID
> * P1_PN
> * P1_SL
> * P2_SubVersion
> * P2.Color
> But I can' access to :
> * PPP_Version
> * PPP_TimeStamp
> and changing the DataLevel does not solve the problem
> Regards,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8483) SpilledRecordBatch memory leak when the program threw an exception during the process of building a hash table
[
https://issues.apache.org/jira/browse/DRILL-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8483.
---
Resolution: Fixed
> SpilledRecordBatch memory leak when the program threw an exception during the
> process of building a hash table
> --
>
> Key: DRILL-8483
> URL: https://issues.apache.org/jira/browse/DRILL-8483
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
>
> During the process of reading data from disk to building hash tables in
> memory, if an exception is thrown, it will result in a memory
> SpilledRecordBatch leak
> exception log as following
> {code:java}
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
> allocate buffer of size 8192 due to memory limit (41943040). Current
> allocation: 3684352
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:411)
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:270)
> at
> org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewVectorContainer(HashPartition.java:215)
> at
> org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewCurrentBatchAndHV(HashPartition.java:238)
> at
> org.apache.drill.exec.physical.impl.common.HashPartition.(HashPartition.java:165){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8484) HashJoinPOP memory leak is caused by an oom exception when read data from Stream with container
[
https://issues.apache.org/jira/browse/DRILL-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton closed DRILL-8484.
---
Resolution: Fixed
> HashJoinPOP memory leak is caused by an oom exception when read data from
> Stream with container
> -
>
> Key: DRILL-8484
> URL: https://issues.apache.org/jira/browse/DRILL-8484
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.21.2
>
>
> *Describe the bug*
> An oom exception occurred When read data from Stream with container
> ,resulting in hashJoinPOP memory leak
> *To Reproduce*
> prepare data for tpch 1s
> # 30 concurrent for tpch sql8
> # set direct memory 5g
> # when it had OutOfMemoryException , stopped all sql.
> # finding memory leak
> *leak info*
> {code:java}
> Allocator(frag:5:0) 500/100/31067136/40041943040
> (res/actual/peak/limit)
> child allocators: 1
> Allocator(op:5:0:1:HashJoinPOP) 100/16384/22822912/41943040
> (res/actual/peak/limit)
> child allocators: 0
> ledgers: 2
> ledger[1882757] allocator: op:5:0:1:HashJoinPOP), isOwning: true,
> size: 8192, references: 2, life: 16936270178816167..0, allocatorManager:
> [1703465, life: 16936270178813617..0] holds 4 buffers.
> DrillBuf[2041995], udle: [1703441 0..957]{code}
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (DRILL-8485) HashJoinPOP memory leak is caused by an oom exception when read data from InputStream
[ https://issues.apache.org/jira/browse/DRILL-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8485. --- Resolution: Fixed > HashJoinPOP memory leak is caused by an oom exception when read data from > InputStream > - > > Key: DRILL-8485 > URL: https://issues.apache.org/jira/browse/DRILL-8485 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.21.1 >Reporter: shihuafeng >Priority: Major > Fix For: 1.21.2 > > > when traversing fieldList druing read data from InputStream, if the > intermediate process throw exception,we can not release previously > constructed vectors. it result in memory leak。 > it is similar to DRILL-8484 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8485) HashJoinPOP memory leak is caused by an oom exception when read data from InputStream
[ https://issues.apache.org/jira/browse/DRILL-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8485: Fix Version/s: 1.21.2 (was: 1.21.1) > HashJoinPOP memory leak is caused by an oom exception when read data from > InputStream > - > > Key: DRILL-8485 > URL: https://issues.apache.org/jira/browse/DRILL-8485 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.21.1 >Reporter: shihuafeng >Priority: Major > Fix For: 1.21.2 > > > when traversing fieldList druing read data from InputStream, if the > intermediate process throw exception,we can not release previously > constructed vectors. it result in memory leak。 > it is similar to DRILL-8484 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8475) Update the binary disttributions LICENSE
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8475: Summary: Update the binary disttributions LICENSE (was: The binary version License and NOTICE do not comply with the corresponding terms.) > Update the binary disttributions LICENSE > > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: dependencies.txt, drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8475) Update the binary distributions LICENSE
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8475: Summary: Update the binary distributions LICENSE (was: Update the binary disttributions LICENSE) > Update the binary distributions LICENSE > --- > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: dependencies.txt, drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8352) Log noise when attempting to look up Unix user groups for a non-existent user
[
https://issues.apache.org/jira/browse/DRILL-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8352:
Issue Type: Bug (was: Improvement)
> Log noise when attempting to look up Unix user groups for a non-existent user
> -
>
> Key: DRILL-8352
> URL: https://issues.apache.org/jira/browse/DRILL-8352
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
>
> With Vault authn configured, with the consequence that Drill's users are not
> known to the local OS, messages such as the following are logged by the
> Hadoop libs used by Drill.
> {code:java}
> 09:33:30.805 [qtp2108455110-42] WARN o.a.h.s.ShellBasedUnixGroupsMapping -
> unable to return groups for user user-1-1
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
> The user name 'user-1-1' is not found. id: user-1-1: no such user
> id: user-1-1: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:387)
> at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:321)
> at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:270)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3962)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3985)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:228)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1620)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1608)
> at
> org.apache.drill.exec.util.ImpersonationUtil.hasAdminPrivileges(ImpersonationUtil.java:244)
> at
> org.apache.drill.exec.server.rest.auth.DrillRestLoginService.login(DrillRestLoginService.java:85)
> at
> org.eclipse.jetty.security.authentication.LoginAuthenticator.login(LoginAuthenticator.java:67)
> at
> org.eclipse.jetty.security.authentication.BasicAuthenticator.validateRequest(BasicAuthenticator.java:89)
> at
> org.eclipse.jetty.security.authentication.DeferredAuthentication.authenticate(DeferredAuthentication.java:66)
> at org.eclipse.jetty.server.Request.getUserPrincipal(Request.java:1715)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[
https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805432#comment-17805432
]
James Turton commented on DRILL-8475:
-
The current NOTICE file is reproduced below. It states that third party
licenses and copyright notices are to be found in LICENSE and that Class B libs
are located in jar/classb. Can we continue with this arrangement? Do we then
only need to ensure that LICENSE is fully populated and that we're still
sending all the relevant stuff to jars/classb?
{noformat}
Apache Drill
Copyright 2013-2024 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
This product depends on third party software that falls under a variety of
licenses.
All dependencies with licenses other than Apache are specified in the LICENSE
file.
For easy reference, all of the dependencies that fall under Class B licenses
Reciprocal
Licenses are included in the binary release under a separate directory located
at jar/classb.
More information on Class B licenses is available here:
http://www.apache.org/legal/3party.html
Please see LICENSE for additional copyright and licensing information.
{noformat}
> The binary version License and NOTICE do not comply with the corresponding
> terms.
> -
>
> Key: DRILL-8475
> URL: https://issues.apache.org/jira/browse/DRILL-8475
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.21.1
>Reporter: Calvin Kirs
>Assignee: James Turton
>Priority: Blocker
> Fix For: 1.21.2
>
> Attachments: drill-dep-list.txt
>
>
> I checked the latest released version, and it does not follow the
> corresponding rules[1]. This is very important and I hope it will be taken
> seriously by the PMC team. I'd be happy to do it if needed.
> [1] [https://infra.apache.org/licensing-howto.html#binary]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8475: Attachment: drill-dep-list.txt > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805061#comment-17805061 ] James Turton commented on DRILL-8475: - I've attached a sorted and deduplicated list of dependency module names taken from Drill's distribution module. > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805061#comment-17805061 ] James Turton edited comment on DRILL-8475 at 1/10/24 10:36 AM: --- I've attached a sorted and deduplicated list of 482 dependency module names taken from Drill's distribution module. was (Author: dzamo): I've attached a sorted and deduplicated list of dependency module names taken from Drill's distribution module. > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > Attachments: drill-dep-list.txt > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8475: Priority: Blocker (was: Major) > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Blocker > Fix For: 1.21.2 > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8475: Fix Version/s: 1.21.2 > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Priority: Major > Fix For: 1.21.2 > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8475) The binary version License and NOTICE do not comply with the corresponding terms.
[ https://issues.apache.org/jira/browse/DRILL-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8475: --- Assignee: James Turton > The binary version License and NOTICE do not comply with the corresponding > terms. > - > > Key: DRILL-8475 > URL: https://issues.apache.org/jira/browse/DRILL-8475 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Calvin Kirs >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > I checked the latest released version, and it does not follow the > corresponding rules[1]. This is very important and I hope it will be taken > seriously by the PMC team. I'd be happy to do it if needed. > [1] [https://infra.apache.org/licensing-howto.html#binary] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8375: Affects Version/s: 1.21.1 > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8375) Incomplete support for non-projected complex vectors
[ https://issues.apache.org/jira/browse/DRILL-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8375: Fix Version/s: 1.21.2 > Incomplete support for non-projected complex vectors > > > Key: DRILL-8375 > URL: https://issues.apache.org/jira/browse/DRILL-8375 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.21.1 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.21.2 > > > The `ResultSetLoader` implementation supports all of Drill's vector types. > However, DRILL-8188 discovered holes in support for non-projected vectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8465: Fix Version/s: (was: 1.21.2) > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Security, Storage - Iceberg >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8465: Component/s: Security > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Security, Storage - Iceberg >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8465: Affects Version/s: 1.21.1 > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Iceberg >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8465: --- Assignee: PJ Fanning > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Iceberg >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8465: Fix Version/s: 1.21.2 > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Iceberg >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8465) Check data input when loading iceberg data
[ https://issues.apache.org/jira/browse/DRILL-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8465: Summary: Check data input when loading iceberg data (was: check data input when loading iceberg data) > Check data input when loading iceberg data > -- > > Key: DRILL-8465 > URL: https://issues.apache.org/jira/browse/DRILL-8465 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Iceberg >Reporter: PJ Fanning >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8471) Bump DeltaLake Driver to Version 3.0.0
[ https://issues.apache.org/jira/browse/DRILL-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8471: Fix Version/s: 1.21.2 > Bump DeltaLake Driver to Version 3.0.0 > -- > > Key: DRILL-8471 > URL: https://issues.apache.org/jira/browse/DRILL-8471 > Project: Apache Drill > Issue Type: Task > Components: Format - DeltaLake >Affects Versions: 1.21.1 >Reporter: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > Bump DeltaLake Driver to Version 3.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8472) Bump Image Metadata Library to Latest Version
[ https://issues.apache.org/jira/browse/DRILL-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8472. --- Resolution: Fixed See https://github.com/apache/drill/pull/2862. > Bump Image Metadata Library to Latest Version > - > > Key: DRILL-8472 > URL: https://issues.apache.org/jira/browse/DRILL-8472 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 1.21.2 > > > Bump Metadata Extractor dependency to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8471) Bump DeltaLake Driver to Version 3.0.0
[ https://issues.apache.org/jira/browse/DRILL-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8471. --- Assignee: Charles Givre Resolution: Fixed > Bump DeltaLake Driver to Version 3.0.0 > -- > > Key: DRILL-8471 > URL: https://issues.apache.org/jira/browse/DRILL-8471 > Project: Apache Drill > Issue Type: Task > Components: Format - DeltaLake >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > Bump DeltaLake Driver to Version 3.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8471) Bump DeltaLake Driver to Version 3.0.0
[ https://issues.apache.org/jira/browse/DRILL-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8471: Affects Version/s: 1.21.1 > Bump DeltaLake Driver to Version 3.0.0 > -- > > Key: DRILL-8471 > URL: https://issues.apache.org/jira/browse/DRILL-8471 > Project: Apache Drill > Issue Type: Task > Components: Format - DeltaLake >Affects Versions: 1.21.1 >Reporter: Charles Givre >Priority: Major > > Bump DeltaLake Driver to Version 3.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8471) Bump DeltaLake Driver to Version 3.0.0
[ https://issues.apache.org/jira/browse/DRILL-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17803416#comment-17803416 ] James Turton commented on DRILL-8471: - See https://github.com/apache/drill/pull/2862. > Bump DeltaLake Driver to Version 3.0.0 > -- > > Key: DRILL-8471 > URL: https://issues.apache.org/jira/browse/DRILL-8471 > Project: Apache Drill > Issue Type: Task > Components: Format - DeltaLake >Reporter: Charles Givre >Priority: Major > > Bump DeltaLake Driver to Version 3.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8460) Upgrade ZooKeeper 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8460. --- Resolution: Fixed > Upgrade ZooKeeper 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket > led to test failures and so to keep this upgrade backportable the upgrade is > limited to 3.5.10. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8470) Bump MongoDB Driver to Latest Version
[ https://issues.apache.org/jira/browse/DRILL-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8470. --- Resolution: Fixed > Bump MongoDB Driver to Latest Version > - > > Key: DRILL-8470 > URL: https://issues.apache.org/jira/browse/DRILL-8470 > Project: Apache Drill > Issue Type: Task > Components: Storage - MongoDB >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.21.2 > > > Bump mongoDB driver to latest version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8415) Upgrade Jackson 2.14.3 → 2.16.1
[ https://issues.apache.org/jira/browse/DRILL-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8415: Summary: Upgrade Jackson 2.14.3 → 2.16.1 (was: Jackson 2.15) > Upgrade Jackson 2.14.3 → 2.16.1 > --- > > Key: DRILL-8415 > URL: https://issues.apache.org/jira/browse/DRILL-8415 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > I'm not advocating for an upgrade to [Jackson > 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. > 2.15.0-rc1 has just been released and 2.15.0 should be out soon. > There are some security focused enhancements including a new class called > StreamReadConstraints. The defaults on > [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] > are pretty high but it is not inconceivable that some Drill users might need > to relax them. Parsing large strings as numbers is sub-quadratic, thus the > default limit of 1000 chars or bytes (depending on input context). > When the Drill team consider upgrading to Jackson 2.15 or above, you might > also want to consider adding some way for users to configure the > StreamReadConstraints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8415) Jackson 2.15
[ https://issues.apache.org/jira/browse/DRILL-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8415: Affects Version/s: 1.21.1 > Jackson 2.15 > > > Key: DRILL-8415 > URL: https://issues.apache.org/jira/browse/DRILL-8415 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > > I'm not advocating for an upgrade to [Jackson > 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. > 2.15.0-rc1 has just been released and 2.15.0 should be out soon. > There are some security focused enhancements including a new class called > StreamReadConstraints. The defaults on > [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] > are pretty high but it is not inconceivable that some Drill users might need > to relax them. Parsing large strings as numbers is sub-quadratic, thus the > default limit of 1000 chars or bytes (depending on input context). > When the Drill team consider upgrading to Jackson 2.15 or above, you might > also want to consider adding some way for users to configure the > StreamReadConstraints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8415) Jackson 2.15
[ https://issues.apache.org/jira/browse/DRILL-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8415: Fix Version/s: 1.22.0 > Jackson 2.15 > > > Key: DRILL-8415 > URL: https://issues.apache.org/jira/browse/DRILL-8415 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > I'm not advocating for an upgrade to [Jackson > 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. > 2.15.0-rc1 has just been released and 2.15.0 should be out soon. > There are some security focused enhancements including a new class called > StreamReadConstraints. The defaults on > [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] > are pretty high but it is not inconceivable that some Drill users might need > to relax them. Parsing large strings as numbers is sub-quadratic, thus the > default limit of 1000 chars or bytes (depending on input context). > When the Drill team consider upgrading to Jackson 2.15 or above, you might > also want to consider adding some way for users to configure the > StreamReadConstraints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8437) Add Header Index Pagination
[ https://issues.apache.org/jira/browse/DRILL-8437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8437. --- Resolution: Fixed > Add Header Index Pagination > --- > > Key: DRILL-8437 > URL: https://issues.apache.org/jira/browse/DRILL-8437 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - HTTP >Affects Versions: 1.21.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > Some APIs include pagination fields in the HTTP response headers. This PR > adds a new pagination method called Header Index which supports that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Upgrade ZooKeeper 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 (was: Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0) > Upgrade ZooKeeper 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 > -- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket > led to test failures and so to keep this upgrade backportable the upgrade is > limited to 3.5.10. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Description: Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket led to test failures and so to keep this upgrade backportable the upgrade is limited to 3.5.10. (was: Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket led to test failures and so is not considered suitable for the stable branch.) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket > led to test failures and so to keep this upgrade backportable the upgrade is > limited to 3.5.10. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 (was: Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket > led to test failures and so is not considered suitable for the stable branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Description: Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket led to test failures and so is not considered suitable for the stable branch. > Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21 > - > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > Note that the initial attempt to upgrade ZooKeeper to 3.7.2 for this ticket > led to test failures and so is not considered suitable for the stable branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21 (was: Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 and Curator 5.2.0 → 5.5.0 in Drill 1.21 > - > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Description: (was: [https://github.com/apache/drill/security/dependabot/51]) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton reassigned DRILL-8460: --- Assignee: James Turton > Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: James Turton >Priority: Major > Fix For: 1.21.2 > > > [https://github.com/apache/drill/security/dependabot/51] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Fix Version/s: 1.21.2 (was: 1.22.0) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > [https://github.com/apache/drill/security/dependabot/51] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21 (was: Upgrade ZooKeeper: 3.5.7 → 3.5.10) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 in Drill 1.21 > --- > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > [https://github.com/apache/drill/security/dependabot/51] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Description: Further (was: https://github.com/apache/drill/security/dependabot/51) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 > - > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > Further -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Description: [https://github.com/apache/drill/security/dependabot/51] (was: Further) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 > - > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > [https://github.com/apache/drill/security/dependabot/51] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8460) Upgrade ZooKeeper: 3.5.7 → 3.5.10
[ https://issues.apache.org/jira/browse/DRILL-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8460: Summary: Upgrade ZooKeeper: 3.5.7 → 3.5.10 (was: Bump zookeeper jar to 3.7.2 due to CVE) > Upgrade ZooKeeper: 3.5.7 → 3.5.10 > - > > Key: DRILL-8460 > URL: https://issues.apache.org/jira/browse/DRILL-8460 > Project: Apache Drill > Issue Type: Sub-task >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Priority: Major > Fix For: 1.22.0 > > > https://github.com/apache/drill/security/dependabot/51 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8464) GitHubActions: checkout action needs to be upgraded to v4 due to node16 deprecation
[ https://issues.apache.org/jira/browse/DRILL-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8464: Fix Version/s: 1.21.2 > GitHubActions: checkout action needs to be upgraded to v4 due to node16 > deprecation > > > Key: DRILL-8464 > URL: https://issues.apache.org/jira/browse/DRILL-8464 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > The following actions uses node12 which is deprecated and will be forced to > run on node16: actions/checkout@v2. For more info: > https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8464) GitHubActions: checkout action needs to be upgraded to v4 due to node16 deprecation
[ https://issues.apache.org/jira/browse/DRILL-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8464. --- Resolution: Fixed > GitHubActions: checkout action needs to be upgraded to v4 due to node16 > deprecation > > > Key: DRILL-8464 > URL: https://issues.apache.org/jira/browse/DRILL-8464 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.21.1 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Fix For: 1.21.2 > > > The following actions uses node12 which is deprecated and will be forced to > run on node16: actions/checkout@v2. For more info: > https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
