LB-Yu commented on issue #321:
URL: https://github.com/apache/fluss/issues/321#issuecomment-3135828492
Hi @loserwang1024. Wha's the progress about this ? I also meet this bug in
Flink job.
```
2025-07-30 16:42:18,191 [xxx[27]: Writer (153/1024)#63] INFO
com.alibaba.fluss.client.write.WriterClient [] - Closing
writer.
2025-07-30 16:42:18,192 [xxx[27]: Writer (153/1024)#63] ERROR
com.alibaba.fluss.shaded.arrow.org.apache.arrow.memory.BaseAllocator [] -
Memory was leaked by query. Memory leaked: (29574584)
Allocator(ROOT) 0/29574712/29574712/9223372036854775807
(res/actual/peak/limit)
2025-07-30 16:42:18,192 [xxx[27]: Writer (153/1024)#63] WARN
com.alibaba.fluss.flink.sink.writer.FlinkSinkWriter [] - Exception
occurs while closing Fluss Connection.
java.lang.IllegalStateException: Memory was leaked by query. Memory leaked:
(29574584)
Allocator(ROOT) 0/29574712/29574712/9223372036854775807
(res/actual/peak/limit)
at
com.alibaba.fluss.shaded.arrow.org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:477)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at
com.alibaba.fluss.shaded.arrow.org.apache.arrow.memory.RootAllocator.close(RootAllocator.java:29)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at
com.alibaba.fluss.client.write.RecordAccumulator.close(RecordAccumulator.java:926)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at com.alibaba.fluss.client.write.Sender.initiateClose(Sender.java:600)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at
com.alibaba.fluss.client.write.WriterClient.close(WriterClient.java:290)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at
com.alibaba.fluss.client.FlussConnection.close(FlussConnection.java:180)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at
com.alibaba.fluss.flink.sink.writer.FlinkSinkWriter.close(FlinkSinkWriter.java:180)
~[blob_p-02e183c52725f33401961c6310dc5758b1e7e11d-6ff4e20658cfe414b942d44d5c607f1d:0.6-ali-3-SNAPSHOT]
at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:234)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:223)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.close(SinkWriterOperator.java:249)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1275)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
~[ververica-connector-vvp-1.17-vvr-8.0.11-1-aidata-SNAPSHOT-jar-with-dependencies.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:1183)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$1(Task.java:950)
~[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:968)
[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:950)
[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:754)
[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:569)
[flink-dist-1.17-vvr-8.0.11-1-aidata-SNAPSHOT.jar:1.17-vvr-8.0.11-1-aidata-SNAPSHOT]
at java.lang.Thread.run(Thread.java:879) [?:1.8.0_372]
```
I think we can close() only after all the batches have been sent
successfully. If you don't have time to fix this issue right now, I can take it
over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
