dwsmith1983 commented on code in PR #5565:
URL: https://github.com/apache/datafusion-comet/pull/5565#discussion_r3891759797
##########
native/shuffle/src/ipc.rs:
##########
@@ -15,23 +15,52 @@
// specific language governing permissions and limitations
// under the License.
+use crate::codec_context::ShuffleDecodeContext;
use arrow::array::RecordBatch;
use arrow::ipc::reader::StreamReader;
use datafusion::common::DataFusionError;
use datafusion::error::Result;
+use std::cell::RefCell;
use std::io::{Error, ErrorKind, Read};
+thread_local! {
+ /// Backs the entry points below. They're called from many JVM task
threads; a
+ /// thread-local gets each thread context reuse without changing any
caller.
+ static DECODE_CONTEXT: RefCell<ShuffleDecodeContext> =
Review Comment:
Good catch — I had not realized SessionOnly keeps the window allocation.
Went with the bounded-size option since the thread-local has no close hook:
after every decode, error paths included, the context is dropped if its
measured size exceeds 8 MiB (common levels sit at ~1-5 MiB, so they keep reuse;
a wide-window frame pays per-frame creation like before). Added the regressions
you suggested — one decodes a wide-window level-22 frame and asserts the
workspace is released, one does the same through a decode failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
