[
https://issues.apache.org/jira/browse/AVRO-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4307:
-------------------------------
Description:
BlockDecoder decompresses blocks asynchronously. When a block callback fails
(corrupt block, invalid sync marker, unknown codec, or the decompression size
guard from AVRO-4287), the failure path called emit(*error*) once per failing
block with a bare emit and never destroyed the stream.
Confirmed empirically on plain master (no dependency on the decompression-limit
work) with a 394 KB multi-block file decoded through the real createFileDecoder
path (fs.createReadStream(path).pipe(new BlockDecoder())) using a custom codec
that always errors:
* errorCount: 947 -- one *error* event per failing block (an error storm)
* bytesReadAfterFirstError: 65536 -- the upstream file stream kept reading a
file already known to be bad
* srcDestroyed: false, decDestroyed: false -- neither stream was torn down;
the file descriptor leaked
Root cause is in BlockDecoder._createBlockCallback (bare emit, no destroy, no
guard), which predates and is independent of AVRO-4287. createFileDecoder using
plain .pipe() compounds it: a destination error only unpipes the source, it
never destroys it.
Fix: route fatal errors through a guarded _onError() that calls destroy(err) so
exactly one error surfaces and the stream unwinds; switch createFileDecoder to
stream.pipeline() so the source file stream is destroyed and its descriptor
released. After the fix: errorCount 1, bytesReadAfterFirstError 0, source and
decoder both destroyed.
was:When a BlockDecoder used via createFileDecoder
(ReadStream.pipe(BlockDecoder)) hits a decompression/size error, it emits an
error event but does not destroy itself. The upstream file stream can continue
reading and feeding data after the error, doing unnecessary I/O/CPU work and
possibly emitting multiple errors. BlockDecoder should tear down (destroy) on a
fatal error so the pipe unwinds. This touches the hand-rolled finish/push(null)
lifecycle in files.js and needs careful testing; split out from AVRO-4287.
> [javascript] BlockDecoder should destroy itself on decompression error to
> stop upstream I/O
> -------------------------------------------------------------------------------------------
>
> Key: AVRO-4307
> URL: https://issues.apache.org/jira/browse/AVRO-4307
> Project: Apache Avro
> Issue Type: Improvement
> Components: javascript
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
>
> BlockDecoder decompresses blocks asynchronously. When a block callback fails
> (corrupt block, invalid sync marker, unknown codec, or the decompression size
> guard from AVRO-4287), the failure path called emit(*error*) once per failing
> block with a bare emit and never destroyed the stream.
> Confirmed empirically on plain master (no dependency on the
> decompression-limit work) with a 394 KB multi-block file decoded through the
> real createFileDecoder path (fs.createReadStream(path).pipe(new
> BlockDecoder())) using a custom codec that always errors:
> * errorCount: 947 -- one *error* event per failing block (an error storm)
> * bytesReadAfterFirstError: 65536 -- the upstream file stream kept reading a
> file already known to be bad
> * srcDestroyed: false, decDestroyed: false -- neither stream was torn down;
> the file descriptor leaked
> Root cause is in BlockDecoder._createBlockCallback (bare emit, no destroy, no
> guard), which predates and is independent of AVRO-4287. createFileDecoder
> using plain .pipe() compounds it: a destination error only unpipes the
> source, it never destroys it.
> Fix: route fatal errors through a guarded _onError() that calls destroy(err)
> so exactly one error surfaces and the stream unwinds; switch
> createFileDecoder to stream.pipeline() so the source file stream is destroyed
> and its descriptor released. After the fix: errorCount 1,
> bytesReadAfterFirstError 0, source and decoder both destroyed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
