[
https://issues.apache.org/jira/browse/HTTPCORE-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated HTTPCORE-799:
------------------------------
Affects Version/s: 5.4.3
Description:
h2. Description
Several repeatable async entity producers do not fully reset their
per-production state when {{releaseResources()}} is called before the current
production has completed.
h3. Buffered output
{{FileEntityProducer}}, {{PathEntityProducer}},
{{AbstractCharAsyncEntityProducer}}, and {{AbstractBinAsyncEntityProducer}} can
retain bytes that the data channel did not accept. Although the underlying
source is reset for a replay, the internal staging buffer still contains the
unwritten tail, so those stale bytes are emitted before the complete entity.
{{StringAsyncEntityProducer}} is affected through
{{AbstractCharAsyncEntityProducer}}. Repeatable custom binary producers can be
affected through {{AbstractBinAsyncEntityProducer}}.
For example, with content {{abcdef}}:
# The producer reads {{abcdef}} and the data channel accepts only {{abc}}.
# {{releaseResources()}} is called, simulating an interrupted request.
# The same repeatable producer is used again.
# The second production emits {{defabcdef}} instead of {{abcdef}}.
h3. Digest state
{{DigestingEntityProducer.releaseResources()}} delegates to the wrapped
producer but does not reset its {{MessageDigest}}. If the first production
writes {{abc}} before interruption and the replay writes {{abcdef}}, the replay
body can be correct while {{getDigest()}} and the {{digest}} trailer contain
the MD5 of {{abcabcdef}} ({{89091e2d8ada6ecb23f3d2b71fb1764b}}) instead of the
MD5 of {{abcdef}} ({{e80b5017098950fc58aad83c8c14978e}}).
h2. Context
This was exposed while testing an HTTP/2 authentication retry after a server
returned a {{401}} response and then reset the stream with
{{RST_STREAM(NO_ERROR)}}. The request producer was released mid-write and
reused for the retry. See
[HTTPCLIENT-2433|https://issues.apache.org/jira/browse/HTTPCLIENT-2433] for the
related HttpClient issue.
The HttpCore problem is not specific to HTTP/2 or authentication; it can occur
whenever a repeatable producer is released after a partial write and then
reused.
h2. Expected behavior
Calling {{releaseResources()}} should discard pending output and reset all
per-production digest state. A subsequent production should emit exactly the
original entity, and its digest should cover only that production.
h2. Actual behavior
Buffered but unwritten bytes can be prepended to the next production, and bytes
accepted during an interrupted production can be included in the replay's
digest.
h2. Affected code
* {{org.apache.hc.core5.http.nio.entity.FileEntityProducer}}
* {{org.apache.hc.core5.http.nio.entity.PathEntityProducer}}
* {{org.apache.hc.core5.http.nio.entity.AbstractCharAsyncEntityProducer}}
(including {{StringAsyncEntityProducer}})
* {{org.apache.hc.core5.http.nio.entity.AbstractBinAsyncEntityProducer}} (for
repeatable subclasses)
* {{org.apache.hc.core5.http.nio.entity.DigestingEntityProducer}}
The problem is present in HttpCore 5.4.3, 5.5-beta2, and the current {{5.4.x}}
and {{master}} branches.
h2. Proposed fix
Clear the internal staging buffers when the file, path, character, and binary
producers are released. Reset {{DigestingEntityProducer}}'s {{MessageDigest}}
before releasing its wrapped producer, while preserving the completed
{{digest}} value returned by {{getDigest()}}.
Add regression tests that interrupt production after a partial channel write,
release the producer, and verify that the next production contains the original
entity exactly once and calculates its digest only from the replayed bytes.
Implementation:
[apache/httpcomponents-core#698|https://github.com/apache/httpcomponents-core/pull/698]
was:
h2. Description
{{FileEntityProducer}} and {{PathEntityProducer}} are repeatable, but they do
not fully reset their internal state when {{releaseResources()}} is called
before the current production has completed.
Both implementations close and clear the reference to the underlying file
channel and reset {{eof}}, but leave any unwritten bytes in their internal
{{ByteBuffer}}. On the next production, the file is reopened at position zero
while those stale bytes remain buffered. The stale suffix is therefore emitted
before the complete file.
For example, with file content {{abcdef}}:
# The producer reads {{abcdef}} and the data channel accepts only {{abc}}.
# {{releaseResources()}} is called, simulating an interrupted request.
# The same repeatable producer is used again.
# The second production emits {{defabcdef}} instead of {{abcdef}}.
This was exposed while testing an HTTP/2 authentication retry after the server
completed a {{401}} response and reset the request stream with
{{RST_STREAM(NO_ERROR)}}. That graceful reset releases the request producer
before the higher layer retries it. See
[HTTPCLIENT-2433|https://issues.apache.org/jira/browse/HTTPCLIENT-2433] for the
related HttpClient issue.
h2. Expected behavior
Calling {{releaseResources()}} should discard pending bytes and restore a
repeatable producer to the beginning of its source. A subsequent production
should emit exactly the original file.
h2. Actual behavior
Any bytes that were buffered but not accepted by the previous data channel are
prepended to the next production.
h2. Affected code
* {{org.apache.hc.core5.http.nio.entity.FileEntityProducer}}
* {{org.apache.hc.core5.http.nio.entity.PathEntityProducer}}
The problem is present in HttpCore 5.5-beta2 and current {{master}}.
h2. Proposed fix
Clear the internal {{ByteBuffer}} in each producer's {{releaseResources()}}
method. Add regression tests that interrupt production after a partial channel
write, release the producer, and verify that its next production contains the
original file exactly once.
Summary: Async entity producers retain stale state after
interrupted writes (was: FileEntityProducer and PathEntityProducer retain
stale buffered data after an interrupted write)
> Async entity producers retain stale state after interrupted writes
> ------------------------------------------------------------------
>
> Key: HTTPCORE-799
> URL: https://issues.apache.org/jira/browse/HTTPCORE-799
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore
> Affects Versions: 5.4.3, 5.5-beta2
> Reporter: Matthias
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Description
> Several repeatable async entity producers do not fully reset their
> per-production state when {{releaseResources()}} is called before the current
> production has completed.
> h3. Buffered output
> {{FileEntityProducer}}, {{PathEntityProducer}},
> {{AbstractCharAsyncEntityProducer}}, and {{AbstractBinAsyncEntityProducer}}
> can retain bytes that the data channel did not accept. Although the
> underlying source is reset for a replay, the internal staging buffer still
> contains the unwritten tail, so those stale bytes are emitted before the
> complete entity.
> {{StringAsyncEntityProducer}} is affected through
> {{AbstractCharAsyncEntityProducer}}. Repeatable custom binary producers can
> be affected through {{AbstractBinAsyncEntityProducer}}.
> For example, with content {{abcdef}}:
> # The producer reads {{abcdef}} and the data channel accepts only {{abc}}.
> # {{releaseResources()}} is called, simulating an interrupted request.
> # The same repeatable producer is used again.
> # The second production emits {{defabcdef}} instead of {{abcdef}}.
> h3. Digest state
> {{DigestingEntityProducer.releaseResources()}} delegates to the wrapped
> producer but does not reset its {{MessageDigest}}. If the first production
> writes {{abc}} before interruption and the replay writes {{abcdef}}, the
> replay body can be correct while {{getDigest()}} and the {{digest}} trailer
> contain the MD5 of {{abcabcdef}} ({{89091e2d8ada6ecb23f3d2b71fb1764b}})
> instead of the MD5 of {{abcdef}} ({{e80b5017098950fc58aad83c8c14978e}}).
> h2. Context
> This was exposed while testing an HTTP/2 authentication retry after a server
> returned a {{401}} response and then reset the stream with
> {{RST_STREAM(NO_ERROR)}}. The request producer was released mid-write and
> reused for the retry. See
> [HTTPCLIENT-2433|https://issues.apache.org/jira/browse/HTTPCLIENT-2433] for
> the related HttpClient issue.
> The HttpCore problem is not specific to HTTP/2 or authentication; it can
> occur whenever a repeatable producer is released after a partial write and
> then reused.
> h2. Expected behavior
> Calling {{releaseResources()}} should discard pending output and reset all
> per-production digest state. A subsequent production should emit exactly the
> original entity, and its digest should cover only that production.
> h2. Actual behavior
> Buffered but unwritten bytes can be prepended to the next production, and
> bytes accepted during an interrupted production can be included in the
> replay's digest.
> h2. Affected code
> * {{org.apache.hc.core5.http.nio.entity.FileEntityProducer}}
> * {{org.apache.hc.core5.http.nio.entity.PathEntityProducer}}
> * {{org.apache.hc.core5.http.nio.entity.AbstractCharAsyncEntityProducer}}
> (including {{StringAsyncEntityProducer}})
> * {{org.apache.hc.core5.http.nio.entity.AbstractBinAsyncEntityProducer}} (for
> repeatable subclasses)
> * {{org.apache.hc.core5.http.nio.entity.DigestingEntityProducer}}
> The problem is present in HttpCore 5.4.3, 5.5-beta2, and the current
> {{5.4.x}} and {{master}} branches.
> h2. Proposed fix
> Clear the internal staging buffers when the file, path, character, and binary
> producers are released. Reset {{DigestingEntityProducer}}'s {{MessageDigest}}
> before releasing its wrapped producer, while preserving the completed
> {{digest}} value returned by {{getDigest()}}.
> Add regression tests that interrupt production after a partial channel write,
> release the producer, and verify that the next production contains the
> original entity exactly once and calculates its digest only from the replayed
> bytes.
> Implementation:
> [apache/httpcomponents-core#698|https://github.com/apache/httpcomponents-core/pull/698]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
