jamf-software opened a new issue, #26185: URL: https://github.com/apache/pulsar/issues/26185
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment Environment │ Pulsar │ 4.0.12 │ Java │ 21 │ Jackson BOM │ 2.18.8 │ jersey-media-json-jackson │ 2.42 │ OS │ Linux (Docker) Works on: Pulsar 4.0.11 (Jackson BOM 2.18.6) Fails on: Pulsar 4.0.12 (Jackson BOM 2.18.8) when any broker interceptor NAR is loaded ### Issue Description When a broker interceptor NAR is loaded in Pulsar 4.0.12, admin API calls with a JSON request body (e.g. resetSubscriptionToPosition, setBookieRack) fail with HTTP 400: javax.ws.rs.BadRequestException: HTTP 400 Trailing token (of type START_OBJECT) found after value (bound as `org.apache.pulsar.client.impl.ResetCursorData`): not allowed as per `DeserializationFeature.FAIL_ON_TRAILING_TOKENS` The same NAR works correctly on Pulsar 4.0.11. Without the NAR, 4.0.12 works correctly. This is a regression introduced between 4.0.11 and 4.0.12. Root cause analysis PreInterceptFilter is only registered when a BrokerInterceptor is loaded. It wraps every incoming HTTP request in a RequestWrapper that buffers the body. The wrapped request is then passed to Jersey for deserialization. jackson-jaxrs-providers (used by Jersey) enables FAIL_ON_TRAILING_TOKENS unconditionally via JaxRSFeature.READ_FULL_STREAM (default true since 2.15.0): // ProviderBase._configForReading(): if (JaxRSFeature.READ_FULL_STREAM.enabledIn(_jaxRSFeatures)) { r = r.withFeatures(DeserializationFeature.FAIL_ON_TRAILING_TOKENS); } This means any trailing bytes after the valid JSON in the body will cause a 400. The feature was active in 4.0.11 as well, but no trailing bytes were present. Between 4.0.11 and 4.0.12, the Jackson BOM bumped from 2.18.6 to 2.18.8. The error appears on the forwarded request leg (note the AsyncHttpClient thread) — when broker A forwards an admin call to the partition-owning broker B, broker B's PreInterceptFilter receives the forwarded request and Jersey rejects it. This suggests the forwarded request body may be constructed differently in 4.0.12, resulting in trailing content. Investigation confirmed: - PreInterceptFilter and RequestWrapper class bytecode is identical between 4.0.11 and 4.0.12 - The RequestWrapper IOUtils.toByteArray(Reader, Charset) round-trip is byte-for-byte lossless (verified by unit test) - FAIL_ON_TRAILING_TOKENS is not referenced anywhere in Pulsar's own codebase — it is enabled by the JAX-RS provider - Jersey and Jetty versions are unchanged between 4.0.11 and 4.0.12 The precise source of the trailing content in the forwarded request body is still under investigation. ### Error messages ```text Error log [AsyncHttpClient-50-1] WARN org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [persistent://public/default/my-topic] got an error while copying the subscription to the partition javax.ws.rs.BadRequestException: HTTP 400 Trailing token (of type START_OBJECT) found after value (bound as `org.apache.pulsar.client.impl.ResetCursorData`): not allowed as per `DeserializationFeature.FAIL_ON_TRAILING_TOKENS` ``` ### Reproducing the issue 1. Build any broker interceptor NAR and load it via brokerInterceptors config 2. Create a partitioned topic 3. Call resetSubscriptionToPosition (or any admin API that POSTs a JSON body) on that topic 4. Observe HTTP 400 in the broker logs, originating from the forwarded request to the partition-owning broker ### Additional information Possible fix RequestWrapper reads the body using a charset round-trip: IOUtils.toByteArray(new InputStreamReader(request.getInputStream()), Charset.defaultCharset()) This could be simplified to a direct stream read with no charset conversion: IOUtils.toByteArray(request.getInputStream()) This is safer and avoids any potential encoding-related body mutation. However, since we confirmed the round-trip is lossless for ASCII JSON, the real fix likely needs to address how the forwarded request body is constructed in the partition-copy path. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
