Yes, even if we don't expose a property and pick a very large default, it
is going to require going through each extension one by one, since they all
control the creation of their own ObjectMapper.
We may be able to create a shared utility module that they could all depend
on with a factory to create an ObjectMapper.
For this particular case it came from JoltTransformJson which uses a jolt
class called JsonUtils to call
final Object inputJson;
try (final InputStream in = session.read(original)) {
inputJson = JsonUtils.jsonToObject(in);
}
It has a helper to create a util instance with your own ObjectMapper,
so we'd have to switch to calling that:
JsonUtil customJsonUtil( ObjectMapper mapper )
On Tue, Sep 19, 2023 at 5:05 PM David Handermann <
[email protected]> wrote:
> Changing the default value to an unlimited number would negate the
> protection that the new restriction provides.
>
> It should not be necessary to make global changes, but focusing on
> Processor and Controller Service references would be a good starting point
> for introducing configurable sizes.
>
> Regards,
> David Handermann
>
> On Tue, Sep 19, 2023 at 3:45 PM Chris Sampson <[email protected]>
> wrote:
>
>> Probably worth noting that the Jackson ObjectMapper is used in lots of
>> places across the code base - 182 hits in GitHub on current main branch
>> (some of those are duplicates within classes).
>>
>> So it could be a significant change to update and test all places where
>> it's currently used. The string length limit might not be limited to the
>> ObjectMapper either, so that would need checking and working through.
>>
>> A more centralised configuration might be worthwhile if possible.
>>
>>
>> On Tue, 19 Sept 2023, 20:42 Gregory M. Foreman, <
>> [email protected]> wrote:
>>
>>> Defaulting maxStringLength to Integer.MAX_VALUE would be my vote. It
>>> would work as it had previously and as expected. I do not currently see a
>>> use case where my client would want to restrict the size of a json
>>> string…using fileSize to route is adequate.
>>>
>>> On Sep 18, 2023, at 4:53 PM, Bryan Bende <[email protected]> wrote:
>>>
>>> It looks like Jackson 2.15 introduced a max length for a single string
>>> value inside of the json document which defaults to 20MB.
>>>
>>> It can be configured on the ObjectMapper's factory [1] like:
>>>
>>>
>>> objectMapper.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(40000000).build()));
>>>
>>> If we want to make this configurable, we'd probably have to expose a
>>> property in any controller-service/processor that does json parsing with an
>>> ObjectMapper.
>>>
>>> [1]
>>> https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940
>>>
>>> On Mon, Sep 18, 2023 at 4:23 PM Gregory M. Foreman <
>>> [email protected]> wrote:
>>>
>>>> Yes
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users <
>>>> [email protected]> wrote:
>>>>
>>>>
>>>>
>>>> Is this what you are referring to? 20MB string values vs 20 MB document
>>>> size…
>>>>
>>>>
>>>>
>>>> https://github.com/FasterXML/jackson-core/issues/1082
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* Joe Witt <[email protected]>
>>>> *Sent:* Monday, September 18, 2023 2:25 PM
>>>> *To:* [email protected]
>>>> *Subject:* [EXTERNAL] Re: Large json string field errors
>>>>
>>>>
>>>>
>>>> EXT email: be mindful of links/attachments.
>>>>
>>>>
>>>>
>>>>
>>>> Greg
>>>>
>>>>
>>>>
>>>> Can you share details on how you're seeing this limit? Is there a
>>>> resulting stack trace or other output for it in the nifi-app.log? What
>>>> version of NiFi are you on?
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman <
>>>> [email protected]> wrote:
>>>>
>>>> Hello:
>>>>
>>>> It appears Jackson recently set a 20mb upper boundary on json string
>>>> field length. Is there a way to remove/override this limit in Nifi?
>>>>
>>>> Thanks,
>>>> Greg
>>>>
>>>>
>>>
