Created a jira ticket for this: https://issues.apache.org/jira/browse/NIFI-12117
> On Sep 19, 2023, at 5:09 PM, Bryan Bende <[email protected]> wrote: > > Yes, even if we don't expose a property and pick a very large default, it is > going to require going through each extension one by one, since they all > control the creation of their own ObjectMapper. > > We may be able to create a shared utility module that they could all depend > on with a factory to create an ObjectMapper. > > For this particular case it came from JoltTransformJson which uses a jolt > class called JsonUtils to call > final Object inputJson; > try (final InputStream in = session.read(original)) { > inputJson = JsonUtils.jsonToObject(in); > } > > It has a helper to create a util instance with your own ObjectMapper, so we'd > have to switch to calling that: > > JsonUtil customJsonUtil( ObjectMapper mapper ) > > > On Tue, Sep 19, 2023 at 5:05 PM David Handermann <[email protected] > <mailto:[email protected]>> wrote: > Changing the default value to an unlimited number would negate the protection > that the new restriction provides. > > It should not be necessary to make global changes, but focusing on Processor > and Controller Service references would be a good starting point for > introducing configurable sizes. > > Regards, > David Handermann > > On Tue, Sep 19, 2023 at 3:45 PM Chris Sampson <[email protected] > <mailto:[email protected]>> wrote: > Probably worth noting that the Jackson ObjectMapper is used in lots of places > across the code base - 182 hits in GitHub on current main branch (some of > those are duplicates within classes). > > So it could be a significant change to update and test all places where it's > currently used. The string length limit might not be limited to the > ObjectMapper either, so that would need checking and working through. > > A more centralised configuration might be worthwhile if possible. > > > On Tue, 19 Sept 2023, 20:42 Gregory M. Foreman, > <[email protected] <mailto:[email protected]>> > wrote: > Defaulting maxStringLength to Integer.MAX_VALUE would be my vote. It would > work as it had previously and as expected. I do not currently see a use case > where my client would want to restrict the size of a json string…using > fileSize to route is adequate. > >> On Sep 18, 2023, at 4:53 PM, Bryan Bende <[email protected] >> <mailto:[email protected]>> wrote: >> >> It looks like Jackson 2.15 introduced a max length for a single string value >> inside of the json document which defaults to 20MB. >> >> It can be configured on the ObjectMapper's factory [1] like: >> >> objectMapper.getFactory().setStreamReadConstraints(StreamReadConstraints.builder().maxStringLength(40000000).build())); >> >> If we want to make this configurable, we'd probably have to expose a >> property in any controller-service/processor that does json parsing with an >> ObjectMapper. >> >> [1] >> https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940 >> >> <https://github.com/spring-projects/spring-boot/issues/34709#issuecomment-1482939940> >> On Mon, Sep 18, 2023 at 4:23 PM Gregory M. Foreman >> <[email protected] <mailto:[email protected]>> >> wrote: >> Yes >> >> Sent from my iPhone >> >>> On Sep 18, 2023, at 2:29 PM, Greene (US), Geoffrey N via users >>> <[email protected] <mailto:[email protected]>> wrote: >>> >>> >>> Is this what you are referring to? 20MB string values vs 20 MB document >>> size… >>> >>> >>> >>> https://github.com/FasterXML/jackson-core/issues/1082 >>> <https://github.com/FasterXML/jackson-core/issues/1082> >>> >>> >>> >>> >>> From: Joe Witt <[email protected] <mailto:[email protected]>> >>> Sent: Monday, September 18, 2023 2:25 PM >>> To: [email protected] <mailto:[email protected]> >>> Subject: [EXTERNAL] Re: Large json string field errors >>> >>> >>> >>> EXT email: be mindful of links/attachments. >>> >>> >>> >>> Greg >>> >>> >>> >>> Can you share details on how you're seeing this limit? Is there a >>> resulting stack trace or other output for it in the nifi-app.log? What >>> version of NiFi are you on? >>> >>> >>> >>> Thanks >>> >>> >>> >>> On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>> Hello: >>> >>> It appears Jackson recently set a 20mb upper boundary on json string field >>> length. Is there a way to remove/override this limit in Nifi? >>> >>> Thanks, >>> Greg >>> >
