Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

hello.

First of all, I apologize for sending spam emails.

But when I looked at the 1.23.2 source, it was the same. (ToT)

have a nice day

Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

Hello.
I found that there is no problem when writing with pretty json.
I am currently guessing that there is no problem when loading with bytes
like the pretty json part.

parsed = json.loads(flowFile.getContentsAsBytes())

Can this part be included in the next bug fix if it is correct?
I will look into it a little more.
Have a nice day.

Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

hello.

Thank you for taking the time to look into the issue.

I found that whether or not an error occurs changes depending on the
version of JsonTreeReader.

I will also look into the differences and let you know if I find any.

Thank you very much again.

Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

hello.

I guess the source I attached above has nothing to do with it.

if (prettyPrint) {

generator.useDefaultPrettyPrinter();

} else if (OutputGrouping.OUTPUT_ONELINE.equals(outputGrouping)) {

// Use a minimal pretty printer with a newline object
separator, will output one JSON object per line

generator.setPrettyPrinter(new MinimalPrettyPrinter("\n"));

}

Perhaps the behavior is different here.


Have a nice day.

Re: hello. I would like to inquire about an error when upgrading the version.

[Pierre Villard]

Hi,

I downloaded NiFi 1.23 and I confirm that the problem is not happening
there.

However I can also confirm that this is not because of the Jackson version.
If I use NiFi 2 and an old version of Jackson (the one used in NiFi 1.23),
the problem will happen.

I will keep looking into it to understand what change could explain this
difference.

Thanks,
Pierre

Le mer. 6 nov. 2024 à 05:28, 박정화  a écrit :

> hello.
>
> The data provided appears to be sufficient for testing as we have tested it
> several times and sent the problematic data.
>
> I think it's almost certain because we've tested it multiple times on Linux
> and Mac and only the problematic data has been clearly extracted.
>
> I'm almost 100% sure it doesn't happen in version 1.23.2.
>
> If you test with the data provided in versions 1.27.0 and 1.23.2, you will
> probably be able to see that it works differently.
>
> Since the data is wrong, it is correct to change the data itself, but in
> reality it may be difficult, so we are looking for a way to solve this
> problem without losing performance.
>
> nifi 2.0.0 version has been released, but I am worried because if this
> problem is not resolved, I may not be able to upgrade to version 2.x. (1.x
> version of nar does not run in 2.x)
>

Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

hello.

The data provided appears to be sufficient for testing as we have tested it
several times and sent the problematic data.

I think it's almost certain because we've tested it multiple times on Linux
and Mac and only the problematic data has been clearly extracted.

I'm almost 100% sure it doesn't happen in version 1.23.2.

If you test with the data provided in versions 1.27.0 and 1.23.2, you will
probably be able to see that it works differently.

Since the data is wrong, it is correct to change the data itself, but in
reality it may be difficult, so we are looking for a way to solve this
problem without losing performance.

nifi 2.0.0 version has been released, but I am worried because if this
problem is not resolved, I may not be able to upgrade to version 2.x. (1.x
version of nar does not run in 2.x)

Re: hello. I would like to inquire about an error when upgrading the version.

[Pierre Villard]

Hi,

I looked into it and this seems to be a rather interesting situation :)

First of all, this does not relate to Jackson and I don't think there is a
regression in NiFi versions. I think you may have received specific data
for the first time that is causing this issue. Are you absolutely sure that
the exact same data is working in NiFi 1.23?

It is very easy to simulate the problem in a unit test on the NiFi side and
the problem appears as soon as you're using a JSON Reader and JSON Writer
with the data that you shared.

Specifically the unicode character \uD83D is problematic because, according
to this RFC [1]:
"The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form
(as surrogate pairs) and do not directly represent characters."

So it looks like you need UTF-16 when writing data - currently we use UTF-8
as the default. Do you have more details on how this data is generated?
Is {\"name\":\"\uD83D*\"} the exact data you're receiving? or do you have
additional characters in that field? If yes, can you give the full content
of the name field?

[1] https://datatracker.ietf.org/doc/html/rfc3629#page-4

Thanks,
Pierre

Le lun. 4 nov. 2024 à 14:50, 박정화  a écrit :

> hello
>
> I waited because I thought you didn't receive a reply, but I saw that you
> had responded through the following link.
>
> https://lists.apache.org/thread/qgc2ffgwv00htpykvmvxcq4fx7df2qkx
>
>
> 
>
> First of all, I would like to thank you.  I downgraded the library version
> as you said, and the error did not occur.
>
> I am attaching the error data in case it may be of help.
>
> I know the data is in the wrong format, but I was worried because it
> affected the overall data loading, and the advice you sent was very
> helpful.
>
>
> Thank you again.  have a good day.
>
>
> Data Sample
>
>
> {"timestamp":"2024-10-20T16:10:00.322+09:00","data":"[{\"name\":\"\uD83D*\"}]"}
>
>
> PartitionRecord property
> mmddhh : format(toDate( /timestamp, "-MM-dd'T'HH:mm:ss.SSSXXX",
> "Asia/Seoul"),"MMddHH")
>
> 2024년 10월 22일 (화) 오후 10:14, 박정화 (Bank) 님이 작성:
>
> > hello.
> >
> > I am using nifi well.
> >
> > I recently upgraded the version from 1.23.2 to 1.27.0 and encountered the
> > following error in the PartitionRecord process.
> >
> >
> >
> > com.fasterxml.jackson.core.JsonGenerationException: Incomplete surrogate
> > pair: first char 0xD83D, second 0x002A
> >
> >
> >
> > It seems to be a serrogate pair issue, but it seems to have been caused
> by
> > masking some characters on our side.
> >
> > Nevertheless, I wish it were possible to process json like in previous
> > versions or like python.  Is there any way?
> >
> > Could it be that the error occurring differently from the previous
> version
> > is related to the update of the jackson library?
> >
> > The jackson version of nifi 1.23.2 is 2.15.2, and the jackson version of
> > nifi 1.27.0 is 2.17.1, so I thought it might be a problem.
> >
> > If so, we can change the jackson version and compile it, but we will need
> > to continuously change the nifi version in the future.  Could you please
> > give me some advice on this?
> >
> > thank you  have a good day
> >
>
>
> --
> *박정화*
> Data Engineer, Data Platform Team (뱅크)
> 010-8000-6713 | vesuv...@toss.im
> 서울시 강남구 테헤란로 131, 한국지식재산센터 13층 (06133)
> [image: Toss BI]
>

Re: hello. I would like to inquire about an error when upgrading the version.

[박정화]

hello

I waited because I thought you didn't receive a reply, but I saw that you
had responded through the following link.

https://lists.apache.org/thread/qgc2ffgwv00htpykvmvxcq4fx7df2qkx




First of all, I would like to thank you.  I downgraded the library version
as you said, and the error did not occur.

I am attaching the error data in case it may be of help.

I know the data is in the wrong format, but I was worried because it
affected the overall data loading, and the advice you sent was very helpful.


Thank you again.  have a good day.


Data Sample

{"timestamp":"2024-10-20T16:10:00.322+09:00","data":"[{\"name\":\"\uD83D*\"}]"}


PartitionRecord property
mmddhh : format(toDate( /timestamp, "-MM-dd'T'HH:mm:ss.SSSXXX",
"Asia/Seoul"),"MMddHH")

2024년 10월 22일 (화) 오후 10:14, 박정화 (Bank) 님이 작성:

> hello.
>
> I am using nifi well.
>
> I recently upgraded the version from 1.23.2 to 1.27.0 and encountered the
> following error in the PartitionRecord process.
>
>
>
> com.fasterxml.jackson.core.JsonGenerationException: Incomplete surrogate
> pair: first char 0xD83D, second 0x002A
>
>
>
> It seems to be a serrogate pair issue, but it seems to have been caused by
> masking some characters on our side.
>
> Nevertheless, I wish it were possible to process json like in previous
> versions or like python.  Is there any way?
>
> Could it be that the error occurring differently from the previous version
> is related to the update of the jackson library?
>
> The jackson version of nifi 1.23.2 is 2.15.2, and the jackson version of
> nifi 1.27.0 is 2.17.1, so I thought it might be a problem.
>
> If so, we can change the jackson version and compile it, but we will need
> to continuously change the nifi version in the future.  Could you please
> give me some advice on this?
>
> thank you  have a good day
>


-- 
*박정화*
Data Engineer, Data Platform Team (뱅크)
010-8000-6713 | vesuv...@toss.im
서울시 강남구 테헤란로 131, 한국지식재산센터 13층 (06133)
[image: Toss BI]

Re: hello. I would like to inquire about an error when upgrading the version.

[Pierre Villard]

Hi,

Is there any chance you can provide some crafted dummy data to reproduce
the issue on our side? If this is due to an upgrade in some dependencies in
NiFi, we might want to look into it.

Note that you can always download the NARs from previous versions of NiFi
and use those with a more recent version of NiFi. However, keep in mind
that you would need multiple NARs in your case (standard processors NAR +
NAR(s) for the readers/writers). In the end, you'd use NiFi 1.27.0 but
you'd have the PartitionRecord processor and readers/writers from NiFi
1.23.2.

Hope this helps,
Pierre


Le mar. 22 oct. 2024 à 15:36, 박정화 (Bank)  a écrit :

> hello.
>
> I am using nifi well.
>
> I recently upgraded the version from 1.23.2 to 1.27.0 and encountered the
> following error in the PartitionRecord process.
>
>
>
> com.fasterxml.jackson.core.JsonGenerationException: Incomplete surrogate
> pair: first char 0xD83D, second 0x002A
>
>
>
> It seems to be a serrogate pair issue, but it seems to have been caused by
> masking some characters on our side.
>
> Nevertheless, I wish it were possible to process json like in previous
> versions or like python.  Is there any way?
>
> Could it be that the error occurring differently from the previous version
> is related to the update of the jackson library?
>
> The jackson version of nifi 1.23.2 is 2.15.2, and the jackson version of
> nifi 1.27.0 is 2.17.1, so I thought it might be a problem.
>
> If so, we can change the jackson version and compile it, but we will need
> to continuously change the nifi version in the future.  Could you please
> give me some advice on this?
>
> thank you  have a good day
>