Re: JoltTransformJSON error from NiFi, but is valid in tester
Thank you Arvind. I removed the comment but the processor continues to fail
its configuration check with the same error.
Our nifi version is 1.16.3.
I experimented with other configuration properties of the processor and
seem to have hit on success. I had tried setting Jolt Transformation DSL to
Shift, then to Default, then to Modify - Default. I finally understood that
because I was executing two transforms - a Shift, then a Default - it
should probably be set to Chain. And that worked.
Thank you again.
--
Jim
On Mon, Feb 19, 2024 at 12:02 AM Arvind Singh Tomar <
arvind.to...@hotwaxsystems.com> wrote:
> Hi James,
>
> Sharing your Nifi version would help identify the issue.
>
> However, one thing which I encountered while working with Nifi version
> 1.23.2 that the processor does not like embedded comments in the
> transformations. I needed to remove all the comments.
>
> I tested yours by removing the comment and it seems to be working fine:
>
> [
> {
> "operation": "shift",
> "spec": {
> "ids": "ids",
> "parents": "parents",
> "dates": "dates",
> "triage": "triage",
> "payload": "payload"
> }
> },
> {
> "operation": "default",
> "spec": {
> "payload": ""
> }
> }
> ]
>
> Regards
> --
> Arvind Singh Tomar
>
>
> On Mon, Feb 19, 2024 at 8:49 AM James McMahon
> wrote:
>
>> I have this JSON as flowfile content:
>>
>> {
>> "dates" : {
>> "date_file" : "20240115184407",
>> "ingested" : "20240217175748",
>> "latest_date" : "1980",
>> "earliest_date" : "1980",
>> "date_info" : "MMDD"
>> },
>> "parents" : {
>> "md5" : "86107362084b86ea64dc33dfde5e14ff",
>> "sha256" :
>> "4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f"
>> },
>> "triage" : {
>> "filetype" : "unstructured",
>> "languages" : "[\"gaelic\"]",
>> "filename" : "gaelicTest.txt"
>> },
>> "ids" : {
>> "md5" : "86107362084b86ea64dc33dfde5e14ff",
>> "sha256" :
>> "4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f",
>> "sku" : "0"
>> }
>> }
>>
>> I am trying to apply a simple JOLT transform that adds a key payload at
>> the same level in my json as keys dates, parents, triage, and ids.
>>
>> I developed this transform - just about as simple a case you could
>> possibly hope for:
>> [
>> {
>> "operation": "shift",
>> "spec": {
>> "ids": "ids",
>> "parents": "parents",
>> "dates": "dates",
>> "triage": "triage",
>> "payload": "payload"
>> }
>> },
>> {
>> "operation": "default",
>> "spec": {
>> // if payload does not exist, then apply a default of null
>> "payload": ""
>> }
>> }
>> ]
>>
>> I tested this successfully here: https://jolt-demo.appspot.com/#inception
>> .
>> It works.
>> No big surprise - this is about as simple as it gets.
>>
>> Should work in nifi, right? Can't get much easier than this.
>>
>> But in nifi, it throws this error:
>>
>> JoltTransformJSON[id=4d6c3f69-a72e-16b2-8cfe-2a9adb9303c7] processor is not
>> valid: : com.bazaarvoice.jolt.exception.SpecException: Shiftr expected a
>> spec of Map type, got ArrayList
>>
>>
>> Why? How can I get this to work in NiFi?
>>
>>
>>
>>
Re: JoltTransformJSON error from NiFi, but is valid in tester
Hi James,
Sharing your Nifi version would help identify the issue.
However, one thing which I encountered while working with Nifi version
1.23.2 that the processor does not like embedded comments in the
transformations. I needed to remove all the comments.
I tested yours by removing the comment and it seems to be working fine:
[
{
"operation": "shift",
"spec": {
"ids": "ids",
"parents": "parents",
"dates": "dates",
"triage": "triage",
"payload": "payload"
}
},
{
"operation": "default",
"spec": {
"payload": ""
}
}
]
Regards
--
Arvind Singh Tomar
On Mon, Feb 19, 2024 at 8:49 AM James McMahon wrote:
> I have this JSON as flowfile content:
>
> {
> "dates" : {
> "date_file" : "20240115184407",
> "ingested" : "20240217175748",
> "latest_date" : "1980",
> "earliest_date" : "1980",
> "date_info" : "MMDD"
> },
> "parents" : {
> "md5" : "86107362084b86ea64dc33dfde5e14ff",
> "sha256" :
> "4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f"
> },
> "triage" : {
> "filetype" : "unstructured",
> "languages" : "[\"gaelic\"]",
> "filename" : "gaelicTest.txt"
> },
> "ids" : {
> "md5" : "86107362084b86ea64dc33dfde5e14ff",
> "sha256" :
> "4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f",
> "sku" : "0"
> }
> }
>
> I am trying to apply a simple JOLT transform that adds a key payload at
> the same level in my json as keys dates, parents, triage, and ids.
>
> I developed this transform - just about as simple a case you could
> possibly hope for:
> [
> {
> "operation": "shift",
> "spec": {
> "ids": "ids",
> "parents": "parents",
> "dates": "dates",
> "triage": "triage",
> "payload": "payload"
> }
> },
> {
> "operation": "default",
> "spec": {
> // if payload does not exist, then apply a default of null
> "payload": ""
> }
> }
> ]
>
> I tested this successfully here: https://jolt-demo.appspot.com/#inception
> .
> It works.
> No big surprise - this is about as simple as it gets.
>
> Should work in nifi, right? Can't get much easier than this.
>
> But in nifi, it throws this error:
>
> JoltTransformJSON[id=4d6c3f69-a72e-16b2-8cfe-2a9adb9303c7] processor is not
> valid: : com.bazaarvoice.jolt.exception.SpecException: Shiftr expected a spec
> of Map type, got ArrayList
>
>
> Why? How can I get this to work in NiFi?
>
>
>
>
JoltTransformJSON error from NiFi, but is valid in tester
I have this JSON as flowfile content:
{
"dates" : {
"date_file" : "20240115184407",
"ingested" : "20240217175748",
"latest_date" : "1980",
"earliest_date" : "1980",
"date_info" : "MMDD"
},
"parents" : {
"md5" : "86107362084b86ea64dc33dfde5e14ff",
"sha256" :
"4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f"
},
"triage" : {
"filetype" : "unstructured",
"languages" : "[\"gaelic\"]",
"filename" : "gaelicTest.txt"
},
"ids" : {
"md5" : "86107362084b86ea64dc33dfde5e14ff",
"sha256" :
"4ffe010f3392dddb3a880f0e60a6bf35e1e41444b7e83be0746097557d555e0f",
"sku" : "0"
}
}
I am trying to apply a simple JOLT transform that adds a key payload at the
same level in my json as keys dates, parents, triage, and ids.
I developed this transform - just about as simple a case you could possibly
hope for:
[
{
"operation": "shift",
"spec": {
"ids": "ids",
"parents": "parents",
"dates": "dates",
"triage": "triage",
"payload": "payload"
}
},
{
"operation": "default",
"spec": {
// if payload does not exist, then apply a default of null
"payload": ""
}
}
]
I tested this successfully here: https://jolt-demo.appspot.com/#inception .
It works.
No big surprise - this is about as simple as it gets.
Should work in nifi, right? Can't get much easier than this.
But in nifi, it throws this error:
JoltTransformJSON[id=4d6c3f69-a72e-16b2-8cfe-2a9adb9303c7] processor
is not valid: : com.bazaarvoice.jolt.exception.SpecException: Shiftr
expected a spec of Map type, got ArrayList
Why? How can I get this to work in NiFi?
