If the output is supposed to look as Tamas mentioned, you can try
JoltTransformRecord with the following Chain spec:
[
{
"operation": "shift",
"spec": {
"values": {
"*": {
"0": "[#2].id",
"1": "[#2].msg"
}
}
}
}
]
That should output 3 records (using your sample input), each with an
id and msg field. Using a CsvRecordSetWriter with the corresponding
schema, you should get a CSV out that looks like what Tamas wrote.
Regards,
Matt
On Fri, Nov 15, 2019 at 10:03 AM Tamás Pálfy <[email protected]> wrote:
>
> I'd guess the output should look something like this:
>
> 1558994400000,Malformed virtual protocol
> 1558994400000,BitTorrent
> 1558994400000,Advance Message Queuing Protocol
>
> I tried to play around and the best I could come up with is adding a
> SplitJson before the ConvertRecord (set the jsonpath to "values").
> This will create new flowfiles for each element in the array.
>
> However this was not enough for me. I had to change the input as well so that
> the "values" array contained records instead of arrays (otherwise the
> JsonTreeReader ignored the data), like this:
>
> {
> "values": [
> {
> "id": 1558994400000,
> "msg": "Malformed virtual protocol"
> },
> {
> "id": 1558994400000,
> "msg": "Advance Message Queuing Protocol"
> },
> {
> "id": 1558994400000,
> "msg": "BitTorrent"
> }
> ]
> }
>
> Also CSVRecordSetWriter property "Include Header Line" had to be set to
> "false" (otherwise you get multiple header lines in the csv, one for each
> element in the original array).
>
>
> On Thu, Nov 14, 2019 at 7:30 PM Mark Payne <[email protected]> wrote:
>>
>> The CSVRecordSetWriter just writes out each field in the Record as a String.
>> If a field is an array, you end up with the "[[Ljava.lang.Object;@195d9bda"
>> syntax because that's how Java's toString() works for arrays. The CSV Writer
>> doesn't really have any knowledge of how to write out an Array of objects
>> because those are not well represented in CSV. So would recommend you
>> instead use QueryRecord or UpdateRecord or JoltTransformRecord. Some sort of
>> processor that has the capability to manipulate the data before writing out
>> as CSV.
>>
>> Do you have an example of what you would want that resulting CSV file to
>> look like?
>>
>>
>> On Nov 14, 2019, at 1:24 PM, Joe Ferner <[email protected]> wrote:
>>
>> Your data has arrays of mixed types, but your schema is indicating they are
>> all strings. Not sure if this would be an issue.
>>
>> On Thu, Nov 14, 2019 at 12:56 PM José Maria Vieira <[email protected]>
>> wrote:
>>>
>>> Yes, i am using JsonTreeReader.
>>> I have tried both with infer schema and with the following schema.
>>>
>>>
>>> {
>>> "name": "MyClass",
>>> "type": "record",
>>> "namespace": "com.acme.avro",
>>> "fields": [
>>> {
>>> "name": "values",
>>> "type": {
>>> "type": "array",
>>> "items": {
>>> "type": "array",
>>> "items": "string"
>>> }
>>> }
>>> }
>>> ]
>>> }
>>>
>>>
>>> On 2019/11/14 17:49:35, Joe Ferner <[email protected]> wrote:
>>> > Are you using JsonTreeReader? Can you please share your schema?
>>> >
>>> > On Thu, Nov 14, 2019 at 5:55 AM José Maria Vieira <[email protected]>
>>> > wrote:
>>> >
>>> > > Hi there fellow Nifi users:
>>> > >
>>> > > Today I was developing a flow to convert some Json files to CSV.
>>> > > I tried using the ConvertRecord processor, which seemed to be the best
>>> > > option. However I got java objects instead of the expected values.
>>> > > I think the cause of the issue, is the json has nested objects.
>>> > > I'm not sure if there is a bug in the CSVRecordSetWriter implementation
>>> > > or
>>> > > if I am doing something wrong.
>>> > > What do you think?
>>> > >
>>> > >
>>> > >
>>> > > Sample Json I used:
>>> > >
>>> > > {
>>> > > "values": [
>>> > > [
>>> > > 1558994400000,
>>> > > "Malformed virtual protocol"
>>> > > ],
>>> > > [
>>> > > 1558994400000,
>>> > > "Advance Message Queuing Protocol"
>>> > > ],
>>> > > [
>>> > > 1558994400000,
>>> > > "BitTorrent"
>>> > > ]
>>> > > ]
>>> > > }
>>> > >
>>> > >
>>> > > ConvertRecord Result:
>>> > > values
>>> > > "[[Ljava.lang.Object;@195d9bda, [Ljava.lang.Object;@248d0236,
>>> > > [Ljava.lang.Object;@192b2034]"
>>> > >
>>> > > All the best,
>>> > > José Vieira
>>> > >
>>> >
>>
>>
