Currently I don't think there is a way that config value can be set without a code change, but if you want to create a JIRA it would probably make sense to expose that as a property in the processor to toggle between true and false, or we can also allow make it so that any dynamic properties get passed through to the Parquet writer's conf.
On Thu, Oct 25, 2018 at 2:58 AM Ken Tore Tallakstad <[email protected]> wrote: > Hi, > > We have an issue with PutParquet (NiFi 1.7.1), well with the parquet lib > to be precise, and array type data containing null values. > This is a schema snippet of the field in question: > { > "name": "adresse", > "type" : ["null", { "type" : "array", "items" : > ["null","string"], "default": null } ], "default": null > }, > > And a corresponding data example: > "adresse" : [ null, "value1" ], > "adresse" : [ null, "value2" ], > "adresse" : [ "value3", null, "value4" ], > > Avro does not seem to have a problem with this and all our records pass, > but Put parquet fails with the following error: "Array contains a null > element at X". > > Apparently there is a parquet config to allow > this: parquet.avro.write-old-list-structure=false. Any tips on how to set > it? Are there any other ways around this, besides stripping the raw data of > nulls in arrays? > > Thanks! > > KT :) > > > [image: parq1.png] > > > [image: parq2.png] >
