Very interesting re your discovery. Thanks very much for sharing that Matt. I do want to keep the schema intact. I will try ScriptedTransformRecord. Thanks again for replying Matt. Cheers, Jim
On Thu, Jun 1, 2023 at 10:13 AM Matt Burgess <[email protected]> wrote: > Jim, > > I tried to use Jolt for this but I found in the doc that if you try to > set an empty array or map to null or the empty string it will retain > the empty array or map (no idea why). Since you know the name of the > fields (and I assume want to keep the schema intact) you can use > ScriptedTransformRecord to get at those fields by name and set them to > null. > > Regards, > Matt > > On Mon, May 29, 2023 at 10:38 AM James McMahon <[email protected]> > wrote: > > > > I have incoming JSON data that begins like this, and that I am trying to > flatten with FlattenJSON v1.16.3: > > > > { > > "meta" : { > > "view" : { > > "id" : "kku6-nxdu", > > "name" : "Demographic Statistics By Zip Code", > > "assetType" : "dataset", > > "attribution" : "Department of Youth and Community Development > (DYCD)", > > "averageRating" : 0, > > "category" : "City Government", > > "createdAt" : 1311775554, > > "description" : "Demographic statistics broken down by zip code", > > "displayType" : "table", > > "downloadCount" : 1017553, > > "hideFromCatalog" : false, > > "hideFromDataJson" : false, > > "indexUpdatedAt" : 1536596131, > > "newBackend" : true, > > "numberOfComments" : 3, > > "oid" : 4208790, > > "provenance" : "official", > > "publicationAppendEnabled" : false, > > "publicationDate" : 1372266760, > > "publicationGroup" : 238846, > > "publicationStage" : "published", > > "rowClass" : "", > > "rowsUpdatedAt" : 1372266747, > > "rowsUpdatedBy" : "uurm-7z6x", > > "tableId" : 942474, > > "totalTimesRated" : 0, > > "viewCount" : 70554, > > "viewLastModified" : 1652135219, > > "viewType" : "tabular", > > "approvals" : [ { > > "reviewedAt" : 1372266760, > > "reviewedAutomatically" : true, > > "state" : "approved", > > "submissionId" : 1064760, > > "submissionObject" : "public_audience_request", > > "submissionOutcome" : "change_audience", > > "submittedAt" : 1372266760, > > "workflowId" : 2285, > > "submissionDetails" : { > > "permissionType" : "READ" > > }, > > "submissionOutcomeApplication" : { > > "failureCount" : 0, > > "status" : "success" > > }, > > "submitter" : { > > "id" : "5fuc-pqz2", > > "displayName" : "NYC OpenData" > > } > > } ], > > "clientContext" : { > > "clientContextVariables" : [ ], > > "inheritedVariables" : { } > > }, > > "columns" : [ { > > "id" : -1, > > "name" : "sid", > > "dataTypeName" : "meta_data", > > "fieldName" : ":sid", > > "position" : 0, > > "renderTypeName" : "meta_data", > > "format" : { }, > > "flags" : [ "hidden" ] > > }, { ..... > > > > This is my configuration of my FlattenJson processor: > > Separator . > > Flatten mode normal > > Ignore Reserved Characters false > > Return Type flatten > > Character Set UTF-8 > > Pretty Print JSON true > > > > Those lines in red appear in my output like this: > > "meta.view.clientContext.clientContextVariables" : [ ], > > "meta.view.clientContext.inheritedVariables" : { > > > > }, > > > > I don't want to preserve the empty list and empty map. I want to set the > values for these keys to the empty string or null is acceptable. > > > > Can I do this natively through the FlattenJson configuration? If not, > what would be the most efficient means to post-process to what I seek in my > flow? > > > > Thanks in advance for any help. >
