[
https://issues.apache.org/jira/browse/NIFI-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991341#comment-15991341
]
ASF GitHub Bot commented on NIFI-3739:
--------------------------------------
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/1695
one more observation and cannot recall if I mentioned this yet. But here
is some original JSON we'd get from provenance. This is before turning into
avro and then turning back into json.
{code}
{
"eventId":"3ddbed75-c436-4112-a2bd-bbdd05320b23",
"eventOrdinal":13720400,
"eventType":"CREATE",
"timestampMillis":1493439571667,
"timestamp":"2017-04-29T04:19:31.667Z",
"durationMillis":-1,
"lineageStart":1493439571667,
"componentId":"b259f36c-015b-1000-3ca6-c664e156cdf7",
"componentType":"GenerateFlowFile",
"componentName":"GenerateFlowFile",
"entityId":"2ab3c2d8-cb45-44ae-bd45-72080aff3e78",
"entityType":"org.apache.nifi.flowfile.FlowFile",
"entitySize":0,
"updatedAttributes":
{
"path":"./",
"uuid":"2ab3c2d8-cb45-44ae-bd45-72080aff3e78",
"filename":"464576540850588"
},
"previousAttributes":{},
"actorHostname":"10.0.0.15",
"contentURI":"http://10.0.0.15:8080/nifi-api/provenance-events/13720400/content/output";,
"previousContentURI":"http://10.0.0.15:8080/nifi-api/provenance-events/13720400/content/input";,
"parentIds":[],
"childIds":[],
"platform":"nifi",
"application":"NiFi Flow"
}
{code}
Here is the after avro and then back to json view
{code}
{
"eventId" : "edf16cc9-b50d-4cbb-9356-58eb4cf8fd3e",
"eventOrdinal" : 13585491,
"eventType" : "CREATE",
"timestampMillis" : "2017-04-29 04:18:50",
"durationMillis" : -1,
"lineageStart" : "2017-04-29 04:18:50",
"details" : null,
"componentId" : "b259f36c-015b-1000-3ca6-c664e156cdf7",
"componentType" : "GenerateFlowFile",
"componentName" : "GenerateFlowFile",
"entityId" : "ef12f52c-4ae4-46be-bc45-698200f08e54",
"entityType" : "org.apache.nifi.flowfile.FlowFile",
"entitySize" : 0,
"previousEntitySize" : null,
"updatedAttributes" : {
"path" : "./",
"filename" : "464535310815392",
"uuid" : "ef12f52c-4ae4-46be-bc45-698200f08e54"
},
"previousAttributes" : { },
"actorHostname" : "10.0.0.15",
"contentURI" :
"http://10.0.0.15:8080/nifi-api/provenance-events/13585491/content/output";,
"previousContentURI" :
"http://10.0.0.15:8080/nifi-api/provenance-events/13585491/content/input";,
"parentIds" : [ ],
"childIds" : [ ],
"platform" : "nifi",
"application" : "NiFi Flow",
"transitUri" : null
}
{code}
Notice timestamp millis. Seems like we should let json and csv writing use
the timestamp like it would be in an avro 'timestamp-millis' sort of way rather
than a formatted date/time string. We'd want to do this consistently on
reading and writing. The current default changes its representation i think
and makes it a bit tougher to do math with.
> Create Processors for publishing records to and consuming records from Kafka
> ----------------------------------------------------------------------------
>
> Key: NIFI-3739
> URL: https://issues.apache.org/jira/browse/NIFI-3739
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> With the new record readers & writers that have been added in now, it would
> be good to allow records to be pushed to and pulled from kafka. Currently, we
> support demarcated data but sometimes we can't correctly demarcate data in a
> way that keeps the format valid (json is a good example). We should have
> processors that use the record readers and writers for this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
