[
https://issues.apache.org/jira/browse/NIFI-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kulyk updated NIFI-7773:
-------------------------------
Description:
Using *JoltTransformJSON* to transform kafka JSON messages before making an
update on Postgres DB
In some rare cases messages (received from same topic partition) are "mixing"
up an order within a *same node*
Therefore updates are invoked in wrong order, affecting on data consistency
and correctness
*Provenance* logs for some found cases:
Partition and offset for each message provided in screen (with blue)
!image-2020-08-28-13-06-03-269.png|width=726,height=478!
!image-2020-08-28-13-06-20-964.png|width=730,height=417!
*Flow* (mentioned "flow pipeline" marked with yellow):
!image-2020-08-28-13-07-39-001.png|width=728,height=389!
*Flow code* has been attached
was:
Using *JoltTransformJSON* to transform kafka JSON messages before making an
update on Postgres DB
In some rare cases messages (received from same topic partition) are "mixing"
up an order within a node
Therefore updates are invoked in wrong order, affecting on data consistency
and correctness
*Provenance* logs for some found cases:
Partition and offset for each message provided in screen (with blue)
!image-2020-08-28-13-06-03-269.png|width=726,height=478!
!image-2020-08-28-13-06-20-964.png|width=730,height=417!
*Flow* (mentioned "flow pipeline" marked with yellow):
!image-2020-08-28-13-07-39-001.png|width=728,height=389!
*Flow code* has been attached
> JoltTransformJSON mixing up flow files order
> --------------------------------------------
>
> Key: NIFI-7773
> URL: https://issues.apache.org/jira/browse/NIFI-7773
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.11.4
> Environment: Ubuntu 16 x64, 3 nodes cluster
> Reporter: Dmytro Kulyk
> Priority: Critical
> Labels: jolt, order
> Attachments: Sportsbook (2).json, image-2020-08-28-13-06-03-269.png,
> image-2020-08-28-13-06-20-964.png, image-2020-08-28-13-07-39-001.png
>
>
> Using *JoltTransformJSON* to transform kafka JSON messages before making an
> update on Postgres DB
> In some rare cases messages (received from same topic partition) are "mixing"
> up an order within a *same node*
> Therefore updates are invoked in wrong order, affecting on data consistency
> and correctness
> *Provenance* logs for some found cases:
> Partition and offset for each message provided in screen (with blue)
> !image-2020-08-28-13-06-03-269.png|width=726,height=478!
> !image-2020-08-28-13-06-20-964.png|width=730,height=417!
> *Flow* (mentioned "flow pipeline" marked with yellow):
> !image-2020-08-28-13-07-39-001.png|width=728,height=389!
> *Flow code* has been attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
