Re: Replay Event UUID

2018-01-09 Thread Koji Kawamura 
Hi Rotem,

You are correct, the WriteAheadProvenanceRepository returns provenance
event FlowFile UUID value differently than the default
PersistentProvenanceRepository.
Here is the lines of code that set FlowFile UUID for the provenance events.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/schema/LookupTableEventRecord.java#L276-L280

WriteAheadProvenanceRepository does not seem to have 'FlowFile UUID'
value persisted, which is set by FlowController when replay events are
registered. Instead, WriteAheadProvenanceRepository fill 'FlowFile
UUID' from updated or previous 'UUID' attribute.
I don't know much background on why it is implemented this way, but it
seems it drops 'FlowFile UUID' to reduce IO based on an assumption
that it can be set by attributes.

Thanks again for reporting this. I filed NIFI-4752.
https://issues.apache.org/jira/browse/NIFI-4752

Koji

On Thu, Dec 28, 2017 at 5:33 PM, Rotem Mayo  wrote:
> Hi,
>
> We had another server that was not experiencing this bug, but when we
> changed  to the WriteAheadProvenanceRepository (like on the server where we
> first noticed this) we did see it. Have there been changes in that
> implementation? Could this cause what we are seeing?
>
> Thanks!
> Rotem
>
>
> On 28 Dec 2017 3:35 am, "Koji Kawamura"  wrote:
>
> Hi Rotem,
>
> When I tested it with NiFi 1.5.0-SNAPSHOT, the REPLAY event has its
> FlowFile UUID as the parent (original) FlowFile UUID as expected.
>
> Type
> REPLAY
> FlowFile Uuid
> 8c61fdd7-c084-4756-946c-f5669dc4442d
> File Size
> 4 bytes
> Component Id
> 9abc21e3-0160-1000-6d6f-a1c408f75b7a
> Component Name
> UpdateAttribute
> Component Type
> UpdateAttribute
>
> Parent FlowFiles (1)
> 8c61fdd7-c084-4756-946c-f5669dc4442d
> Child FlowFiles (1)
> 0428b60c-44cb-46df-85ab-3787a38693e9
>
> The implementation has not been modified. It's been using parentUUID.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L4241
>
> Is this different than what you are seeing?
>
> Thanks,
> Koji
>
> On Thu, Dec 28, 2017 at 2:46 AM, Rotem Mayo  wrote:
>> Hi!
>>
>> I have been using nifi for a while, and have recently upgraded from 1.2.0
>> to
>> 1.4.0.
>>
>> I am having problems with replay events. In version 1.2.0, the UUID of a
>> replay event is the id of the original flowfile that is being replayed.
>> This
>> is the behavior that is documented in the developer guid (
>>
>> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#provenance_events).
>>
>> But, in nifi 1.4.0, the replay event's UUID is that of the child flowfile
>> that is created. This change is not documented in the changes log, and I
>> could not find any ticket related to the subject. In addition, other
>> events
>> that create child flowfiles, like clone and fork, still behave according
>> to
>> the documented way.
>>
>> Was this change done on purpose? Will other similar events change in the
>> future?
>>
>> Thanks!
>> Rotem
>
>
>

Re: Replay Event UUID

2017-12-28 Thread Rotem Mayo 
Hi,

We had another server that was not experiencing this bug, but when we
changed  to the WriteAheadProvenanceRepository (like on the server where we
first noticed this) we did see it. Have there been changes in that
implementation? Could this cause what we are seeing?

Thanks!
Rotem


On 28 Dec 2017 3:35 am, "Koji Kawamura"  wrote:

Hi Rotem,

When I tested it with NiFi 1.5.0-SNAPSHOT, the REPLAY event has its
FlowFile UUID as the parent (original) FlowFile UUID as expected.

Type
REPLAY
FlowFile Uuid
8c61fdd7-c084-4756-946c-f5669dc4442d
File Size
4 bytes
Component Id
9abc21e3-0160-1000-6d6f-a1c408f75b7a
Component Name
UpdateAttribute
Component Type
UpdateAttribute

Parent FlowFiles (1)
8c61fdd7-c084-4756-946c-f5669dc4442d
Child FlowFiles (1)
0428b60c-44cb-46df-85ab-3787a38693e9

The implementation has not been modified. It's been using parentUUID.
https://github.com/apache/nifi/blob/master/nifi-nar-
bundles/nifi-framework-bundle/nifi-framework/nifi-framework-
core/src/main/java/org/apache/nifi/controller/FlowController.java#L4241

Is this different than what you are seeing?

Thanks,
Koji

On Thu, Dec 28, 2017 at 2:46 AM, Rotem Mayo  wrote:
> Hi!
>
> I have been using nifi for a while, and have recently upgraded from 1.2.0
to
> 1.4.0.
>
> I am having problems with replay events. In version 1.2.0, the UUID of a
> replay event is the id of the original flowfile that is being replayed.
This
> is the behavior that is documented in the developer guid (
> https://nifi.apache.org/docs/nifi-docs/html/developer-
guide.html#provenance_events).
>
> But, in nifi 1.4.0, the replay event's UUID is that of the child flowfile
> that is created. This change is not documented in the changes log, and I
> could not find any ticket related to the subject. In addition, other
events
> that create child flowfiles, like clone and fork, still behave according
to
> the documented way.
>
> Was this change done on purpose? Will other similar events change in the
> future?
>
> Thanks!
> Rotem

Re: Replay Event UUID

2017-12-27 Thread Koji Kawamura 
Hi Rotem,

When I tested it with NiFi 1.5.0-SNAPSHOT, the REPLAY event has its
FlowFile UUID as the parent (original) FlowFile UUID as expected.

Type
REPLAY
FlowFile Uuid
8c61fdd7-c084-4756-946c-f5669dc4442d
File Size
4 bytes
Component Id
9abc21e3-0160-1000-6d6f-a1c408f75b7a
Component Name
UpdateAttribute
Component Type
UpdateAttribute

Parent FlowFiles (1)
8c61fdd7-c084-4756-946c-f5669dc4442d
Child FlowFiles (1)
0428b60c-44cb-46df-85ab-3787a38693e9

The implementation has not been modified. It's been using parentUUID.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L4241

Is this different than what you are seeing?

Thanks,
Koji

On Thu, Dec 28, 2017 at 2:46 AM, Rotem Mayo  wrote:
> Hi!
>
> I have been using nifi for a while, and have recently upgraded from 1.2.0 to
> 1.4.0.
>
> I am having problems with replay events. In version 1.2.0, the UUID of a
> replay event is the id of the original flowfile that is being replayed. This
> is the behavior that is documented in the developer guid (
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#provenance_events).
>
> But, in nifi 1.4.0, the replay event's UUID is that of the child flowfile
> that is created. This change is not documented in the changes log, and I
> could not find any ticket related to the subject. In addition, other events
> that create child flowfiles, like clone and fork, still behave according to
> the documented way.
>
> Was this change done on purpose? Will other similar events change in the
> future?
>
> Thanks!
> Rotem