Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-08-05 Thread Marton Szasz
Good points raised. My thinking was that with exceptions and causing a bigger issue at runtime, hopefully Python processor developers would be more likely to notice any issues during development, and the issues wouldn't ever reach production. But the flip side is that there is a risk of an unfo

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-08-05 Thread Gábor Gyimesi
I was thinking the same that first I also wanted to throw an exception, but that would just trigger a rollback and we would retry the flow file later. Still it is reasonable to at least write an error message in the logs about the bad usage. I also think that even though it is documented, it is no

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-08-05 Thread Mark Payne
Marton, Sure, the idea of “an API should be easy to use correctly and hard to use incorrectly” is certainly a great principle in API design. But I’m not entirely sure that it applies here directly. The idea of “make it hard to use incorrectly” doesn’t mean to throw an Exception when used incorre

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-08-01 Thread Marton Szasz
I think good APIs are easy to use correctly and hard to use incorrectly. (I stole this from C++ books.) For this reason, I'm +1 for throwing an exception when the API user specifies a content with a transform result headed to failure. Also +1 for making this clear in the docs. If an exceptio

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-07-31 Thread Mark Payne
Agreed, it should absolutely be documented. I don’t have a particularly strong opinion regarding whether or not to log a warning. Thanks -Mark > On Jul 31, 2024, at 1:00 PM, Gábor Gyimesi wrote: > > Hi, > > MiNiFi C++ copied the behavior of the NiFi Python API to be compatible > with the NiF

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-07-31 Thread Gábor Gyimesi
Hi, MiNiFi C++ copied the behavior of the NiFi Python API to be compatible with the NiFi python processors (although some minor differences may apply [1]), so it works the same way in this case. I think this behavior is understandable and can be accepted, but I think Ferenc has a point that this a

Re: Handling of the contents of the failure result in the python FlowFileTransform API

2024-07-31 Thread Mark Payne
Hi Ferenc, I’m not overly familiar with MiNiFi C++ but this is how it works in traditional NiFi as well. This is intended. While Processors are generally free to treat relationships as they wish, the convention is always to route the original incoming FlowFile to ‘failure’, never a modified ver

Handling of the contents of the failure result in the python FlowFileTransform API

2024-07-31 Thread Ferenc Gerlits
Hello NiFi community, We have implemented NiFi's FlowFileTransform API for Python processors in MiNiFi C++, and we have written a few processors using it. We have noticed that the "failure" relationship works like this: if the transform() method returns FlowFileTransformResult(relationship="failur