Hi,

which operators in your application are writing to external systems?

When you look at the example from the blog (
https://github.com/DataTorrent/examples/tree/master/tutorials/exactly-once),
there is Kafka input, which is configured to be idempotent. The results are
written to JDBC. That operator by itself supports exactly-once through
transactions (in conjunction with idempotent input), hence there is no need
to configure the processing mode at all.

Thomas


On Thu, Oct 6, 2016 at 1:52 PM, Bandaru, Srinivas <
srinivas.band...@optum.com> wrote:

> Thanks Thomas. We have set the set all three operator properties, Kafka
> input operator has set to “Exactly_once” and downstream two operators set
> as AT_MOST_ONCE same as mentioned in documentation.  We have also referred
> the documentation you have shared but couldn’t find the right properties
> for exactly_once scenario. Is there anything we are missing or need to add
> additional properties if someone else used this functionality.
>
>
>
> *Any help would be greatly appreciated??*
>
>
>
> We tried with different combinations and trying to set all three
> properties with “EXACTLY_ONCE”, But application throwing an with error
> saying *“If the processing mode for an operator is specified as
> EXACTLY_ONCE then the processing mode for all downstream operators should
> be specified as AT_MOST_ONCE otherwise it will result in an error.*”
> Application behavior is same as the Datatorrent documentation but it’s not
> working.
>
>
>
> Thanks,
>
> Srinivas
>
>
>
>
>
> *From:* Thomas Weise [mailto:t...@apache.org]
> *Sent:* Thursday, October 06, 2016 3:03 PM
> *To:* users@apex.apache.org
> *Subject:* Re: Datatorrent fault tolerance
>
>
>
> Hi,
>
>
>
> It would be necessary to know a bit more about your application for
> specific recommendations, but from what I see above, a few things don't
> look right.
>
>
>
> It appears that you are setting the processing mode on the input operator,
> which only reads from Kafka. Exactly-once is important for operators that
> affect the state of external systems, for example when writing to a
> database, or, producing messages to Kafka.
>
>
>
> There is also a problem with the documentation that may contribute to the
> confusion. The processing mode you are trying to use isn't really
> exactly-once and probably will be deprecated in a future release. Here are
> some better resources on fault tolerance and processing semantics:
>
>
>
> http://apex.apache.org/docs.html
>
> https://www.youtube.com/watch?v=FCMY6Ii89Nw
>
> http://www.slideshare.net/ApacheApexOrganizer/webinar-fault-
> toleranceandprocessingsemantics
>
> https://www.datatorrent.com/blog/end-to-end-exactly-once-with-apache-apex/
>
>
>
> HTH,
>
> Thomas
>
>
>
>
>
> On Thu, Oct 6, 2016 at 11:26 AM, Bandaru, Srinivas <
> srinivas.band...@optum.com> wrote:
>
> Hi, Need some help!
>
> Need your advice in implementing the fault tolerance mechanism using
> datatorrent. Currently we are trying to implement with EXACTLY_ONCE
> scenario.
>
> We are referring the below documentation. The highlighted line says if we
> need to use the EXACTLY_ONCE processing mode, the downstream operators
> should have AT_MOST_ONCE. We added the operator level attributes in the
> properties.xml (attached below) and when we launch the application, no
> messages are coming through the last operator *topicUpdate *but we are
> able to see the messages when we comment those attributes.
>
> *Please have a look at it, either we are specifying the attributes
> incorrectly or is there any additional step to be done*.
>
>
>
>
>
> ·         *PROCESSING_MODE*
>
> static final Attribute 
> <https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Attribute.html><Operator.ProcessingMode
>  
> <https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Operator.ProcessingMode.html>>
>  PROCESSING_MODE
>
> The payload processing mode for this operator - at most once, exactly
> once, or default at least once. If the processing mode for an operator is
> specified as AT_MOST_ONCE and no processing mode is specified for the
> downstream operators if any, the processing mode of the downstream
> operators is automatically set to AT_MOST_ONCE. If a different processing
> mode is specified for the downstream operators it will result in an error. If
> the processing mode for an operator is specified as EXACTLY_ONCE then the
> processing mode for all downstream operators should be specified as
> AT_MOST_ONCE otherwise it will result in an error.
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
> Srinivas
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>

Reply via email to