Re: Datatorrent fault tolerance

Oh I see, you want to send to different topics. Well, then you have to give some dummy value to the topic property on the operator. Regards, Siyuan On Fri, Oct 7, 2016 at 11:38 AM, Jaspal Singh wrote: > Siyuan, > > So for the output operator, we have specified it as

Re: Datatorrent fault tolerance

Siyuan, So for the output operator, we have specified it as a part of our logic itself. public class KafkaSinglePortExactlyOnceOutputOperator extends AbstractKafkaOutputOperator { private static final Logger LOG = LoggerFactory.getLogger(KafkaSinglePortExactlyOnceOutputOperator.class);

Re: Datatorrent fault tolerance

Jaspal, I think you miss the kafkaOut :) Regards, Siyuan On Fri, Oct 7, 2016 at 11:32 AM, Jaspal Singh wrote: > Siyuan, > > That's how we have given it in properties file: > > [image: Inline image 1] > > > Thanks!! > > On Fri, Oct 7, 2016 at 1:27 PM,

Re: Datatorrent fault tolerance

After making the change, we are getting the below error while application launch: *An error occurred trying to launch the application. Server message: javax.validation.ConstraintViolationException: Operator kafkaOut violates constraints

Re: Datatorrent fault tolerance

Should we use malhar-library version 3.5 then ? Thanks!! On Fri, Oct 7, 2016 at 1:04 PM, Thomas Weise wrote: > Please make sure to depend on version 3.5 of malhar-kafka in pom.xml. This > operator is not in malhar-library, it's a separate module. > > > On Fri, Oct 7, 2016 at

Re: Datatorrent fault tolerance

Hey Jaspal, Did you add any code to existing KafkaSinglePortExactlyOnceOutputOperator from malhar? If so please make sure the producer you use here is org.apache.kafka.clients.producer.KafkaProducer instead of kafka.javaapi.producer.Producer. That is old api and that is not supported by MapR

Re: Datatorrent fault tolerance

Thomas, We have added the dependency in pom.xml for lafka client API and also for malhar kafka. Please highlight if you are specifying some other dependency that we need to add. org.apache.apex malhar-kafka ${malhar.version} org.apache.kafka kafka-clients

Re: Datatorrent fault tolerance

Thomas, Below is the operator implementation we are trying to run. This operator is getting an object of tenant class from updtream operator. public class KafkaSinglePortExactlyOnceOutputOperator extends AbstractKafkaOutputOperator { private static final Logger LOG =

Re: Datatorrent fault tolerance

The KafkaSinglePortExactlyOnceOutputOperator takes whatever output from previous operator and writes to Kafka. Sent from my iPhone > On Oct 7, 2016, at 07:59, Jaspal Singh wrote: > > Hi Thomas, > > I have a question, so when we are using >

Re: Datatorrent fault tolerance

Hi Thomas, I have a question, so when we are using *KafkaSinglePortExactlyOnceOutputOperator* to write results into maprstream topic will it be able to read messgaes from the previous operator ? Thanks Jaspal On Thu, Oct 6, 2016 at 6:28 PM, Thomas Weise wrote: > For recovery

Re: Datatorrent fault tolerance

For recovery you need to set the window data manager like so: https://github.com/DataTorrent/examples/blob/master/tutorials/exactly-once/src/main/java/com/example/myapexapp/Application.java#L33 That will also apply to stateful restart of the entire application (relaunch from previous instance's

Re: Datatorrent fault tolerance

What you want is the effect of exactly-once output (that's why we call it also end-to-end exactly-once). There is no such thing as exactly-once processing in a distributed system. In this case it would be rather "produce exactly-once. Upstream operators, on failure, will recover to checkpointed

RE: Datatorrent fault tolerance

not working. Thanks, Srinivas From: Thomas Weise [mailto:t...@apache.org] Sent: Thursday, October 06, 2016 3:03 PM To: users@apex.apache.org Subject: Re: Datatorrent fault tolerance Hi, It would be necessary to know a bit more about your application for specific recommendations, but from what I

Re: Datatorrent fault tolerance

Hi, It would be necessary to know a bit more about your application for specific recommendations, but from what I see above, a few things don't look right. It appears that you are setting the processing mode on the input operator, which only reads from Kafka. Exactly-once is important for