Thanks Priyanka ! From: Priyanka Gugale <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, May 23, 2016 at 5:18 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Time controlled ingestion operator
Update: This Bandwidth Limiting can be used with any input operator. So you can apply similar changes to AbstractJDBCPollInputOperator shared by Devendra earlier. -Priyanka On Mon, May 23, 2016 at 4:54 PM, Priyanka Gugale <[email protected]<mailto:[email protected]>> wrote: Hi Sunil, This<https://github.com/apache/incubator-apex-malhar/compare/master...DT-Priyanka:jdbc-bandwidth-example?expand=1> is the updated JDBCPojoInputOperator to use Bandwidth limiting feature (Just look at JDBCPojoInputOperator changes, other classes are from bandwidth PR) Once you update your code to use extend BandwidthLimitingOperator input operator and use BandwidthManager (as done in example code), all you need to do is set bandwidth value. In your properties file by adding, <property> <name>dt.operator.<OperatorName>.prop.bandwidthManager.bandwidth</name> <value>1073741824</value> // 1 GB (change it to desired value) </property> During hours when you want application to stop reading you can modify value of bandwidth to say -1 using dtcli command: set-operator-property operator-name property-name property-value You can write cron jobs to change bandwidth property values. The right solution would be to update BandwidthManager to accept the bandwidth schedule and update consumption accordingly. Also please note in this solution the application will still be running and operators will be in deployed state, which means using some of your cluster resources. -Priyanka On Mon, May 23, 2016 at 4:27 PM, Sunil Parmar <[email protected]<mailto:[email protected]>> wrote: Thanks Ram ! That's the precise reason. Priyanka, Can you elaborate or give more little more guidance to update the application property run time ? How to do that ? -Sunil From: Munagala Ramanath <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, May 23, 2016 at 3:42 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Time controlled ingestion operator >From Sunil's note it sounds like the DB is in production use by other systems >during certain hours and wants the Apex based migration to happen at other >(non-overlapping) times. Ram On May 23, 2016 2:32 PM, "Priyanka Gugale" <[email protected]<mailto:[email protected]>> wrote: Hi Sunil, I don't think we have exactly pause and resume feature in Malhar yet. But I can think of some other way to achieve this. We are adding bandwidth control feature to Malhar. The pull request<https://github.com/apache/incubator-apex-malhar/pull/279> is already opened for the same. This doesn't have option to configure timed bandwidth consumption (i.e. use x bandwidth during given time hours), but I think it's easy to do that in BandwidthManager. Or quick thing is you can try to update the property "bandwidth" during the time when u don't want operator to work using some external script (this is short term solution). Can you tell the reason why you want to pause the read during certain time hours? -Priyanka On Mon, May 23, 2016 at 2:21 PM, Sunil Parmar <[email protected]<mailto:[email protected]>> wrote: All, We're writing an app for migration data from legacy JDBC system to Kafka. Legacy system as it is still being used in production, we would want to pause the JDBC poll input operator for configured hours. Does malhar ibrary has any operator that we can use for time controlled operator ? Or is there a way to configure input operators to pause and resume for given time ? -Sunil
