If you have access to the ideal state for which the state transition was sent, you would be able to tell if a partition was removed or moved. Is the ideal state already exposed in the context object? A fresh read of ideal state could lead to race conditions
Swaroop On 5/19/14 10:23 AM, "Kanak Biscuitwala" <[email protected]> wrote: >If your meta-resource follows the CUSTOMIZED rebalance mode, then you >won't get dropped transitions unless you explicitly remove the entry from >the ideal state. > >An initial version of the task framework is in the helix-0.6.x branch. >Unfortunately we don¹t have much documentation at this point, but you can >take a look at TestTaskRebalancer for an example. There¹s still work to >do from there, but this should be enough to hopefully get started. > >---------------------------------------- >> Date: Mon, 19 May 2014 09:42:03 -0700 >> From: [email protected] >> To: [email protected] >> Subject: Re: Need two kinds of DROPPED states? >> >> Yes, your interpretation is correct. >> >> I thought about using your meta-resource idea, but it suffers from a >> similar problem as what I am currently facing. The partitions of the >> meta-resource will be "placed" on some participant by Helix. Now imagine >> that participant fails. The partition of the meta-resource will go >> through its DROPPED state to get placed elsewhere. So that DROPPED state >> is still discernible from the partition being dropped to reflect >> dropping of the resource it represents. Either I misunderstood your >> solution (1) or I need to somehow have a participant that never fails. >> >> I can see how to use the task framework to build a reliable workflow >> with a sequence of steps where one of the steps is to do the final >> cleanup. Is this already available in the codebase (if I was willing to >> make my own build)? >> >> Thanks, >> Vinayak >> >> >> >> On 5/19/14, 9:11 AM, Kanak Biscuitwala wrote: >>> So if I understand correctly, you basically want to know if state >>>should be kept in case the partition could move back, or not. Right >>>now, Helix treats a dropped resource as if all its partitions have been >>>dropped. Separately, Helix treats a moved partition as a dropped >>>partition on one participant and an added partition on another >>>participant. So they're currently very much linked. >>> >>> This requires some more thought, but here's what comes to mind: >>> >>> 1. Have a meta-resource whose partitions are simply the names of the >>>other resources in the cluster. When you drop a resource, the operation >>>would be to simultaneously drop the resource and drop the partition >>>from the meta resource. Then you can get a separate transition for >>>dropped resource. I haven't thought about the race conditions here, and >>>there could be some impact depending on your app. >>> >>> 2. In the upcoming task framework, create a task that manages the drop >>>resource scenario from beginning to end, for instance call >>>helixadmin#dropresource, wait for external view to converge, issue >>>cleanup requests to participants. Participants would implement a >>>cleanup callback. This is something we're get out the door this quarter. >>> >>> 3. Something that works, but you would like to avoid: ask HelixAdmin >>>if the resource exists >>> >>> Perhaps others can chime in with ideas. >>> >>> ---------------------------------------- >>>> Date: Sun, 18 May 2014 12:08:15 -0700 >>>> From: [email protected] >>>> To: [email protected] >>>> Subject: Need two kinds of DROPPED states? >>>> >>>> Hi Guys, >>>> >>>> >>>> It looks like when a partition that is on a participant (P1) is moved >>>>to >>>> another participant (P2), P1 is sent a transition request from OFFLINE >>>> -> DROPPED. >>>> >>>> In an other scenario, when a resource is dropped using HelixAdmin, the >>>> partitions undergo a similar transition to DROPPED. >>>> >>>> As an application, one might need to do different things in those two >>>> cases. For example, in the first case its being dropped to become live >>>> somewhere else and so any shared state for the resource should not be >>>> lost. On the other hand, in the second scenario, the application might >>>> want to clean up all state associated with the resource. >>>> >>>> Is there a way for the application to distinguish between the first >>>>kind >>>> of DROPPED and the second kind? I am looking to have the state machine >>>> itself handle both the scenarios without the need for the application >>>>to >>>> trigger some special activity to perform the cleanup in the second >>>>scenario. >>>> >>>> Thanks, >>>> Vinayak >>> >>> >> >
