I am not entirely conversant with the internals so dont have enough to sway one way or the other.
But it seems like we are mixing two separate concerns here. Modeling an entity to be surfaced in an API should have no bearing on how it is passivated. You can still store the data in zookeeper the way you are doing so today. Having an SPI which talks to how you passivate data v/s an API to how the user interacts with data are two separate concerns in my view. Sandeep On Sun, Feb 23, 2014 at 11:33 AM, Kanak Biscuitwala <[email protected]> wrote: > > Changing/removing entities in ZK is highly undesirable. We still want > old-version controllers to manage new-version participants, and vice versa. > Making a change like that would require system-wide upgrades and downtime. > > What we can do is change how the entities are exposed to the users through an > API. That is really the goal here: avoid changing the backing store, but > rethink how the user interacts with it. > > Kanak > > ---------------------------------------- >> Date: Sun, 23 Feb 2014 11:22:55 -0800 >> Subject: Re: [DISCUSS] New Helix Api >> From: [email protected] >> To: [email protected] >> >> Hey guys, >> >> I understand we want to quickly move to defining APIs but I am hoping >> the entities defined in the first mail are not a closed decision and >> is open for discussion >> >> So here is the list of entities defined in the first mail >> >> Entities >> -- Idealstate >> -- ExternalView >> -- CurrentState >> -- LiveInstance >> -- HelixConfig >> -- UserConfig >> -- StateModelDefinition >> -- Alerts >> >> I feel like there is a overlap of a lot of different constructs here >> which are not really entities. >> >> The basic entities in Helix IMO should be something like >> >> * Entities >> -- Cluster >> -- Instance >> -- Resource //I imagine replicas are a specialization of a resource so >> dont seem to be first class entities >> -- Partition >> -- User? //not clear how this is modelled but I put a place holder, >> probably not needed >> >> I imagine given that we could have multiple clusters, we may need >> something which contains clusters i.e. a HelixSystem entity. >> >> * Entities have configurations (there are again two perspectives to >> configurations) >> -- Configuration passed in during creation i.e. which is not yet realized >> -- Runtime configuration of the entity i.e. the current values in the >> system which are realized >> >> * Entities may or may not have a role >> -- This is where all the instance roles come into play >> >> * Entities have a state >> >> Given that we are redefining things anyway what do you guys think >> about this perspective? >> >> Thanks, >> >> Sandeep >> >> On Sun, Feb 23, 2014 at 10:20 AM, Kanak Biscuitwala <[email protected]> >> wrote: >>> Entities are the basic units of information necessary in order for Helix to >>> function. It's the "language" Helix speaks. If you look at ZooKeeper, this >>> is what you would see. >>> >>> Scopes are the levels at which things can be configured. That is to say, >>> you can configure the entire cluster, a single resource, a participant, etc. >>> >>> Roles are the way nodes are labeled in the system. Currently Helix nodes >>> can have the following roles: Participant, Spectator, Controller, and >>> Administrator. A single machine may take on multiple roles. >>> >>> The purpose of the API is to sufficiently abstract the entities at >>> appropriate scopes for each role. >>> >>> When we started brainstorming APIs, we identified the following use cases: >>> configuration, registration, state transitions, and rebalancing. Here is a >>> take on what part of configuration and registration could look like: >>> >>> Administrator admin = new ZKHelixAdministrator(); >>> admin.connect(); >>> ClusterConfig = new ClusterConfig().setName().setuserproperties(); >>> admin.createCluster(clusterconfig); >>> StateModelDef def = new MasterSlaveModel(); >>> ResourceConfig resourceConfig = new >>> ResourceConfig(resourceId).withFSM(def).partitioned(24).replicated(3).withRebalancer(RebalancerConfig); >>> admin.addResource(resourceConfig); >>> >>> //add Instances, we shud probably call this ParticipantConfig >>> ParticipantConfig instanceConfig = new ParticipantConfig(); >>> >>> // We can probably have controllerService that wraps controller -- this can >>> help people who want to control which events trigger the pipeline >>> >>> // not StandaloneController? no -- standalone means its only one >>> controller, people think its not fault tolerant >>> SingleClusterController controller = new SingleClusterController(); >>> controller.start(); >>> >>> // controls multiple clusters >>> MultiClusterController controller = new MultiClusterController(); >>> controller.start(); >>> >>> Participant participant = new Participant(); >>> ParticipantService service = new PartiticipantService(participant); >>> service.start(); // service has some callbacks init/start getListeners >>> callback that will be called appropriately >>> >>> Specator spectator = new Spectator(); >>> SpecatorService service = new SpectatorService().watch(resource(s)); >>> service.start(); //similar to participant gives a callback to get the >>> listeners to be added >>> >>> ---------------------------------------- >>>> Date: Sun, 23 Feb 2014 09:51:01 -0800 >>>> Subject: Re: [DISCUSS] New Helix Api >>>> From: [email protected] >>>> To: [email protected] >>>> >>>> Hey guys, >>>> >>>> For the sake of the newcomer could you please define/describe what you >>>> mean by the following >>>> >>>> (a) Entity >>>> (b) Scope >>>> >>>> I have some thoughts on the entities etc but wanted to understand what >>>> their definitions are before I suggest something. >>>> >>>> Thanks, >>>> >>>> Sandeep >>>> >>>> On Sun, Feb 23, 2014 at 7:54 AM, kishore g <[email protected]> wrote: >>>>> Hi, >>>>> >>>>> Here are the entities, scopes and roles defined in Helix. >>>>> Entities >>>>> -- Idealstate >>>>> -- ExternalView >>>>> -- CurrentState >>>>> -- LiveInstance >>>>> -- HelixConfig >>>>> -- UserConfig >>>>> -- StateModelDefinition >>>>> -- Alerts >>>>> >>>>> >>>>> Scopes >>>>> -- Cluster >>>>> -- Participant/Group >>>>> -- Resource/Group >>>>> -- Partition >>>>> -- Transition >>>>> >>>>> Misc >>>>> -- One time read v/s cache v/s cache and update automatically on >>>>> notification >>>>> >>>>> ==================== >>>>> HelixConfig >>>>> -- Cluster >>>>> -- Participant >>>>> -- Resource >>>>> -- Partition >>>>> >>>>> UserConfig >>>>> -- Cluster >>>>> -- Participant >>>>> -- Resource >>>>> -- Partition >>>>> >>>>> StateModelDefinition >>>>> -- States >>>>> -- Transitions >>>>> -- Constraints (Partition only) >>>>> >>>>> >>>>> Constraints >>>>> -- Cluster >>>>> -- Node >>>>> -- Resource >>>>> -- Partition >>>>> -- Resource/ResourceGroup >>>>> >>>>> Kanak and I did one pass on this and made some changes. Kanak, do you have >>>>> send across the changed version. >>>>> >>>>> We need to come up with the user facing api's that support simple CRUD on >>>>> these entities. >>>>> >>>>> thanks, >>>>> Kishore G >
