subject:"Cluster auto activation design proposal"

Re: Cluster auto activation design proposal

2017-11-07 Thread Alexey Goncharuk

> > > > > > > and
> > > > > > > > > has
> > > > > > > > > > > > data
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > > > >BLT #1 created.
> > > > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide
> it
> > > into
> > > > > two
> > > > > > > > > parts:
> > > > > > > > > > > > > > Part1_grid
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >Part2_grid.
> > > > > > > > > > > > > > > > > >3. We start and activate Part1_grid .
> > > > Topology
> > > > > > has
> > > > > > > > > > changed
> > > > > > > > > > > > ->
> > > > > > > > > > > > > > > BLT#2
> > > > > > > > > > > > > > > > > >created.
> > > > > > > > > > > > > > > > > >After that we shutdown that
> Part1_grid.
> > > > > > > > > > > > > > > > > >4. We start and activate Part2_grid.
> > > > Topology
> > > > > > also
> > > > > > > > has
> > > > > > > > > > > > changed
> > > > > > >

Re: Cluster auto activation design proposal

2017-11-07 Thread Dmitriy Setrakyan

; > > > > > > Dmitriy,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It was my misunderstanding, I believe that
> > > setter
> > > > > is
> > > > > > > not
> > > > > > > > > > enough
> > > > > > > > > > > > and
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > a full-fledged entity.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We should also be able to check if BLTs are
> > > > > > compatible.
> > > > > > > > > > > Interface
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > like this and use case for this
> functionality
> > > is
> > > > > > > > described
> > > > > > > > > > > below.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > interface BaselineTopology {
> > > > > > > > > > > > > > > > >Collection nodes();
> > > > > > > > > > > > > > > > >boolean isCompatibleWith(
> BaselineTopology
> > > > blt);
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Let's consider the following scenario:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >1. We have a grid with N nodes: it is
> up,
> > > > active
> > > > > > and
> > > > > > > > has
> > > > > > > > > > > data
> > > > > > > > > > > > in
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > > >BLT #1 created.
> > > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide it
> > into
> > > > two
> > > > > > > > parts:
> > > > > > > > > > > > > Part1_grid
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >Part2_grid.
> > > > > > > > > > > > > > > > >3. We start and activate Part1_grid .
> > > Topology
> > > > > has
> > > > > > > > > changed
> > > > > > > > > > > ->
> > > > > > > > > > > > > > BLT#2
> > > > > > > > > > > > > > > > >created.
> > > > > > > > > > > > > > > > >After that we shutdown that Part1_grid.
> > > > > > > > > > > > > > > > >4. We start and activate Part2_grid.
> > > Topology
> > > > > also
> > > > > > > has
> > > > > > > > > > > changed
> > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > > BLT#3
> > > > > > > > > > > > > > > > >created.
> > > > > > > > > > > > > > > > >5. Then we start Part1_grid and it's
> nodes
> > > try
> > > > > to
> > > > > > > join
> > > > > > > > > > > > > Part2_grid.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >

Re: Cluster auto activation design proposal

2017-11-07 Thread Alexey Goncharuk

 it is up,
> > > active
> > > > > and
> > > > > > > has
> > > > > > > > > > data
> > > > > > > > > > > in
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > ->
> > > > > > > > > > > > > > > >BLT #1 created.
> > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide it
> into
> > > two
> > > > > > > parts:
> > > > > > > > > > > > Part1_grid
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >Part2_grid.
> > > > > > > > > > > > > > > >3. We start and activate Part1_grid .
> > Topology
> > > > has
> > > > > > > > changed
> > > > > > > > > > ->
> > > > > > > > > > > > > BLT#2
> > > > > > > > > > > > > > > >created.
> > > > > > > > > > > > > > > >After that we shutdown that Part1_grid.
> > > > > > > > > > > > > > > >4. We start and activate Part2_grid.
> > Topology
> > > > also
> > > > > > has
> > > > > > > > > > changed
> > > > > > > > > > > > ->
> > > > > > > > > > > > > > > BLT#3
> > > > > > > > > > > > > > > >created.
> > > > > > > > > > > > > > > >5. Then we start Part1_grid and it's nodes
> > try
> > > > to
> > > > > > join
> > > > > > > > > > > > Part2_grid.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If join is successful we have an undefined
> > state
> > > of
> > > > > the
> > > > > > > > > > resulting
> > > > > > > > > > > > > grid:
> > > > > > > > > > > > > > > > values for the same key may (and will) differ
> > > > between
> > > > > > > grid
> > > > > > > > > > parts.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So to prevent this we should keep nodes with
> > > BLT#2
> > > > > from
> > > > > > > > > joining
> > > > > > > > > > > the
>

Re: Cluster auto activation design proposal

2017-11-07 Thread Dmitriy Setrakyan

 > > > > the
> > > > > > > > > > > > > grid
> > > > > > > > > > > > > > > with BLT#3. And we should fail nodes with an
> > error
> > > > > > message.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy
> > Setrakyan
> > > <
> > > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Sergey, I am still confused. What is the
> > > > > > BaselineTopology
> > > > > > > > > > > interface
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > example? I thought that you agreed with me
> that
> > > we
> > > > > > simply
> > > > > > > > > need
> > > > > > > > > > a
> > > > > > > > > > > > > setter
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > activation nodes, no?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey
> > Chugunov
> > > <
> > > > > > > > > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As I understand you use the term
> > > > > > > "minimalActivationNodes"
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > > > synonym
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > > > > > In that case I agree with you that we can
> > > replace
> > > > > > both
> > > > > > > > > > > > "establish*"
> > > > > > > > > > > > > > > > methods
> > > > > > > > > > > > > > > > > with a simple setter method (see below in
> > > > summary).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Summing up the whole discussion I see the
> > > > > > functionality
> > > > > > > > as
> > > > > > > > > > > > > following:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced.
> > The
> > > > > main
> > > > > > > > > features
> > > > > > > > > > > it
> > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >2. easy management of cluster topology
> > > changes
> > > > > > > > (planned
> > > > > > > > > > > nodes
> > > > > > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > > > > > short-term
> > > > > > > > node
> > > > > > > > > > > > > failures.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >1. user starts up new cluster of desired
> > > > number
> > > > > of
> > > > > > > > nodes
> > > > > > > > > > and
> > > > > > > > > > > > > > > activates
> > > > > > > > > > > > > > > > >it using existing API. BLT is created
> with
> > > all
> > > > > > nodes
> > > > > > > > > > > presented
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >cluster at the moment of activation, no
> > API
> > > is
> > > > > > > needed;
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >2. user prepares BLT using web-console
> or
> > > > visor
> > > > > > CMD
> > > > > > > > > tools
> > > > > > > > > > > and
> > > > > > > > > > > > > sets
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > > > > > eTopology(BaselineTopology
> > > > > > > > > > > > blt);
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >3. user provides via static
> configuration
> > a
> > > > list
> > > > > > of
> > > > > > > > > nodes
> > > > > > > > > > > that
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > > > > > >User starts nodes one by one; when all
> > > > > > preconfigured
> > > > > > > > > nodes
> > > > > > > > > > > are
> > > > > > > > > > > > > > > started
> > > > > > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > > > > > >As list of nodes may be huge it is
> > provided
> > > > via
> > > > > > > > separate
> > > > > > > > > > > file
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Igniters, does this description match with
> > your
> > > > > > > > > understanding
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > > functionality? If it does I'll create a set
> > of
> > > > > > tickets
> > > > > > > > and
> > > > > > > > > > > start
> > > > > > > > > > > > > > > working
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy
> > > > Setrakyan
> > > > > <
> > > > > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I still do not see why anyone would
> > > explicitly
> > > > > call
> > > > > > > > > these 2
> > > > > > > > > > > > > > methods:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > establishBaselineTopology(
> > > > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For example, if a web console, or some
> > other
> > > > > admin
> > > > > > > > > process,
> > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > automatically set currently started nodes
> > as
> > > > the
> > > > > > > > baseline
> > > > > > > > > > > > > topology,
> > > > > > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > > > > > minimalActivationNodes?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> > > > > Dmitriev <
> > > > > > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > API is proposed in the head of the
> thread
> > > by
> > > > > > > Sergey,
> > > > > > > > > as I
> > > > > > > > > > > > > > > understood:
> > > > > > > > > > > > > > > > > > > __
> > > > > > > > 
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation
> may
> > > > look
> > > > > > like
> > > > > > > > > this:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > > establishBaselineTopology(
> > > > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Both methods will establish BT and
> > activate
> > > > > > cluster
> > > > > > > > > once
> > > > > > > > > > it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The first one allows user to establish
> BT
> > > > using
> > > > > > > > current
> > > > > > > > > > > > > topology.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > changes happen to the topology during
> > > > > > establishing
> > > > > > > > > > process,
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > notified and allowed to proceed or
> abort
> > > the
> > > > > > > > procedure.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Second method allows to use some
> > > > > > > > > monitoring'n'management
> > > > > > > > > > > > tools
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > WebConsole where user can prepare a
> list
> > of
> > > > > > nodes,
> > > > > > > > > using
> > > > > > > > > > > them
> > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > > > > > and send to the cluster a command to
> > > finally
> > > > > > > > establish
> > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity
> > > > > contains
> > > > > > > only
> > > > > > > > > > > > > collection
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *TopologyNode* here contains
> information
> > > > about
> > > > > > > node -
> > > > > > > > > its
> > > > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > set of user attributes used to
> calculate
> > > > > affinity
> > > > > > > > > > function.
> > > > > > > > > > > > > > > > > > > __
> > > __
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > View this message in context:
> > > > > > > http://apache-ignite-
> > > > > > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers
> > > > mailing
> > > > > > list
> > > > > > > > > > archive
> > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-11-07 Thread Alexey Goncharuk

; > > > > ->
> > > > > > > > > > > > > >BLT #1 created.
> > > > > > > > > > > > > >2. We shutdown the grid. Then divide it into
> two
> > > > > parts:
> > > > > > > > > > Part1_grid
> > > > > > > > > > > > and
> > > > > > > > > > > > > >Part2_grid.
> > > > > > > > > > > > > >3. We start and activate Part1_grid . Topology
> > has
> > > > > > changed
> > > > > > > > ->
> > > > > > > > > > > BLT#2
> > > > > > > > > > > > > >created.
> > > > > > > > > > > > > >After that we shutdown that Part1_grid.
> > > > > > > > > > > > > >4. We start and activate Part2_grid. Topology
> > also
> > > > has
> > > > > > > > changed
> > > > > > > > > > ->
> > > > > > > > > > > > > BLT#3
> > > > > > > > > > > > > >created.
> > > > > > > > > > > > > >5. Then we start Part1_grid and it's nodes try
> > to
> > > > join
> > > > > > > > > > Part2_grid.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If join is successful we have an undefined state
> of
> > > the
> > > > > > > > resulting
> > > > > > > > > > > grid:
> > > > > > > > > > > > > > values for the same key may (and will) differ
> > between
> > > > > grid
> > > > > > > > parts.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So to prevent this we should keep nodes with
> BLT#2
> > > from
> > > > > > > joining
> > > > > > > > > the
> > > > > > > > > > > > grid
> > > > > > > > > > > > > > with BLT#3. And we should fail nodes with an
> error
> > > > > message.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy
> Setrakyan
> > <
> > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sergey, I am still confused. What is the
> > > > > BaselineTopology
> > > > > > > > > > interface
> > > > > > > > > > > > in
> > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > example? I thought that you agreed with me that
> > we
> > > > > simply
> > > > > > > > need
> > > > > > > > > a
> > > > > > > > > > > > setter
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > activation nodes, no?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey
> Chugunov
> > <
> > > > > > > > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As I understand you use the term
> > > > > > "minimalActivationNodes"
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > > > synonym
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > > > > In that case I agree with you that we can
> > replace
> > > > > both
> > > > > > > > > > > "establish*"
> > > > > > > > > > > > > > > methods
> > > > > > > > > > > > > > > > with a simple setter method (see below in
> > > summary).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Summing up the whole discussion I see the
> > > > > functionality
> > > > > > > as
> > > > > > > > > > > > following:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > New concept BaselineTopology is introduced.
> The
> > > > main
> > > > > > > > features
> > > > > > > > > > it
> > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >2. easy management of cluster topology
> > changes
> > > > > > > (planned
> > > > > > > > > > nodes
> > > > > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > > > > short-term
> > > > > > > node
> > > > > > > > > > > > failures.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >1. user starts up new cluster of desired
> > > number
> > > > of
> > > > > > > nodes
> > > > > > > > > and
> > > > > > > > > > > > > > activates
> > > > > > > > > > > > > > > >it using existing API. BLT is created with
> > all
> > > > > nodes
> > > > > > > > > > presented
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >cluster at the moment of activation, no
> API
> > is
> > > > > > needed;
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >2. user prepares BLT using web-console or
> > > visor
> > > > > CMD
> > > > > > > > tools
> > > > > > > > > > and
> > > > > > > > > > > > sets
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > > > > eTopology(BaselineTopology
> > > > > > > > > > > blt);
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >3. user provides via static configuration
> a
> > > list
> > > > > of
> > > > > > > > nodes
> > > > > > > > > > that
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > > > > >User starts nodes one by one; when all
> > > > > preconfigured
> > > > > > > > nodes
> > > > > > > > > > are
> > > > > > > > > > > > > > started
> > > > > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > > > > >As list of nodes may be huge it is
> provided
> > > via
> > > > > > > separate
> > > > > > > > > > file
> > > > > > > > > > > to
> > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Igniters, does this description match with
> your
> > > > > > > > understanding
> > > > > > > > > > of
> > > > > > > > > > > > > > > > functionality? If it does I'll create a set
> of
> > > > > tickets
> > > > > > > and
> > > > > > > > > > start
> > > > > > > > > > > > > > working
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy
> > > Setrakyan
> > > > <
> > > > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I still do not see why anyone would
> > explicitly
> > > > call
> > > > > > > > these 2
> > > > > > > > > > > > > methods:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > > *Ignite::activation::
> > > establishBaselineTopology(
> > > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For example, if a web console, or some
> other
> > > > admin
> > > > > > > > process,
> > > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > automatically set currently started nodes
> as
> > > the
> > > > > > > baseline
> > > > > > > > > > > > topology,
> > > > > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > > > > minimalActivationNodes?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> > > > Dmitriev <
> > > > > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > API is proposed in the head of the thread
> > by
> > > > > > Sergey,
> > > > > > > > as I
> > > > > > > > > > > > > > understood:
> > > > > > > > > > > > > > > > > > __
> > > > > > > 
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may
> > > look
> > > > > like
> > > > > > > > this:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > establishBaselineTopology(
> > > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Both methods will establish BT and
> activate
> > > > > cluster
> > > > > > > > once
> > > > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The first one allows user to establish BT
> > > using
> > > > > > > current
> > > > > > > > > > > > topology.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > changes happen to the topology during
> > > > > establishing
> > > > > > > > > process,
> > > > > > > > > > > > user
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > notified and allowed to proceed or abort
> > the
> > > > > > > procedure.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Second method allows to use some
> > > > > > > > monitoring'n'management
> > > > > > > > > > > tools
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > WebConsole where user can prepare a list
> of
> > > > > nodes,
> > > > > > > > using
> > > > > > > > > > them
> > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > > > > and send to the cluster a command to
> > finally
> > > > > > > establish
> > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > From high level BaselineTopology entity
> > > > contains
> > > > > > only
> > > > > > > > > > > > collection
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *TopologyNode* here contains information
> > > about
> > > > > > node -
> > > > > > > > its
> > > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > set of user attributes used to calculate
> > > > affinity
> > > > > > > > > function.
> > > > > > > > > > > > > > > > > > __
> > __
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > View this message in context:
> > > > > > http://apache-ignite-
> > > > > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers
> > > mailing
> > > > > list
> > > > > > > > > archive
> > > > > > > > > > > at
> > > > > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-11-06 Thread Dmitriy Setrakyan

e should keep nodes with BLT#2
> > from
> > > > > > joining
> > > > > > > > the
> > > > > > > > > > > grid
> > > > > > > > > > > > > with BLT#3. And we should fail nodes with an error
> > > > message.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan
> <
> > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Sergey, I am still confused. What is the
> > > > BaselineTopology
> > > > > > > > > interface
> > > > > > > > > > > in
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > example? I thought that you agreed with me that
> we
> > > > simply
> > > > > > > need
> > > > > > > > a
> > > > > > > > > > > setter
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > activation nodes, no?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > D.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov
> <
> > > > > > > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As I understand you use the term
> > > > > "minimalActivationNodes"
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > > synonym
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > > > In that case I agree with you that we can
> replace
> > > > both
> > > > > > > > > > "establish*"
> > > > > > > > > > > > > > methods
> > > > > > > > > > > > > > > with a simple setter method (see below in
> > summary).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Summing up the whole discussion I see the
> > > > functionality
> > > > > > as
> > > > > > > > > > > following:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The
> > > main
> > > > > > > features
> > > > > > > > > it
> > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >2. easy management of cluster topology
> changes
> > > > > > (planned
> > > > > > > > > nodes
> > > > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > > > short-term
> > > > > > node
> > > > > > > > > > > failures.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >1. user starts up new cluster of desired
> > number
> > > of
> > > > > > nodes
> > > > > > > > and
> > > > > > > > > > > > > activates
> > > > > > > > > > > > > > >it using existing API. BLT is created with
> all
> > > > nodes
> > > > > > > > > presented
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >cluster at the moment of activation, no API
> is
> > > > > needed;
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >2. user prepares BLT using web-console or
> > visor
> > > > CMD
> > > > > > > tools
> > > > > > > > > and
> > > > > > > > > > > sets
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > > > eTopology(BaselineTopology
> > > > > > > > > > blt);
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >3. user provides via static configuration a
> > list
> > > > of
> > > > > > > nodes
> > > > > > > > > that
> > > > > > > > > > > are
> > > > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > > > >User starts nodes one by one; when all
> > > > preconfigured
> > > > > > > nodes
> > > > > > > > > are
> > > > > > > > > > > > > started
> > > > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > > > >As list of nodes may be huge it is provided
> > via
> > > > > > separate
> > > > > > > > > file
> > > > > > > > > > to
> > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Igniters, does this description match with your
> > > > > > > understanding
> > > > > > > > > of
> > > > > > > > > > > > > > > functionality? If it does I'll create a set of
> > > > tickets
> > > > > > and
> > > > > > > > > start
> > > > > > > > > > > > > working
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy
> > Setrakyan
> > > <
> > > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I still do not see why anyone would
> explicitly
> > > call
> > > > > > > these 2
> > > > > > > > > > > > methods:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Ignite::activation::
> > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology(
> > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For example, if a web console, or some other
> > > admin
> > > > > > > process,
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > automatically set currently started nodes as
> > the
> > > > > > baseline
> > > > > > > > > > > topology,
> > > > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > > > minimalActivationNodes?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> > > Dmitriev <
> > > > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > API is proposed in the head of the thread
> by
> > > > > Sergey,
> > > > > > > as I
> > > > > > > > > > > > > understood:
> > > > > > > > > > > > > > > > > __
> > > > > > 
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may
> > look
> > > > like
> > > > > > > this:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Ignite::activation::
> > > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > > *Ignite::activation::
> > > establishBaselineTopology(
> > > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Both methods will establish BT and activate
> > > > cluster
> > > > > > > once
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The first one allows user to establish BT
> > using
> > > > > > current
> > > > > > > > > > > topology.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > changes happen to the topology during
> > > > establishing
> > > > > > > > process,
> > > > > > > > > > > user
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > notified and allowed to proceed or abort
> the
> > > > > > procedure.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Second method allows to use some
> > > > > > > monitoring'n'management
> > > > > > > > > > tools
> > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > WebConsole where user can prepare a list of
> > > > nodes,
> > > > > > > using
> > > > > > > > > them
> > > > > > > > > > > > > create
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > > > and send to the cluster a command to
> finally
> > > > > > establish
> > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > From high level BaselineTopology entity
> > > contains
> > > > > only
> > > > > > > > > > > collection
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *TopologyNode* here contains information
> > about
> > > > > node -
> > > > > > > its
> > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > id
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > set of user attributes used to calculate
> > > affinity
> > > > > > > > function.
> > > > > > > > > > > > > > > > > __
> __
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > View this message in context:
> > > > > http://apache-ignite-
> > > > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers
> > mailing
> > > > list
> > > > > > > > archive
> > > > > > > > > > at
> > > > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-11-04 Thread Alexey Goncharuk

 > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As I understand you use the term
> > > > "minimalActivationNodes"
> > > > > > as
> > > > > > > a
> > > > > > > > > > > synonym
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > > In that case I agree with you that we can replace
> > > both
> > > > > > > > > "establish*"
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > with a simple setter method (see below in
> summary).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Summing up the whole discussion I see the
> > > functionality
> > > > > as
> > > > > > > > > > following:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > New concept BaselineTopology is introduced. The
> > main
> > > > > > features
> > > > > > > > it
> > > > > > > > > > > > enables
> > > > > > > > > > > > > > are:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >2. easy management of cluster topology changes
> > > > > (planned
> > > > > > > > nodes
> > > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > > short-term
> > > > > node
> > > > > > > > > > failures.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >1. user starts up new cluster of desired
> number
> > of
> > > > > nodes
> > > > > > > and
> > > > > > > > > > > > activates
> > > > > > > > > > > > > >it using existing API. BLT is created with all
> > > nodes
> > > > > > > > presented
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > >cluster at the moment of activation, no API is
> > > > needed;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >2. user prepares BLT using web-console or
> visor
> > > CMD
> > > > > > tools
> > > > > > > > and
> > > > > > > > > > sets
> > > > > > > > > > > > it
> > > > > > > > > > > > > to
> > > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > > eTopology(BaselineTopology
> > > > > > > > > blt);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >3. user provides via static configuration a
> list
> > > of
> > > > > > nodes
> > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > > >User starts nodes one by one; when all
> > > preconfigured
> > > > > > nodes
> > > > > > > > are
> > > > > > > > > > > > started
> > > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > > >As list of nodes may be huge it is provided
> via
> > > > > separate
> > > > > > > > file
> > > > > > > > > to
> > > > > > > > > > > > avoid
> > > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Igniters, does this description match with your
> > > > > > understanding
> > > > > > > > of
> > > > > > > > > > > > > > functionality? If it does I'll create a set of
> > > tickets
> > > > > and
> > > > > > > > start
> > > > > > > > > > > > working
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy
> Setrakyan
> > <
> > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I still do not see why anyone would explicitly
> > call
> > > > > > these 2
> > > > > > > > > > > methods:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology();*
> > > > > > > > > > > > > > > *Ignite::activation::
> establishBaselineTopology(
> > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For example, if a web console, or some other
> > admin
> > > > > > process,
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > > automatically set currently started nodes as
> the
> > > > > baseline
> > > > > > > > > > topology,
> > > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > > minimalActivationNodes?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> > Dmitriev <
> > > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > API is proposed in the head of the thread by
> > > > Sergey,
> > > > > > as I
> > > > > > > > > > > > understood:
> > > > > > > > > > > > > > > > __
> > > > > 
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > API for BaselineTopology manipulation may
> look
> > > like
> > > > > > this:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Ignite::activation::
> > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology(
> > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Both methods will establish BT and activate
> > > cluster
> > > > > > once
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The first one allows user to establish BT
> using
> > > > > current
> > > > > > > > > > topology.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > changes happen to the topology during
> > > establishing
> > > > > > > process,
> > > > > > > > > > user
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > notified and allowed to proceed or abort the
> > > > > procedure.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Second method allows to use some
> > > > > > monitoring'n'management
> > > > > > > > > tools
> > > > > > > > > > > like
> > > > > > > > > > > > > > > > WebConsole where user can prepare a list of
> > > nodes,
> > > > > > using
> > > > > > > > them
> > > > > > > > > > > > create
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > > and send to the cluster a command to finally
> > > > > establish
> > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > From high level BaselineTopology entity
> > contains
> > > > only
> > > > > > > > > > collection
> > > > > > > > > > > of
> > > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *TopologyNode* here contains information
> about
> > > > node -
> > > > > > its
> > > > > > > > > > > > consistent
> > > > > > > > > > > > > id
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > set of user attributes used to calculate
> > affinity
> > > > > > > function.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > View this message in context:
> > > > http://apache-ignite-
> > > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers
> mailing
> > > list
> > > > > > > archive
> > > > > > > > > at
> > > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-11-04 Thread Ilya Lantukh

> > > > > > > > > >
> > > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As I understand you use the term
> > > > "minimalActivationNodes"
> > > > > > as
> > > > > > > a
> > > > > > > > > > > synonym
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > > In that case I agree with you that we can replace
> > > both
> > > > > > > > > "establish*"
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > with a simple setter method (see below in
> summary).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Summing up the whole discussion I see the
> > > functionality
> > > > > as
> > > > > > > > > > following:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > New concept BaselineTopology is introduced. The
> > main
> > > > > > features
> > > > > > > > it
> > > > > > > > > > > > enables
> > > > > > > > > > > > > > are:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >2. easy management of cluster topology changes
> > > > > (planned
> > > > > > > > nodes
> > > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > > short-term
> > > > > node
> > > > > > > > > > failures.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >1. user starts up new cluster of desired
> number
> > of
> > > > > nodes
> > > > > > > and
> > > > > > > > > > > > activates
> > > > > > > > > > > > > >it using existing API. BLT is created with all
> > > nodes
> > > > > > > > presented
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > >cluster at the moment of activation, no API is
> > > > needed;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >2. user prepares BLT using web-console or
> visor
> > > CMD
> > > > > > tools
> > > > > > > > and
> > > > > > > > > > sets
> > > > > > > > > > > > it
> > > > > > > > > > > > > to
> > > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > > eTopology(BaselineTopology
> > > > > > > > > blt);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >3. user provides via static configuration a
> list
> > > of
> > > > > > nodes
> > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > > >User starts nodes one by one; when all
> > > preconfigured
> > > > > > nodes
> > > > > > > > are
> > > > > > > > > > > > started
> > > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > > >As list of nodes may be huge it is provided
> via
> > > > > separate
> > > > > > > > file
> > > > > > > > > to
> > > > > > > > > > > > avoid
> > > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Igniters, does this description match with your
> > > > > > understanding
> > > > > > > > of
> > > > > > > > > > > > > > functionality? If it does I'll create a set of
> > > tickets
> > > > > and
> > > > > > > > start
> > > > > > > > > > > > working
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy
> Setrakyan
> > <
> > > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I still do not see why anyone would explicitly
> > call
> > > > > > these 2
> > > > > > > > > > > methods:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology();*
> > > > > > > > > > > > > > > *Ignite::activation::
> establishBaselineTopology(
> > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For example, if a web console, or some other
> > admin
> > > > > > process,
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > > automatically set currently started nodes as
> the
> > > > > baseline
> > > > > > > > > > topology,
> > > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > > minimalActivationNodes?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > D.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> > Dmitriev <
> > > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > API is proposed in the head of the thread by
> > > > Sergey,
> > > > > > as I
> > > > > > > > > > > > understood:
> > > > > > > > > > > > > > > > __
> > > > > 
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > API for BaselineTopology manipulation may
> look
> > > like
> > > > > > this:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Ignite::activation::
> > > establishBaselineTopology();*
> > > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology(
> > > > > > > > > BaselineTopology
> > > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Both methods will establish BT and activate
> > > cluster
> > > > > > once
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The first one allows user to establish BT
> using
> > > > > current
> > > > > > > > > > topology.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > changes happen to the topology during
> > > establishing
> > > > > > > process,
> > > > > > > > > > user
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > notified and allowed to proceed or abort the
> > > > > procedure.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Second method allows to use some
> > > > > > monitoring'n'management
> > > > > > > > > tools
> > > > > > > > > > > like
> > > > > > > > > > > > > > > > WebConsole where user can prepare a list of
> > > nodes,
> > > > > > using
> > > > > > > > them
> > > > > > > > > > > > create
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > > and send to the cluster a command to finally
> > > > > establish
> > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > From high level BaselineTopology entity
> > contains
> > > > only
> > > > > > > > > > collection
> > > > > > > > > > > of
> > > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *TopologyNode* here contains information
> about
> > > > node -
> > > > > > its
> > > > > > > > > > > > consistent
> > > > > > > > > > > > > id
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > set of user attributes used to calculate
> > affinity
> > > > > > > function.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > View this message in context:
> > > > http://apache-ignite-
> > > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers
> mailing
> > > list
> > > > > > > archive
> > > > > > > > > at
> > > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Best regards,
Ilya

Re: Cluster auto activation design proposal

2017-11-04 Thread Dmitriy Setrakyan

 > > > > Part2_grid.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If join is successful we have an undefined state of the
> > > > > resulting
> > > > > > > > grid:
> > > > > > > > > > > values for the same key may (and will) differ between
> > grid
> > > > > parts.
> > > > > > > > > > >
> > > > > > > > > > > So to prevent this we should keep nodes with BLT#2 from
> > > > joining
> > > > > > the
> > > > > > > > > grid
> > > > > > > > > > > with BLT#3. And we should fail nodes with an error
> > message.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Sergey.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan <
> > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Sergey, I am still confused. What is the
> > BaselineTopology
> > > > > > > interface
> > > > > > > > > in
> > > > > > > > > > > your
> > > > > > > > > > > > example? I thought that you agreed with me that we
> > simply
> > > > > need
> > > > > > a
> > > > > > > > > setter
> > > > > > > > > > > for
> > > > > > > > > > > > activation nodes, no?
> > > > > > > > > > > >
> > > > > > > > > > > > D.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > > > > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > > >
> > > > > > > > > > > > > As I understand you use the term
> > > "minimalActivationNodes"
> > > > > as
> > > > > > a
> > > > > > > > > > synonym
> > > > > > > > > > > > for
> > > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > > In that case I agree with you that we can replace
> > both
> > > > > > > > "establish*"
> > > > > > > > > > > > methods
> > > > > > > > > > > > > with a simple setter method (see below in summary).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Summing up the whole discussion I see the
> > functionality
> > > > as
> > > > > > > > > following:
> > > > > > > > > > > > >
> > > > > > > > > > > > > New concept BaselineTopology is introduced. The
> main
> > > > > features
> > > > > > > it
> > > > > > > > > > > enables
> > > > > > > > > > > > > are:
> > > > > > > > > > > > >
> > > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > > >
> > > > > > > > > > > > >2. easy management of cluster topology changes
> > > > (planned
> > > > > > > nodes
> > > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > > >
> > > > > > > > > > > > >3. eliminating of rebalancing traffic on
> > short-term
> > > > node
> > > > > > > > > failures.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > > >
> > > > > > > > > > > > >1. user starts up new cluster of desired number
> of
> > > > nodes
> > > > > > and
> > > > > > > > > > > activates
> > > > > > > > > > > > >it using existing API. BLT is created with all
> > nodes
> > > > > > > presented
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > >cluster at the moment of activation, no API is
> > > needed;
> > > > > > > > > > > > >
> > > > > > > > > > > > >2. user prepares BLT using web-console or visor
> > CMD
> > > > > tools
> > > > > > > and
> > > > > > > > > sets
> > > > > > > > > > > it
> > > > > > > > > > > > to
> > > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > > >Ignite.activation().setBaselin
> > > > > eTopology(BaselineTopology
> > > > > > > > blt);
> > > > > > > > > > > > >
> > > > > > > > > > > > >3. user provides via static configuration a list
> > of
> > > > > nodes
> > > > > > > that
> > > > > > > > > are
> > > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > > >User starts nodes one by one; when all
> > preconfigured
> > > > > nodes
> > > > > > > are
> > > > > > > > > > > started
> > > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > > >As list of nodes may be huge it is provided via
> > > > separate
> > > > > > > file
> > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Igniters, does this description match with your
> > > > > understanding
> > > > > > > of
> > > > > > > > > > > > > functionality? If it does I'll create a set of
> > tickets
> > > > and
> > > > > > > start
> > > > > > > > > > > working
> > > > > > > > > > > > on
> > > > > > > > > > > > > implementation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Sergey.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan
> <
> > > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I still do not see why anyone would explicitly
> call
> > > > > these 2
> > > > > > > > > > methods:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Ignite::activation::
> establishBaselineTopology();*
> > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > > > > BaselineTopology
> > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For example, if a web console, or some other
> admin
> > > > > process,
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > > > automatically set currently started nodes as the
> > > > baseline
> > > > > > > > > topology,
> > > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > > minimalActivationNodes?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > D.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey
> Dmitriev <
> > > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > API is proposed in the head of the thread by
> > > Sergey,
> > > > > as I
> > > > > > > > > > > understood:
> > > > > > > > > > > > > > > __
> > > > 
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > API for BaselineTopology manipulation may look
> > like
> > > > > this:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Ignite::activation::
> > establishBaselineTopology();*
> > > > > > > > > > > > > > > *Ignite::activation::
> establishBaselineTopology(
> > > > > > > > BaselineTopology
> > > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Both methods will establish BT and activate
> > cluster
> > > > > once
> > > > > > it
> > > > > > > > is
> > > > > > > > > > > > > > established.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The first one allows user to establish BT using
> > > > current
> > > > > > > > > topology.
> > > > > > > > > > > If
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > changes happen to the topology during
> > establishing
> > > > > > process,
> > > > > > > > > user
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > notified and allowed to proceed or abort the
> > > > procedure.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Second method allows to use some
> > > > > monitoring'n'management
> > > > > > > > tools
> > > > > > > > > > like
> > > > > > > > > > > > > > > WebConsole where user can prepare a list of
> > nodes,
> > > > > using
> > > > > > > them
> > > > > > > > > > > create
> > > > > > > > > > > > a
> > > > > > > > > > > > > BT
> > > > > > > > > > > > > > > and send to the cluster a command to finally
> > > > establish
> > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > From high level BaselineTopology entity
> contains
> > > only
> > > > > > > > > collection
> > > > > > > > > > of
> > > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *TopologyNode* here contains information about
> > > node -
> > > > > its
> > > > > > > > > > > consistent
> > > > > > > > > > > > id
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > set of user attributes used to calculate
> affinity
> > > > > > function.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > View this message in context:
> > > http://apache-ignite-
> > > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing
> > list
> > > > > > archive
> > > > > > > > at
> > > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-11-04 Thread Alexey Goncharuk

 > > > > in
> > > > > > > > > > your
> > > > > > > > > > > example? I thought that you agreed with me that we
> simply
> > > > need
> > > > > a
> > > > > > > > setter
> > > > > > > > > > for
> > > > > > > > > > > activation nodes, no?
> > > > > > > > > > >
> > > > > > > > > > > D.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > > > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dmitriy,
> > > > > > > > > > > >
> > > > > > > > > > > > As I understand you use the term
> > "minimalActivationNodes"
> > > > as
> > > > > a
> > > > > > > > > synonym
> > > > > > > > > > > for
> > > > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > > > In that case I agree with you that we can replace
> both
> > > > > > > "establish*"
> > > > > > > > > > > methods
> > > > > > > > > > > > with a simple setter method (see below in summary).
> > > > > > > > > > > >
> > > > > > > > > > > > Summing up the whole discussion I see the
> functionality
> > > as
> > > > > > > > following:
> > > > > > > > > > > >
> > > > > > > > > > > > New concept BaselineTopology is introduced. The main
> > > > features
> > > > > > it
> > > > > > > > > > enables
> > > > > > > > > > > > are:
> > > > > > > > > > > >
> > > > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > > > >
> > > > > > > > > > > >2. easy management of cluster topology changes
> > > (planned
> > > > > > nodes
> > > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > > >
> > > > > > > > > > > >3. eliminating of rebalancing traffic on
> short-term
> > > node
> > > > > > > > failures.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > > >
> > > > > > > > > > > >1. user starts up new cluster of desired number of
> > > nodes
> > > > > and
> > > > > > > > > > activates
> > > > > > > > > > > >it using existing API. BLT is created with all
> nodes
> > > > > > presented
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > >cluster at the moment of activation, no API is
> > needed;
> > > > > > > > > > > >
> > > > > > > > > > > >2. user prepares BLT using web-console or visor
> CMD
> > > > tools
> > > > > > and
> > > > > > > > sets
> > > > > > > > > > it
> > > > > > > > > > > to
> > > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > > >Ignite.activation().setBaselin
> > > > eTopology(BaselineTopology
> > > > > > > blt);
> > > > > > > > > > > >
> > > > > > > > > > > >3. user provides via static configuration a list
> of
> > > > nodes
> > > > > > that
> > > > > > > > are
> > > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > > >User starts nodes one by one; when all
> preconfigured
> > > > nodes
> > > > > > are
> > > > > > > > > > started
> > > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > > >As list of nodes may be huge it is provided via
> > > separate
> > > > > > file
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > >flooding main configuration.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Igniters, does this description match with your
> > > > understanding
> > > > > > of
> > > > > > > > > > > > functionality? If it does I'll create a set of
> tickets
> > > and
> > > > > > start
> > > > > > > > > > working
> > > > > > > > > > > on
> > > > > > > > > > > > implementation.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Sergey.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I still do not see why anyone would explicitly call
> > > > these 2
> > > > > > > > > methods:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > > > BaselineTopology
> > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > >
> > > > > > > > > > > > > For example, if a web console, or some other admin
> > > > process,
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > > > > automatically set currently started nodes as the
> > > baseline
> > > > > > > > topology,
> > > > > > > > > > > > > shouldn't they just call a setter for
> > > > > minimalActivationNodes?
> > > > > > > > > > > > >
> > > > > > > > > > > > > D.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > API is proposed in the head of the thread by
> > Sergey,
> > > > as I
> > > > > > > > > > understood:
> > > > > > > > > > > > > > __
> > > 
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > API for BaselineTopology manipulation may look
> like
> > > > this:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Ignite::activation::
> establishBaselineTopology();*
> > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > > > BaselineTopology
> > > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Both methods will establish BT and activate
> cluster
> > > > once
> > > > > it
> > > > > > > is
> > > > > > > > > > > > > established.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The first one allows user to establish BT using
> > > current
> > > > > > > > topology.
> > > > > > > > > > If
> > > > > > > > > > > > any
> > > > > > > > > > > > > > changes happen to the topology during
> establishing
> > > > > process,
> > > > > > > > user
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > notified and allowed to proceed or abort the
> > > procedure.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Second method allows to use some
> > > > monitoring'n'management
> > > > > > > tools
> > > > > > > > > like
> > > > > > > > > > > > > > WebConsole where user can prepare a list of
> nodes,
> > > > using
> > > > > > them
> > > > > > > > > > create
> > > > > > > > > > > a
> > > > > > > > > > > > BT
> > > > > > > > > > > > > > and send to the cluster a command to finally
> > > establish
> > > > > it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From high level BaselineTopology entity contains
> > only
> > > > > > > > collection
> > > > > > > > > of
> > > > > > > > > > > > > nodes:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > > *}*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *TopologyNode* here contains information about
> > node -
> > > > its
> > > > > > > > > > consistent
> > > > > > > > > > > id
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > set of user attributes used to calculate affinity
> > > > > function.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > View this message in context:
> > http://apache-ignite-
> > > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing
> list
> > > > > archive
> > > > > > > at
> > > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-09-04 Thread Dmitriy Setrakyan

 automatic activation of cluster;
> > > > > > > > > > >
> > > > > > > > > > >2. easy management of cluster topology changes
> > (planned
> > > > > nodes
> > > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > > >
> > > > > > > > > > >3. eliminating of rebalancing traffic on short-term
> > node
> > > > > > > failures.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Use cases to create BLT:
> > > > > > > > > > >
> > > > > > > > > > >1. user starts up new cluster of desired number of
> > nodes
> > > > and
> > > > > > > > > activates
> > > > > > > > > > >it using existing API. BLT is created with all nodes
> > > > > presented
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >cluster at the moment of activation, no API is
> needed;
> > > > > > > > > > >
> > > > > > > > > > >2. user prepares BLT using web-console or visor CMD
> > > tools
> > > > > and
> > > > > > > sets
> > > > > > > > > it
> > > > > > > > > > to
> > > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > > >Ignite.activation().setBaselin
> > > eTopology(BaselineTopology
> > > > > > blt);
> > > > > > > > > > >
> > > > > > > > > > >3. user provides via static configuration a list of
> > > nodes
> > > > > that
> > > > > > > are
> > > > > > > > > > >expected to be in the cluster.
> > > > > > > > > > >User starts nodes one by one; when all preconfigured
> > > nodes
> > > > > are
> > > > > > > > > started
> > > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > > >As list of nodes may be huge it is provided via
> > separate
> > > > > file
> > > > > > to
> > > > > > > > > avoid
> > > > > > > > > > >flooding main configuration.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Igniters, does this description match with your
> > > understanding
> > > > > of
> > > > > > > > > > > functionality? If it does I'll create a set of tickets
> > and
> > > > > start
> > > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > implementation.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Sergey.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I still do not see why anyone would explicitly call
> > > these 2
> > > > > > > > methods:
> > > > > > > > > > > >
> > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > > BaselineTopology
> > > > > > > > > > > bltTop);*
> > > > > > > > > > > >
> > > > > > > > > > > > For example, if a web console, or some other admin
> > > process,
> > > > > > want
> > > > > > > to
> > > > > > > > > > > > automatically set currently started nodes as the
> > baseline
> > > > > > > topology,
> > > > > > > > > > > > shouldn't they just call a setter for
> > > > minimalActivationNodes?
> > > > > > > > > > > >
> > > > > > > > > > > > D.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > API is proposed in the head of the thread by
> Sergey,
> > > as I
> > > > > > > > > understood:
> > > > > > > > > > > > > __
> > 
> > > > > > > > > > > > >
> > > > > > > > > > > > > API for BaselineTopology manipulation may look like
> > > this:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > > BaselineTopology
> > > > > > > > > > > > bltTop);*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Both methods will establish BT and activate cluster
> > > once
> > > > it
> > > > > > is
> > > > > > > > > > > > established.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The first one allows user to establish BT using
> > current
> > > > > > > topology.
> > > > > > > > > If
> > > > > > > > > > > any
> > > > > > > > > > > > > changes happen to the topology during establishing
> > > > process,
> > > > > > > user
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > notified and allowed to proceed or abort the
> > procedure.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Second method allows to use some
> > > monitoring'n'management
> > > > > > tools
> > > > > > > > like
> > > > > > > > > > > > > WebConsole where user can prepare a list of nodes,
> > > using
> > > > > them
> > > > > > > > > create
> > > > > > > > > > a
> > > > > > > > > > > BT
> > > > > > > > > > > > > and send to the cluster a command to finally
> > establish
> > > > it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > From high level BaselineTopology entity contains
> only
> > > > > > > collection
> > > > > > > > of
> > > > > > > > > > > > nodes:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > > *}*
> > > > > > > > > > > > >
> > > > > > > > > > > > > *TopologyNode* here contains information about
> node -
> > > its
> > > > > > > > > consistent
> > > > > > > > > > id
> > > > > > > > > > > > and
> > > > > > > > > > > > > set of user attributes used to calculate affinity
> > > > function.
> > > > > > > > > > > > > 
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > View this message in context:
> http://apache-ignite-
> > > > > > > > > > > > > developers.2346864.n4.nabble.
> > > > com/Cluster-auto-activation-
> > > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing list
> > > > archive
> > > > > > at
> > > > > > > > > > > > Nabble.com.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-09-04 Thread Sergey Chugunov

ld keep nodes with BLT#2 from
> joining
> > > the
> > > > > > grid
> > > > > > > > with BLT#3. And we should fail nodes with an error message.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Sergey.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan <
> > > > > > > dsetrak...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sergey, I am still confused. What is the BaselineTopology
> > > > interface
> > > > > > in
> > > > > > > > your
> > > > > > > > > example? I thought that you agreed with me that we simply
> > need
> > > a
> > > > > > setter
> > > > > > > > for
> > > > > > > > > activation nodes, no?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > > > > > > > > sergey.chugu...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dmitriy,
> > > > > > > > > >
> > > > > > > > > > As I understand you use the term "minimalActivationNodes"
> > as
> > > a
> > > > > > > synonym
> > > > > > > > > for
> > > > > > > > > > BaselineTopology concept.
> > > > > > > > > > In that case I agree with you that we can replace both
> > > > > "establish*"
> > > > > > > > > methods
> > > > > > > > > > with a simple setter method (see below in summary).
> > > > > > > > > >
> > > > > > > > > > Summing up the whole discussion I see the functionality
> as
> > > > > > following:
> > > > > > > > > >
> > > > > > > > > > New concept BaselineTopology is introduced. The main
> > features
> > > > it
> > > > > > > > enables
> > > > > > > > > > are:
> > > > > > > > > >
> > > > > > > > > >1. automatic activation of cluster;
> > > > > > > > > >
> > > > > > > > > >2. easy management of cluster topology changes
> (planned
> > > > nodes
> > > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > > >
> > > > > > > > > >3. eliminating of rebalancing traffic on short-term
> node
> > > > > > failures.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Use cases to create BLT:
> > > > > > > > > >
> > > > > > > > > >1. user starts up new cluster of desired number of
> nodes
> > > and
> > > > > > > > activates
> > > > > > > > > >it using existing API. BLT is created with all nodes
> > > > presented
> > > > > > in
> > > > > > > > the
> > > > > > > > > >cluster at the moment of activation, no API is needed;
> > > > > > > > > >
> > > > > > > > > >2. user prepares BLT using web-console or visor CMD
> > tools
> > > > and
> > > > > > sets
> > > > > > > > it
> > > > > > > > > to
> > > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > > >Ignite.activation().setBaselin
> > eTopology(BaselineTopology
> > > > > blt);
> > > > > > > > > >
> > > > > > > > > >3. user provides via static configuration a list of
> > nodes
> > > > that
> > > > > > are
> > > > > > > > > >expected to be in the cluster.
> > > > > > > > > >User starts nodes one by one; when all preconfigured
> > nodes
> > > > are
> > > > > > > > started
> > > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > > >As list of nodes may be huge it is provided via
> separate
> > > > file
> > > > > to
> > > > > > > > avoid
> > > > > > > > > >flooding main configuration.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Igniters, does this description match with your
> > understanding
> > > > of
> > > > > > > > > > functionality? If it does I'll create a set of tickets
> and
> > > > start
> > > > > > > > working
> > > > > > > > > on
> > > > > > > > > > implementation.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Sergey.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > > > > dsetrak...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I still do not see why anyone would explicitly call
> > these 2
> > > > > > > methods:
> > > > > > > > > > >
> > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > BaselineTopology
> > > > > > > > > > bltTop);*
> > > > > > > > > > >
> > > > > > > > > > > For example, if a web console, or some other admin
> > process,
> > > > > want
> > > > > > to
> > > > > > > > > > > automatically set currently started nodes as the
> baseline
> > > > > > topology,
> > > > > > > > > > > shouldn't they just call a setter for
> > > minimalActivationNodes?
> > > > > > > > > > >
> > > > > > > > > > > D.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > API is proposed in the head of the thread by Sergey,
> > as I
> > > > > > > > understood:
> > > > > > > > > > > > __
> 
> > > > > > > > > > > >
> > > > > > > > > > > > API for BaselineTopology manipulation may look like
> > this:
> > > > > > > > > > > >
> > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > > BaselineTopology
> > > > > > > > > > > bltTop);*
> > > > > > > > > > > >
> > > > > > > > > > > > Both methods will establish BT and activate cluster
> > once
> > > it
> > > > > is
> > > > > > > > > > > established.
> > > > > > > > > > > >
> > > > > > > > > > > > The first one allows user to establish BT using
> current
> > > > > > topology.
> > > > > > > > If
> > > > > > > > > > any
> > > > > > > > > > > > changes happen to the topology during establishing
> > > process,
> > > > > > user
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > notified and allowed to proceed or abort the
> procedure.
> > > > > > > > > > > >
> > > > > > > > > > > > Second method allows to use some
> > monitoring'n'management
> > > > > tools
> > > > > > > like
> > > > > > > > > > > > WebConsole where user can prepare a list of nodes,
> > using
> > > > them
> > > > > > > > create
> > > > > > > > > a
> > > > > > > > > > BT
> > > > > > > > > > > > and send to the cluster a command to finally
> establish
> > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > From high level BaselineTopology entity contains only
> > > > > > collection
> > > > > > > of
> > > > > > > > > > > nodes:
> > > > > > > > > > > >
> > > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > > *}*
> > > > > > > > > > > >
> > > > > > > > > > > > *TopologyNode* here contains information about node -
> > its
> > > > > > > > consistent
> > > > > > > > > id
> > > > > > > > > > > and
> > > > > > > > > > > > set of user attributes used to calculate affinity
> > > function.
> > > > > > > > > > > > 
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > View this message in context: http://apache-ignite-
> > > > > > > > > > > > developers.2346864.n4.nabble.
> > > com/Cluster-auto-activation-
> > > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > > Sent from the Apache Ignite Developers mailing list
> > > archive
> > > > > at
> > > > > > > > > > > Nabble.com.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-09-01 Thread Dmitriy Setrakyan

t; > >
> > > > > > > > > Dmitriy,
> > > > > > > > >
> > > > > > > > > As I understand you use the term "minimalActivationNodes"
> as
> > a
> > > > > > synonym
> > > > > > > > for
> > > > > > > > > BaselineTopology concept.
> > > > > > > > > In that case I agree with you that we can replace both
> > > > "establish*"
> > > > > > > > methods
> > > > > > > > > with a simple setter method (see below in summary).
> > > > > > > > >
> > > > > > > > > Summing up the whole discussion I see the functionality as
> > > > > following:
> > > > > > > > >
> > > > > > > > > New concept BaselineTopology is introduced. The main
> features
> > > it
> > > > > > > enables
> > > > > > > > > are:
> > > > > > > > >
> > > > > > > > >1. automatic activation of cluster;
> > > > > > > > >
> > > > > > > > >2. easy management of cluster topology changes (planned
> > > nodes
> > > > > > > > >maintenance, adding new nodes etc);
> > > > > > > > >
> > > > > > > > >3. eliminating of rebalancing traffic on short-term node
> > > > > failures.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Use cases to create BLT:
> > > > > > > > >
> > > > > > > > >1. user starts up new cluster of desired number of nodes
> > and
> > > > > > > activates
> > > > > > > > >it using existing API. BLT is created with all nodes
> > > presented
> > > > > in
> > > > > > > the
> > > > > > > > >cluster at the moment of activation, no API is needed;
> > > > > > > > >
> > > > > > > > >2. user prepares BLT using web-console or visor CMD
> tools
> > > and
> > > > > sets
> > > > > > > it
> > > > > > > > to
> > > > > > > > >the cluster. New API setter is needed:
> > > > > > > > >Ignite.activation().setBaselin
> eTopology(BaselineTopology
> > > > blt);
> > > > > > > > >
> > > > > > > > >3. user provides via static configuration a list of
> nodes
> > > that
> > > > > are
> > > > > > > > >expected to be in the cluster.
> > > > > > > > >User starts nodes one by one; when all preconfigured
> nodes
> > > are
> > > > > > > started
> > > > > > > > >cluster is activated and BLT is created.
> > > > > > > > >As list of nodes may be huge it is provided via separate
> > > file
> > > > to
> > > > > > > avoid
> > > > > > > > >flooding main configuration.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Igniters, does this description match with your
> understanding
> > > of
> > > > > > > > > functionality? If it does I'll create a set of tickets and
> > > start
> > > > > > > working
> > > > > > > > on
> > > > > > > > > implementation.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Sergey.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > > > dsetrak...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I still do not see why anyone would explicitly call
> these 2
> > > > > > methods:
> > > > > > > > > >
> > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > BaselineTopology
> > > > > > > > > bltTop);*
> > > > > > > > > >
> > > > > > > > > > For example, if a web console, or some other admin
> process,
> > > > want
> > > > > to
> > > > > > > > > > automatically set currently started nodes as the baseline
> > > > > topology,
> > > > > > > > > > shouldn't they just call a setter for
> > minimalActivationNodes?
> > > > > > > > > >
> > > > > > > > > > D.
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > > > admitr...@gridgain.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > API is proposed in the head of the thread by Sergey,
> as I
> > > > > > > understood:
> > > > > > > > > > > __
> > > > > > > > > > >
> > > > > > > > > > > API for BaselineTopology manipulation may look like
> this:
> > > > > > > > > > >
> > > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > > BaselineTopology
> > > > > > > > > > bltTop);*
> > > > > > > > > > >
> > > > > > > > > > > Both methods will establish BT and activate cluster
> once
> > it
> > > > is
> > > > > > > > > > established.
> > > > > > > > > > >
> > > > > > > > > > > The first one allows user to establish BT using current
> > > > > topology.
> > > > > > > If
> > > > > > > > > any
> > > > > > > > > > > changes happen to the topology during establishing
> > process,
> > > > > user
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > notified and allowed to proceed or abort the procedure.
> > > > > > > > > > >
> > > > > > > > > > > Second method allows to use some
> monitoring'n'management
> > > > tools
> > > > > > like
> > > > > > > > > > > WebConsole where user can prepare a list of nodes,
> using
> > > them
> > > > > > > create
> > > > > > > > a
> > > > > > > > > BT
> > > > > > > > > > > and send to the cluster a command to finally establish
> > it.
> > > > > > > > > > >
> > > > > > > > > > > From high level BaselineTopology entity contains only
> > > > > collection
> > > > > > of
> > > > > > > > > > nodes:
> > > > > > > > > > >
> > > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > > *}*
> > > > > > > > > > >
> > > > > > > > > > > *TopologyNode* here contains information about node -
> its
> > > > > > > consistent
> > > > > > > > id
> > > > > > > > > > and
> > > > > > > > > > > set of user attributes used to calculate affinity
> > function.
> > > > > > > > > > > 
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > View this message in context: http://apache-ignite-
> > > > > > > > > > > developers.2346864.n4.nabble.
> > com/Cluster-auto-activation-
> > > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > > Sent from the Apache Ignite Developers mailing list
> > archive
> > > > at
> > > > > > > > > > Nabble.com.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-09-01 Thread Sergey Chugunov

 > > > > > >
> > > > > > > >1. user starts up new cluster of desired number of nodes
> and
> > > > > > activates
> > > > > > > >it using existing API. BLT is created with all nodes
> > presented
> > > > in
> > > > > > the
> > > > > > > >cluster at the moment of activation, no API is needed;
> > > > > > > >
> > > > > > > >2. user prepares BLT using web-console or visor CMD tools
> > and
> > > > sets
> > > > > > it
> > > > > > > to
> > > > > > > >the cluster. New API setter is needed:
> > > > > > > >Ignite.activation().setBaselineTopology(BaselineTopology
> > > blt);
> > > > > > > >
> > > > > > > >3. user provides via static configuration a list of nodes
> > that
> > > > are
> > > > > > > >expected to be in the cluster.
> > > > > > > >User starts nodes one by one; when all preconfigured nodes
> > are
> > > > > > started
> > > > > > > >cluster is activated and BLT is created.
> > > > > > > >As list of nodes may be huge it is provided via separate
> > file
> > > to
> > > > > > avoid
> > > > > > > >flooding main configuration.
> > > > > > > >
> > > > > > > >
> > > > > > > > Igniters, does this description match with your understanding
> > of
> > > > > > > > functionality? If it does I'll create a set of tickets and
> > start
> > > > > > working
> > > > > > > on
> > > > > > > > implementation.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Sergey.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > > dsetrak...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I still do not see why anyone would explicitly call these 2
> > > > > methods:
> > > > > > > > >
> > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > BaselineTopology
> > > > > > > > bltTop);*
> > > > > > > > >
> > > > > > > > > For example, if a web console, or some other admin process,
> > > want
> > > > to
> > > > > > > > > automatically set currently started nodes as the baseline
> > > > topology,
> > > > > > > > > shouldn't they just call a setter for
> minimalActivationNodes?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > > admitr...@gridgain.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > API is proposed in the head of the thread by Sergey, as I
> > > > > > understood:
> > > > > > > > > > __
> > > > > > > > > >
> > > > > > > > > > API for BaselineTopology manipulation may look like this:
> > > > > > > > > >
> > > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > > BaselineTopology
> > > > > > > > > bltTop);*
> > > > > > > > > >
> > > > > > > > > > Both methods will establish BT and activate cluster once
> it
> > > is
> > > > > > > > > established.
> > > > > > > > > >
> > > > > > > > > > The first one allows user to establish BT using current
> > > > topology.
> > > > > > If
> > > > > > > > any
> > > > > > > > > > changes happen to the topology during establishing
> process,
> > > > user
> > > > > > will
> > > > > > > > be
> > > > > > > > > > notified and allowed to proceed or abort the procedure.
> > > > > > > > > >
> > > > > > > > > > Second method allows to use some monitoring'n'management
> > > tools
> > > > > like
> > > > > > > > > > WebConsole where user can prepare a list of nodes, using
> > them
> > > > > > create
> > > > > > > a
> > > > > > > > BT
> > > > > > > > > > and send to the cluster a command to finally establish
> it.
> > > > > > > > > >
> > > > > > > > > > From high level BaselineTopology entity contains only
> > > > collection
> > > > > of
> > > > > > > > > nodes:
> > > > > > > > > >
> > > > > > > > > > *BaselineTopology {*
> > > > > > > > > > *  Collection nodes;*
> > > > > > > > > > *}*
> > > > > > > > > >
> > > > > > > > > > *TopologyNode* here contains information about node - its
> > > > > > consistent
> > > > > > > id
> > > > > > > > > and
> > > > > > > > > > set of user attributes used to calculate affinity
> function.
> > > > > > > > > > 
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > View this message in context: http://apache-ignite-
> > > > > > > > > > developers.2346864.n4.nabble.
> com/Cluster-auto-activation-
> > > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > > Sent from the Apache Ignite Developers mailing list
> archive
> > > at
> > > > > > > > > Nabble.com.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-30 Thread Dmitriy Setrakyan

gt;flooding main configuration.
> > > > > > >
> > > > > > >
> > > > > > > Igniters, does this description match with your understanding
> of
> > > > > > > functionality? If it does I'll create a set of tickets and
> start
> > > > > working
> > > > > > on
> > > > > > > implementation.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sergey.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I still do not see why anyone would explicitly call these 2
> > > > methods:
> > > > > > > >
> > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > *Ignite::activation::establishBaselineTopology(
> > BaselineTopology
> > > > > > > bltTop);*
> > > > > > > >
> > > > > > > > For example, if a web console, or some other admin process,
> > want
> > > to
> > > > > > > > automatically set currently started nodes as the baseline
> > > topology,
> > > > > > > > shouldn't they just call a setter for minimalActivationNodes?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > > admitr...@gridgain.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > API is proposed in the head of the thread by Sergey, as I
> > > > > understood:
> > > > > > > > > __
> > > > > > > > >
> > > > > > > > > API for BaselineTopology manipulation may look like this:
> > > > > > > > >
> > > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > > *Ignite::activation::establishBaselineTopology(
> > BaselineTopology
> > > > > > > > bltTop);*
> > > > > > > > >
> > > > > > > > > Both methods will establish BT and activate cluster once it
> > is
> > > > > > > > established.
> > > > > > > > >
> > > > > > > > > The first one allows user to establish BT using current
> > > topology.
> > > > > If
> > > > > > > any
> > > > > > > > > changes happen to the topology during establishing process,
> > > user
> > > > > will
> > > > > > > be
> > > > > > > > > notified and allowed to proceed or abort the procedure.
> > > > > > > > >
> > > > > > > > > Second method allows to use some monitoring'n'management
> > tools
> > > > like
> > > > > > > > > WebConsole where user can prepare a list of nodes, using
> them
> > > > > create
> > > > > > a
> > > > > > > BT
> > > > > > > > > and send to the cluster a command to finally establish it.
> > > > > > > > >
> > > > > > > > > From high level BaselineTopology entity contains only
> > > collection
> > > > of
> > > > > > > > nodes:
> > > > > > > > >
> > > > > > > > > *BaselineTopology {*
> > > > > > > > > *  Collection nodes;*
> > > > > > > > > *}*
> > > > > > > > >
> > > > > > > > > *TopologyNode* here contains information about node - its
> > > > > consistent
> > > > > > id
> > > > > > > > and
> > > > > > > > > set of user attributes used to calculate affinity function.
> > > > > > > > > 
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > View this message in context: http://apache-ignite-
> > > > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > > Sent from the Apache Ignite Developers mailing list archive
> > at
> > > > > > > > Nabble.com.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-30 Thread Sergey Chugunov

e should fail nodes with an error message.
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Sergey, I am still confused. What is the BaselineTopology interface
> > in
> > > > your
> > > > > example? I thought that you agreed with me that we simply need a
> > setter
> > > > for
> > > > > activation nodes, no?
> > > > >
> > > > > D.
> > > > >
> > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > > > > sergey.chugu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > As I understand you use the term "minimalActivationNodes" as a
> > > synonym
> > > > > for
> > > > > > BaselineTopology concept.
> > > > > > In that case I agree with you that we can replace both
> "establish*"
> > > > > methods
> > > > > > with a simple setter method (see below in summary).
> > > > > >
> > > > > > Summing up the whole discussion I see the functionality as
> > following:
> > > > > >
> > > > > > New concept BaselineTopology is introduced. The main features it
> > > > enables
> > > > > > are:
> > > > > >
> > > > > >1. automatic activation of cluster;
> > > > > >
> > > > > >2. easy management of cluster topology changes (planned nodes
> > > > > >maintenance, adding new nodes etc);
> > > > > >
> > > > > >3. eliminating of rebalancing traffic on short-term node
> > failures.
> > > > > >
> > > > > >
> > > > > > Use cases to create BLT:
> > > > > >
> > > > > >1. user starts up new cluster of desired number of nodes and
> > > > activates
> > > > > >it using existing API. BLT is created with all nodes presented
> > in
> > > > the
> > > > > >cluster at the moment of activation, no API is needed;
> > > > > >
> > > > > >2. user prepares BLT using web-console or visor CMD tools and
> > sets
> > > > it
> > > > > to
> > > > > >the cluster. New API setter is needed:
> > > > > >Ignite.activation().setBaselineTopology(BaselineTopology
> blt);
> > > > > >
> > > > > >3. user provides via static configuration a list of nodes that
> > are
> > > > > >expected to be in the cluster.
> > > > > >User starts nodes one by one; when all preconfigured nodes are
> > > > started
> > > > > >cluster is activated and BLT is created.
> > > > > >As list of nodes may be huge it is provided via separate file
> to
> > > > avoid
> > > > > >flooding main configuration.
> > > > > >
> > > > > >
> > > > > > Igniters, does this description match with your understanding of
> > > > > > functionality? If it does I'll create a set of tickets and start
> > > > working
> > > > > on
> > > > > > implementation.
> > > > > >
> > > > > > Thanks,
> > > > > > Sergey.
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > I still do not see why anyone would explicitly call these 2
> > > methods:
> > > > > > >
> > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > *Ignite::activation::establishBaselineTopology(
> BaselineTopology
> > > > > > bltTop);*
> > > > > > >
> > > > > > > For example, if a web console, or some other admin process,
> want
> > to
> > > > > > > automatically set currently started nodes as the baseline
> > topology,
> > > > > > > shouldn't they just call a setter for minimalActivationNodes?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > > admitr...@gridgain.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > API is proposed in the head of the thread by Sergey, as I
> > > > understood:
> > > > > > > > __
> > > > > > > >
> > > > > > > > API for BaselineTopology manipulation may look like this:
> > > > > > > >
> > > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > > *Ignite::activation::establishBaselineTopology(
> BaselineTopology
> > > > > > > bltTop);*
> > > > > > > >
> > > > > > > > Both methods will establish BT and activate cluster once it
> is
> > > > > > > established.
> > > > > > > >
> > > > > > > > The first one allows user to establish BT using current
> > topology.
> > > > If
> > > > > > any
> > > > > > > > changes happen to the topology during establishing process,
> > user
> > > > will
> > > > > > be
> > > > > > > > notified and allowed to proceed or abort the procedure.
> > > > > > > >
> > > > > > > > Second method allows to use some monitoring'n'management
> tools
> > > like
> > > > > > > > WebConsole where user can prepare a list of nodes, using them
> > > > create
> > > > > a
> > > > > > BT
> > > > > > > > and send to the cluster a command to finally establish it.
> > > > > > > >
> > > > > > > > From high level BaselineTopology entity contains only
> > collection
> > > of
> > > > > > > nodes:
> > > > > > > >
> > > > > > > > *BaselineTopology {*
> > > > > > > > *  Collection nodes;*
> > > > > > > > *}*
> > > > > > > >
> > > > > > > > *TopologyNode* here contains information about node - its
> > > > consistent
> > > > > id
> > > > > > > and
> > > > > > > > set of user attributes used to calculate affinity function.
> > > > > > > > 
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > View this message in context: http://apache-ignite-
> > > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > > > > > design-proposal-tp20295p21066.html
> > > > > > > > Sent from the Apache Ignite Developers mailing list archive
> at
> > > > > > > Nabble.com.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-30 Thread Nick Pordash

logy changes (planned nodes
> > > > >maintenance, adding new nodes etc);
> > > > >
> > > > >3. eliminating of rebalancing traffic on short-term node
> failures.
> > > > >
> > > > >
> > > > > Use cases to create BLT:
> > > > >
> > > > >1. user starts up new cluster of desired number of nodes and
> > > activates
> > > > >it using existing API. BLT is created with all nodes presented
> in
> > > the
> > > > >cluster at the moment of activation, no API is needed;
> > > > >
> > > > >2. user prepares BLT using web-console or visor CMD tools and
> sets
> > > it
> > > > to
> > > > >the cluster. New API setter is needed:
> > > > >Ignite.activation().setBaselineTopology(BaselineTopology blt);
> > > > >
> > > > >3. user provides via static configuration a list of nodes that
> are
> > > > >expected to be in the cluster.
> > > > >User starts nodes one by one; when all preconfigured nodes are
> > > started
> > > > >cluster is activated and BLT is created.
> > > > >As list of nodes may be huge it is provided via separate file to
> > > avoid
> > > > >flooding main configuration.
> > > > >
> > > > >
> > > > > Igniters, does this description match with your understanding of
> > > > > functionality? If it does I'll create a set of tickets and start
> > > working
> > > > on
> > > > > implementation.
> > > > >
> > > > > Thanks,
> > > > > Sergey.
> > > > >
> > > > >
> > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > I still do not see why anyone would explicitly call these 2
> > methods:
> > > > > >
> > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > > bltTop);*
> > > > > >
> > > > > > For example, if a web console, or some other admin process, want
> to
> > > > > > automatically set currently started nodes as the baseline
> topology,
> > > > > > shouldn't they just call a setter for minimalActivationNodes?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > > admitr...@gridgain.com>
> > > > > > wrote:
> > > > > >
> > > > > > > API is proposed in the head of the thread by Sergey, as I
> > > understood:
> > > > > > > __
> > > > > > >
> > > > > > > API for BaselineTopology manipulation may look like this:
> > > > > > >
> > > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > > > bltTop);*
> > > > > > >
> > > > > > > Both methods will establish BT and activate cluster once it is
> > > > > > established.
> > > > > > >
> > > > > > > The first one allows user to establish BT using current
> topology.
> > > If
> > > > > any
> > > > > > > changes happen to the topology during establishing process,
> user
> > > will
> > > > > be
> > > > > > > notified and allowed to proceed or abort the procedure.
> > > > > > >
> > > > > > > Second method allows to use some monitoring'n'management tools
> > like
> > > > > > > WebConsole where user can prepare a list of nodes, using them
> > > create
> > > > a
> > > > > BT
> > > > > > > and send to the cluster a command to finally establish it.
> > > > > > >
> > > > > > > From high level BaselineTopology entity contains only
> collection
> > of
> > > > > > nodes:
> > > > > > >
> > > > > > > *BaselineTopology {*
> > > > > > > *  Collection nodes;*
> > > > > > > *}*
> > > > > > >
> > > > > > > *TopologyNode* here contains information about node - its
> > > consistent
> > > > id
> > > > > > and
> > > > > > > set of user attributes used to calculate affinity function.
> > > > > > > 
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > View this message in context: http://apache-ignite-
> > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > > > > design-proposal-tp20295p21066.html
> > > > > > > Sent from the Apache Ignite Developers mailing list archive at
> > > > > > Nabble.com.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-30 Thread Sergey Chugunov

ist of nodes may be huge it is provided via separate file to
> > avoid
> > > >flooding main configuration.
> > > >
> > > >
> > > > Igniters, does this description match with your understanding of
> > > > functionality? If it does I'll create a set of tickets and start
> > working
> > > on
> > > > implementation.
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > >
> > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > I still do not see why anyone would explicitly call these 2
> methods:
> > > > >
> > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > bltTop);*
> > > > >
> > > > > For example, if a web console, or some other admin process, want to
> > > > > automatically set currently started nodes as the baseline topology,
> > > > > shouldn't they just call a setter for minimalActivationNodes?
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > > admitr...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > API is proposed in the head of the thread by Sergey, as I
> > understood:
> > > > > > __
> > > > > >
> > > > > > API for BaselineTopology manipulation may look like this:
> > > > > >
> > > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > > bltTop);*
> > > > > >
> > > > > > Both methods will establish BT and activate cluster once it is
> > > > > established.
> > > > > >
> > > > > > The first one allows user to establish BT using current topology.
> > If
> > > > any
> > > > > > changes happen to the topology during establishing process, user
> > will
> > > > be
> > > > > > notified and allowed to proceed or abort the procedure.
> > > > > >
> > > > > > Second method allows to use some monitoring'n'management tools
> like
> > > > > > WebConsole where user can prepare a list of nodes, using them
> > create
> > > a
> > > > BT
> > > > > > and send to the cluster a command to finally establish it.
> > > > > >
> > > > > > From high level BaselineTopology entity contains only collection
> of
> > > > > nodes:
> > > > > >
> > > > > > *BaselineTopology {*
> > > > > > *  Collection nodes;*
> > > > > > *}*
> > > > > >
> > > > > > *TopologyNode* here contains information about node - its
> > consistent
> > > id
> > > > > and
> > > > > > set of user attributes used to calculate affinity function.
> > > > > > 
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > View this message in context: http://apache-ignite-
> > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > > > design-proposal-tp20295p21066.html
> > > > > > Sent from the Apache Ignite Developers mailing list archive at
> > > > > Nabble.com.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-28 Thread Dmitriy Setrakyan

shouldn't they just call a setter for minimalActivationNodes?
> > > >
> > > > D.
> > > >
> > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > admitr...@gridgain.com>
> > > > wrote:
> > > >
> > > > > API is proposed in the head of the thread by Sergey, as I
> understood:
> > > > > __
> > > > >
> > > > > API for BaselineTopology manipulation may look like this:
> > > > >
> > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > bltTop);*
> > > > >
> > > > > Both methods will establish BT and activate cluster once it is
> > > > established.
> > > > >
> > > > > The first one allows user to establish BT using current topology.
> If
> > > any
> > > > > changes happen to the topology during establishing process, user
> will
> > > be
> > > > > notified and allowed to proceed or abort the procedure.
> > > > >
> > > > > Second method allows to use some monitoring'n'management tools like
> > > > > WebConsole where user can prepare a list of nodes, using them
> create
> > a
> > > BT
> > > > > and send to the cluster a command to finally establish it.
> > > > >
> > > > > From high level BaselineTopology entity contains only collection of
> > > > nodes:
> > > > >
> > > > > *BaselineTopology {*
> > > > > *  Collection nodes;*
> > > > > *}*
> > > > >
> > > > > *TopologyNode* here contains information about node - its
> consistent
> > id
> > > > and
> > > > > set of user attributes used to calculate affinity function.
> > > > > 
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context: http://apache-ignite-
> > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > > design-proposal-tp20295p21066.html
> > > > > Sent from the Apache Ignite Developers mailing list archive at
> > > > Nabble.com.
> > > > >
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-28 Thread Sergey Chugunov

; > established.
> > > >
> > > > The first one allows user to establish BT using current topology. If
> > any
> > > > changes happen to the topology during establishing process, user will
> > be
> > > > notified and allowed to proceed or abort the procedure.
> > > >
> > > > Second method allows to use some monitoring'n'management tools like
> > > > WebConsole where user can prepare a list of nodes, using them create
> a
> > BT
> > > > and send to the cluster a command to finally establish it.
> > > >
> > > > From high level BaselineTopology entity contains only collection of
> > > nodes:
> > > >
> > > > *BaselineTopology {*
> > > > *  Collection nodes;*
> > > > *}*
> > > >
> > > > *TopologyNode* here contains information about node - its consistent
> id
> > > and
> > > > set of user attributes used to calculate affinity function.
> > > > 
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context: http://apache-ignite-
> > > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > > design-proposal-tp20295p21066.html
> > > > Sent from the Apache Ignite Developers mailing list archive at
> > > Nabble.com.
> > > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-22 Thread Dmitriy Setrakyan

Sergey, I am still confused. What is the BaselineTopology interface in your
example? I thought that you agreed with me that we simply need a setter for
activation nodes, no?

D.

On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <sergey.chugu...@gmail.com>
wrote:

> Dmitriy,
>
> As I understand you use the term "minimalActivationNodes" as a synonym for
> BaselineTopology concept.
> In that case I agree with you that we can replace both "establish*" methods
> with a simple setter method (see below in summary).
>
> Summing up the whole discussion I see the functionality as following:
>
> New concept BaselineTopology is introduced. The main features it enables
> are:
>
>1. automatic activation of cluster;
>
>2. easy management of cluster topology changes (planned nodes
>maintenance, adding new nodes etc);
>
>3. eliminating of rebalancing traffic on short-term node failures.
>
>
> Use cases to create BLT:
>
>1. user starts up new cluster of desired number of nodes and activates
>it using existing API. BLT is created with all nodes presented in the
>cluster at the moment of activation, no API is needed;
>
>2. user prepares BLT using web-console or visor CMD tools and sets it to
>the cluster. New API setter is needed:
>Ignite.activation().setBaselineTopology(BaselineTopology blt);
>
>3. user provides via static configuration a list of nodes that are
>expected to be in the cluster.
>User starts nodes one by one; when all preconfigured nodes are started
>cluster is activated and BLT is created.
>As list of nodes may be huge it is provided via separate file to avoid
>flooding main configuration.
>
>
> Igniters, does this description match with your understanding of
> functionality? If it does I'll create a set of tickets and start working on
> implementation.
>
> Thanks,
> Sergey.
>
>
> On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <dsetrak...@apache.org>
> wrote:
>
> > I still do not see why anyone would explicitly call these 2 methods:
> >
> > *Ignite::activation::establishBaselineTopology();*
> > *Ignite::activation::establishBaselineTopology(BaselineTopology
> bltTop);*
> >
> > For example, if a web console, or some other admin process, want to
> > automatically set currently started nodes as the baseline topology,
> > shouldn't they just call a setter for minimalActivationNodes?
> >
> > D.
> >
> > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> admitr...@gridgain.com>
> > wrote:
> >
> > > API is proposed in the head of the thread by Sergey, as I understood:
> > > __
> > >
> > > API for BaselineTopology manipulation may look like this:
> > >
> > > *Ignite::activation::establishBaselineTopology();*
> > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > bltTop);*
> > >
> > > Both methods will establish BT and activate cluster once it is
> > established.
> > >
> > > The first one allows user to establish BT using current topology. If
> any
> > > changes happen to the topology during establishing process, user will
> be
> > > notified and allowed to proceed or abort the procedure.
> > >
> > > Second method allows to use some monitoring'n'management tools like
> > > WebConsole where user can prepare a list of nodes, using them create a
> BT
> > > and send to the cluster a command to finally establish it.
> > >
> > > From high level BaselineTopology entity contains only collection of
> > nodes:
> > >
> > > *BaselineTopology {*
> > > *  Collection nodes;*
> > > *}*
> > >
> > > *TopologyNode* here contains information about node - its consistent id
> > and
> > > set of user attributes used to calculate affinity function.
> > > 
> > >
> > >
> > >
> > > --
> > > View this message in context: http://apache-ignite-
> > > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > > design-proposal-tp20295p21066.html
> > > Sent from the Apache Ignite Developers mailing list archive at
> > Nabble.com.
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-22 Thread Sergey Chugunov

Dmitriy,

As I understand you use the term "minimalActivationNodes" as a synonym for
BaselineTopology concept.
In that case I agree with you that we can replace both "establish*" methods
with a simple setter method (see below in summary).

Summing up the whole discussion I see the functionality as following:

New concept BaselineTopology is introduced. The main features it enables
are:

   1. automatic activation of cluster;

   2. easy management of cluster topology changes (planned nodes
   maintenance, adding new nodes etc);

   3. eliminating of rebalancing traffic on short-term node failures.

Use cases to create BLT:

   1. user starts up new cluster of desired number of nodes and activates
   it using existing API. BLT is created with all nodes presented in the
   cluster at the moment of activation, no API is needed;

   2. user prepares BLT using web-console or visor CMD tools and sets it to
   the cluster. New API setter is needed:
   Ignite.activation().setBaselineTopology(BaselineTopology blt);

   3. user provides via static configuration a list of nodes that are
   expected to be in the cluster.
   User starts nodes one by one; when all preconfigured nodes are started
   cluster is activated and BLT is created.
   As list of nodes may be huge it is provided via separate file to avoid
   flooding main configuration.

Igniters, does this description match with your understanding of
functionality? If it does I'll create a set of tickets and start working on
implementation.

Thanks,
Sergey.

On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <dsetrak...@apache.org>
wrote:

> I still do not see why anyone would explicitly call these 2 methods:
>
> *Ignite::activation::establishBaselineTopology();*
> *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);*
>
> For example, if a web console, or some other admin process, want to
> automatically set currently started nodes as the baseline topology,
> shouldn't they just call a setter for minimalActivationNodes?
>
> D.
>
> On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <admitr...@gridgain.com>
> wrote:
>
> > API is proposed in the head of the thread by Sergey, as I understood:
> > __
> >
> > API for BaselineTopology manipulation may look like this:
> >
> > *Ignite::activation::establishBaselineTopology();*
> > *Ignite::activation::establishBaselineTopology(BaselineTopology
> bltTop);*
> >
> > Both methods will establish BT and activate cluster once it is
> established.
> >
> > The first one allows user to establish BT using current topology. If any
> > changes happen to the topology during establishing process, user will be
> > notified and allowed to proceed or abort the procedure.
> >
> > Second method allows to use some monitoring'n'management tools like
> > WebConsole where user can prepare a list of nodes, using them create a BT
> > and send to the cluster a command to finally establish it.
> >
> > From high level BaselineTopology entity contains only collection of
> nodes:
> >
> > *BaselineTopology {*
> > *  Collection nodes;*
> > *}*
> >
> > *TopologyNode* here contains information about node - its consistent id
> and
> > set of user attributes used to calculate affinity function.
> > 
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-
> > developers.2346864.n4.nabble.com/Cluster-auto-activation-
> > design-proposal-tp20295p21066.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
> >
>

Re: Cluster auto activation design proposal

2017-08-19 Thread Dmitriy Setrakyan

I still do not see why anyone would explicitly call these 2 methods:

*Ignite::activation::establishBaselineTopology();*
*Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);*

For example, if a web console, or some other admin process, want to
automatically set currently started nodes as the baseline topology,
shouldn't they just call a setter for minimalActivationNodes?

D.

On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <admitr...@gridgain.com>
wrote:

> API is proposed in the head of the thread by Sergey, as I understood:
> __
>
> API for BaselineTopology manipulation may look like this:
>
> *Ignite::activation::establishBaselineTopology();*
> *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);*
>
> Both methods will establish BT and activate cluster once it is established.
>
> The first one allows user to establish BT using current topology. If any
> changes happen to the topology during establishing process, user will be
> notified and allowed to proceed or abort the procedure.
>
> Second method allows to use some monitoring'n'management tools like
> WebConsole where user can prepare a list of nodes, using them create a BT
> and send to the cluster a command to finally establish it.
>
> From high level BaselineTopology entity contains only collection of nodes:
>
> *BaselineTopology {*
> *  Collection nodes;*
> *}*
>
> *TopologyNode* here contains information about node - its consistent id and
> set of user attributes used to calculate affinity function.
> 
>
>
>
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/Cluster-auto-activation-
> design-proposal-tp20295p21066.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

Re: Cluster auto activation design proposal

2017-08-18 Thread Alexey Dmitriev

API is proposed in the head of the thread by Sergey, as I understood:
__

API for BaselineTopology manipulation may look like this: 

*Ignite::activation::establishBaselineTopology();* 
*Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* 

Both methods will establish BT and activate cluster once it is established. 

The first one allows user to establish BT using current topology. If any 
changes happen to the topology during establishing process, user will be 
notified and allowed to proceed or abort the procedure. 

Second method allows to use some monitoring'n'management tools like 
WebConsole where user can prepare a list of nodes, using them create a BT 
and send to the cluster a command to finally establish it. 

>From high level BaselineTopology entity contains only collection of nodes: 

*BaselineTopology {* 
*  Collection nodes;* 
*}* 

*TopologyNode* here contains information about node - its consistent id and 
set of user attributes used to calculate affinity function. 




--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Cluster-auto-activation-design-proposal-tp20295p21066.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Cluster auto activation design proposal

2017-08-17 Thread Dmitriy Setrakyan

On Thu, Aug 17, 2017 at 7:34 AM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> As automatic creation of Baseline Topology is just another use case about
> the whole concept, lets discuss it in this thread and forget about parallel
> one.
>
> *initialActivationNodes* for desired configuration of Baseline Topology
> sounds good to me, I would like to hear from other Igniters.
>

I don't think this sounds right. It should be either "inititialActiveNodes"
or "initialNodes". I am also open to other suggestions.


>
> However other options to create Baseline Topology were proposed:
>
>- Prepare it manually with WebConsole or visor-console (no
>initialActivationNodes configuration is involved).
>- Create it automatically on the first manual activation (again no
>initial configuration is involved).
>

> Thus we still need Baseline Topology entity not only in documentation.
>

In either case, wouldn't it involve creation of initialActiveNodes
implicitly?


>
> Also please take into account that at some point in time user may want to
> recreate BLT (e.g. to add nodes to existing cluster). User will have to
> interact with some entity; so we need it not only internally but on public
> API.
>

What would the public API look like and who would user interact with it?

Re: Cluster auto activation design proposal

2017-08-17 Thread Sergey Chugunov

Dmitriy,

As automatic creation of Baseline Topology is just another use case about
the whole concept, lets discuss it in this thread and forget about parallel
one.

*initialActivationNodes* for desired configuration of Baseline Topology
sounds good to me, I would like to hear from other Igniters.

However other options to create Baseline Topology were proposed:

   - Prepare it manually with WebConsole or visor-console (no
   initialActivationNodes configuration is involved).
   - Create it automatically on the first manual activation (again no
   initial configuration is involved).

Thus we still need Baseline Topology entity not only in documentation.

Also please take into account that at some point in time user may want to
recreate BLT (e.g. to add nodes to existing cluster). User will have to
interact with some entity; so we need it not only internally but on public
API.

Thanks,
Sergey.



On Sat, Aug 12, 2017 at 4:07 AM, Dmitriy Setrakyan 
wrote:

> Sergey,
>
> As it is becoming clear from another thread, the only configuration users
> will need to provide is the initial set of nodes. Everything else will be
> handled by Ignite automatically.
>
> In this case, the name Baseline Topology will appear only in documentation,
> in which case the name is OK (that is if I am understanding the design
> correctly).
>
> However, the list initial set of nodes can be called
> initialActivationNodes.
>
> Makes sense?
>
> D.
>
> On Fri, Aug 11, 2017 at 2:46 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > More options to name the concept (one may put "node set" or "topology" at
> > the end - for me these are interchangeable):
> >
> >
> >1. essential
> >2. basic
> >3. completed
> >4. prepared
> >5. solid
> >6. enduring
> >
> > To me the name must stress the fact that "node set" or "topology"
> contains
> > all nodes user is going to work with for a long run.
> > It is what user plans to work with and base the work on.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> > On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Going down "node set" road:
> > >
> > > -fixed node set
> > > -established node set
> > > -base node set
> > >
> > > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > >> Can we brainstorm on the names again, I am not sure we have a
> consensus
> > on
> > >> the name "baseline topology". This will be included in Ignite
> > >> configuration, so the name has to be clear.
> > >>
> > >> Some of the proposals were:
> > >>
> > >> - baseline topology
> > >> - minimal node set
> > >> - node restart set
> > >> - minimal topology
> > >>
> > >> Any other suggestions?
> > >>
> > >> D.
> > >>
> > >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
> > >> alexey.goncha...@gmail.com> wrote:
> > >>
> > >> > Denis,
> > >> >
> > >> > This should be handled by the BT triggers. If I have 3 backups
> > >> configured,
> > >> > I actually won't care if my cluster will live 6 hours without an
> > >> additional
> > >> > backup. If for a partition there is only one backup left - a new BT
> > >> should
> > >> > be triggered automatically.
> > >> >
> > >> > 2017-08-10 0:33 GMT+03:00 Denis Magda :
> > >> >
> > >> > > Sergey,
> > >> > >
> > >> > > That’s the only concern I have:
> > >> > >
> > >> > > * 5. User takes out nodes from cluster (e.g. for maintenance
> > >> purposes):
> > >> > no
> > >> > >   rebalance happens until user recreates BLT on new cluster
> > topology.*
> > >> > >
> > >> > > What if a node is crashed (or some other kind of outage) in the
> > >> middle of
> > >> > > the night and the user has to be sure that survived nodes will
> > >> rearrange
> > >> > > and rebalancing partitions?
> > >> > >
> > >> > > —
> > >> > > Denis
> > >> > >
> > >> > >
> > >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov <
> > >> sergey.chugu...@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > Folks,
> > >> > > >
> > >> > > > I've summarized all results from our discussion so far on wiki
> > page:
> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > > Automatic+activation+design+-+draft
> > >> > > >
> > >> > > > I hope I reflected the most important details and going to add
> API
> > >> > > > suggestions for all use cases soon.
> > >> > > >
> > >> > > > Feel free to give feedback here or in comments under the page.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Sergey.
> > >> > > >
> > >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
> > >> > akuznet...@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Hi,
> > >> > > >>
> > >> > >  1. User creates new BLT using WebConsole or other tool and
> > >> "applies"
> > >> > > it
> > >> > > >> to brand-new cluster.
> > >> > > >>
> > >> > > >> Good idea, but we also should implement *command-line utility*
> > for
> > >> the
> >

Re: Cluster auto activation design proposal

2017-08-11 Thread Dmitriy Setrakyan

Sergey,

As it is becoming clear from another thread, the only configuration users
will need to provide is the initial set of nodes. Everything else will be
handled by Ignite automatically.

In this case, the name Baseline Topology will appear only in documentation,
in which case the name is OK (that is if I am understanding the design
correctly).

However, the list initial set of nodes can be called initialActivationNodes.

Makes sense?

D.

On Fri, Aug 11, 2017 at 2:46 AM, Sergey Chugunov 
wrote:

> More options to name the concept (one may put "node set" or "topology" at
> the end - for me these are interchangeable):
>
>
>1. essential
>2. basic
>3. completed
>4. prepared
>5. solid
>6. enduring
>
> To me the name must stress the fact that "node set" or "topology" contains
> all nodes user is going to work with for a long run.
> It is what user plans to work with and base the work on.
>
> Any thoughts on this?
>
> Thanks,
> Sergey.
>
> On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Going down "node set" road:
> >
> > -fixed node set
> > -established node set
> > -base node set
> >
> > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> >> Can we brainstorm on the names again, I am not sure we have a consensus
> on
> >> the name "baseline topology". This will be included in Ignite
> >> configuration, so the name has to be clear.
> >>
> >> Some of the proposals were:
> >>
> >> - baseline topology
> >> - minimal node set
> >> - node restart set
> >> - minimal topology
> >>
> >> Any other suggestions?
> >>
> >> D.
> >>
> >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
> >> alexey.goncha...@gmail.com> wrote:
> >>
> >> > Denis,
> >> >
> >> > This should be handled by the BT triggers. If I have 3 backups
> >> configured,
> >> > I actually won't care if my cluster will live 6 hours without an
> >> additional
> >> > backup. If for a partition there is only one backup left - a new BT
> >> should
> >> > be triggered automatically.
> >> >
> >> > 2017-08-10 0:33 GMT+03:00 Denis Magda :
> >> >
> >> > > Sergey,
> >> > >
> >> > > That’s the only concern I have:
> >> > >
> >> > > * 5. User takes out nodes from cluster (e.g. for maintenance
> >> purposes):
> >> > no
> >> > >   rebalance happens until user recreates BLT on new cluster
> topology.*
> >> > >
> >> > > What if a node is crashed (or some other kind of outage) in the
> >> middle of
> >> > > the night and the user has to be sure that survived nodes will
> >> rearrange
> >> > > and rebalancing partitions?
> >> > >
> >> > > —
> >> > > Denis
> >> > >
> >> > >
> >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov <
> >> sergey.chugu...@gmail.com
> >> > >
> >> > > wrote:
> >> > > >
> >> > > > Folks,
> >> > > >
> >> > > > I've summarized all results from our discussion so far on wiki
> page:
> >> > > > https://cwiki.apache.org/confluence/display/IGNITE/
> >> > > Automatic+activation+design+-+draft
> >> > > >
> >> > > > I hope I reflected the most important details and going to add API
> >> > > > suggestions for all use cases soon.
> >> > > >
> >> > > > Feel free to give feedback here or in comments under the page.
> >> > > >
> >> > > > Thanks,
> >> > > > Sergey.
> >> > > >
> >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
> >> > akuznet...@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > >> Hi,
> >> > > >>
> >> > >  1. User creates new BLT using WebConsole or other tool and
> >> "applies"
> >> > > it
> >> > > >> to brand-new cluster.
> >> > > >>
> >> > > >> Good idea, but we also should implement *command-line utility*
> for
> >> the
> >> > > same
> >> > > >> use case.
> >> > > >>
> >> > > >> --
> >> > > >> Alexey Kuznetsov
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Cluster auto activation design proposal

2017-08-11 Thread Sergey Chugunov

More options to name the concept (one may put "node set" or "topology" at
the end - for me these are interchangeable):


   1. essential
   2. basic
   3. completed
   4. prepared
   5. solid
   6. enduring

To me the name must stress the fact that "node set" or "topology" contains
all nodes user is going to work with for a long run.
It is what user plans to work with and base the work on.

Any thoughts on this?

Thanks,
Sergey.

On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov 
wrote:

> Going down "node set" road:
>
> -fixed node set
> -established node set
> -base node set
>
> On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan 
> wrote:
>
>> Can we brainstorm on the names again, I am not sure we have a consensus on
>> the name "baseline topology". This will be included in Ignite
>> configuration, so the name has to be clear.
>>
>> Some of the proposals were:
>>
>> - baseline topology
>> - minimal node set
>> - node restart set
>> - minimal topology
>>
>> Any other suggestions?
>>
>> D.
>>
>> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
>> alexey.goncha...@gmail.com> wrote:
>>
>> > Denis,
>> >
>> > This should be handled by the BT triggers. If I have 3 backups
>> configured,
>> > I actually won't care if my cluster will live 6 hours without an
>> additional
>> > backup. If for a partition there is only one backup left - a new BT
>> should
>> > be triggered automatically.
>> >
>> > 2017-08-10 0:33 GMT+03:00 Denis Magda :
>> >
>> > > Sergey,
>> > >
>> > > That’s the only concern I have:
>> > >
>> > > * 5. User takes out nodes from cluster (e.g. for maintenance
>> purposes):
>> > no
>> > >   rebalance happens until user recreates BLT on new cluster topology.*
>> > >
>> > > What if a node is crashed (or some other kind of outage) in the
>> middle of
>> > > the night and the user has to be sure that survived nodes will
>> rearrange
>> > > and rebalancing partitions?
>> > >
>> > > —
>> > > Denis
>> > >
>> > >
>> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com
>> > >
>> > > wrote:
>> > > >
>> > > > Folks,
>> > > >
>> > > > I've summarized all results from our discussion so far on wiki page:
>> > > > https://cwiki.apache.org/confluence/display/IGNITE/
>> > > Automatic+activation+design+-+draft
>> > > >
>> > > > I hope I reflected the most important details and going to add API
>> > > > suggestions for all use cases soon.
>> > > >
>> > > > Feel free to give feedback here or in comments under the page.
>> > > >
>> > > > Thanks,
>> > > > Sergey.
>> > > >
>> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
>> > akuznet...@apache.org>
>> > > > wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > >  1. User creates new BLT using WebConsole or other tool and
>> "applies"
>> > > it
>> > > >> to brand-new cluster.
>> > > >>
>> > > >> Good idea, but we also should implement *command-line utility* for
>> the
>> > > same
>> > > >> use case.
>> > > >>
>> > > >> --
>> > > >> Alexey Kuznetsov
>> > > >>
>> > >
>> > >
>> >
>>
>
>

Re: Cluster auto activation design proposal

2017-08-10 Thread Sergey Chugunov

Going down "node set" road:

-fixed node set
-established node set
-base node set

On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan 
wrote:

> Can we brainstorm on the names again, I am not sure we have a consensus on
> the name "baseline topology". This will be included in Ignite
> configuration, so the name has to be clear.
>
> Some of the proposals were:
>
> - baseline topology
> - minimal node set
> - node restart set
> - minimal topology
>
> Any other suggestions?
>
> D.
>
> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Denis,
> >
> > This should be handled by the BT triggers. If I have 3 backups
> configured,
> > I actually won't care if my cluster will live 6 hours without an
> additional
> > backup. If for a partition there is only one backup left - a new BT
> should
> > be triggered automatically.
> >
> > 2017-08-10 0:33 GMT+03:00 Denis Magda :
> >
> > > Sergey,
> > >
> > > That’s the only concern I have:
> > >
> > > * 5. User takes out nodes from cluster (e.g. for maintenance purposes):
> > no
> > >   rebalance happens until user recreates BLT on new cluster topology.*
> > >
> > > What if a node is crashed (or some other kind of outage) in the middle
> of
> > > the night and the user has to be sure that survived nodes will
> rearrange
> > > and rebalancing partitions?
> > >
> > > —
> > > Denis
> > >
> > >
> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com
> > >
> > > wrote:
> > > >
> > > > Folks,
> > > >
> > > > I've summarized all results from our discussion so far on wiki page:
> > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Automatic+activation+design+-+draft
> > > >
> > > > I hope I reflected the most important details and going to add API
> > > > suggestions for all use cases soon.
> > > >
> > > > Feel free to give feedback here or in comments under the page.
> > > >
> > > > Thanks,
> > > > Sergey.
> > > >
> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
> > akuznet...@apache.org>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > >  1. User creates new BLT using WebConsole or other tool and
> "applies"
> > > it
> > > >> to brand-new cluster.
> > > >>
> > > >> Good idea, but we also should implement *command-line utility* for
> the
> > > same
> > > >> use case.
> > > >>
> > > >> --
> > > >> Alexey Kuznetsov
> > > >>
> > >
> > >
> >
>

Re: Cluster auto activation design proposal

2017-08-10 Thread Dmitriy Setrakyan

Can we brainstorm on the names again, I am not sure we have a consensus on
the name "baseline topology". This will be included in Ignite
configuration, so the name has to be clear.

Some of the proposals were:

- baseline topology
- minimal node set
- node restart set
- minimal topology

Any other suggestions?

D.

On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Denis,
>
> This should be handled by the BT triggers. If I have 3 backups configured,
> I actually won't care if my cluster will live 6 hours without an additional
> backup. If for a partition there is only one backup left - a new BT should
> be triggered automatically.
>
> 2017-08-10 0:33 GMT+03:00 Denis Magda :
>
> > Sergey,
> >
> > That’s the only concern I have:
> >
> > * 5. User takes out nodes from cluster (e.g. for maintenance purposes):
> no
> >   rebalance happens until user recreates BLT on new cluster topology.*
> >
> > What if a node is crashed (or some other kind of outage) in the middle of
> > the night and the user has to be sure that survived nodes will rearrange
> > and rebalancing partitions?
> >
> > —
> > Denis
> >
> >
> > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov  >
> > wrote:
> > >
> > > Folks,
> > >
> > > I've summarized all results from our discussion so far on wiki page:
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > Automatic+activation+design+-+draft
> > >
> > > I hope I reflected the most important details and going to add API
> > > suggestions for all use cases soon.
> > >
> > > Feel free to give feedback here or in comments under the page.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
> akuznet...@apache.org>
> > > wrote:
> > >
> > >> Hi,
> > >>
> >  1. User creates new BLT using WebConsole or other tool and "applies"
> > it
> > >> to brand-new cluster.
> > >>
> > >> Good idea, but we also should implement *command-line utility* for the
> > same
> > >> use case.
> > >>
> > >> --
> > >> Alexey Kuznetsov
> > >>
> >
> >
>

Re: Cluster auto activation design proposal

2017-08-10 Thread Alexey Goncharuk

Denis,

This should be handled by the BT triggers. If I have 3 backups configured,
I actually won't care if my cluster will live 6 hours without an additional
backup. If for a partition there is only one backup left - a new BT should
be triggered automatically.

2017-08-10 0:33 GMT+03:00 Denis Magda :

> Sergey,
>
> That’s the only concern I have:
>
> * 5. User takes out nodes from cluster (e.g. for maintenance purposes): no
>   rebalance happens until user recreates BLT on new cluster topology.*
>
> What if a node is crashed (or some other kind of outage) in the middle of
> the night and the user has to be sure that survived nodes will rearrange
> and rebalancing partitions?
>
> —
> Denis
>
>
> > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov 
> wrote:
> >
> > Folks,
> >
> > I've summarized all results from our discussion so far on wiki page:
> > https://cwiki.apache.org/confluence/display/IGNITE/
> Automatic+activation+design+-+draft
> >
> > I hope I reflected the most important details and going to add API
> > suggestions for all use cases soon.
> >
> > Feel free to give feedback here or in comments under the page.
> >
> > Thanks,
> > Sergey.
> >
> > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov 
> > wrote:
> >
> >> Hi,
> >>
>  1. User creates new BLT using WebConsole or other tool and "applies"
> it
> >> to brand-new cluster.
> >>
> >> Good idea, but we also should implement *command-line utility* for the
> same
> >> use case.
> >>
> >> --
> >> Alexey Kuznetsov
> >>
>
>

Re: Cluster auto activation design proposal

2017-08-09 Thread Denis Magda

Sergey,

That’s the only concern I have:

* 5. User takes out nodes from cluster (e.g. for maintenance purposes): no
  rebalance happens until user recreates BLT on new cluster topology.*

What if a node is crashed (or some other kind of outage) in the middle of the 
night and the user has to be sure that survived nodes will rearrange and 
rebalancing partitions? 

—
Denis

> On Aug 4, 2017, at 9:21 AM, Sergey Chugunov  wrote:
> 
> Folks,
> 
> I've summarized all results from our discussion so far on wiki page:
> https://cwiki.apache.org/confluence/display/IGNITE/Automatic+activation+design+-+draft
> 
> I hope I reflected the most important details and going to add API
> suggestions for all use cases soon.
> 
> Feel free to give feedback here or in comments under the page.
> 
> Thanks,
> Sergey.
> 
> On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov 
> wrote:
> 
>> Hi,
>> 
 1. User creates new BLT using WebConsole or other tool and "applies" it
>> to brand-new cluster.
>> 
>> Good idea, but we also should implement *command-line utility* for the same
>> use case.
>> 
>> --
>> Alexey Kuznetsov
>>

Re: Cluster auto activation design proposal

2017-08-04 Thread Sergey Chugunov

Folks,

I've summarized all results from our discussion so far on wiki page:
https://cwiki.apache.org/confluence/display/IGNITE/Automatic+activation+design+-+draft

I hope I reflected the most important details and going to add API
suggestions for all use cases soon.

Feel free to give feedback here or in comments under the page.

Thanks,
Sergey.

On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov 
wrote:

> Hi,
>
> >>1. User creates new BLT using WebConsole or other tool and "applies" it
>  to brand-new cluster.
>
> Good idea, but we also should implement *command-line utility* for the same
> use case.
>
> --
> Alexey Kuznetsov
>

Re: Cluster auto activation design proposal

2017-08-03 Thread Sergey Chugunov

I also would like to provide more use cases of how BLT is supposed to work
(let me call it this way until we come up with a better one):

   1. User creates new BLT using WebConsole or other tool and "applies" it
   to brand-new cluster.

   2. User starts up brand-new cluster with desired amount of nodes and
   activates it. At the moment of activation BLT is created with all server
   non-daemon nodes presented in the cluster.

   3. User starts up a cluster with previously prepared BLT -> when set of
   nodes in the cluster matches with BLT cluster gets automatically activated.

   4. User has an up-and-running active cluster and starts a few more
   nodes. They join the cluster but no partitions are assigned to them.
   User recreates BLT on new cluster topology -> partitions are assigned to
   new nodes.

   5. User takes out nodes from cluster (e.g. for maintenance purposes): no
   rebalance happens until user recreates BLT on new cluster topology.

   6. If some parameters reach critical levels (e.g. number of backups for
   a partition is too low) coordinator automatically recreates BLT and thus
   triggers rebalancing.

I hope these use cases will help to clarify purposes of the proposed
feature.

On Thu, Aug 3, 2017 at 4:08 PM, Alexey Goncharuk  wrote:

> My understanding of Baseline Topology is the set of nodes which are
> *expected* to be in the cluster.
> Let me go a little bit further because BT (or whatever name we choose) may
> and will solve more issues than just auto-activation:
>
> 1) More graceful control over rebalancing than just rebalance delay. If a
> server is shut down for maintenance and there are enough backup nodes in
> the cluster, there is no need to rebalance.
> 2) Guarantee that there will be no conflicting key-value mappings due to
> incorrect cluster activation. For example, consider a scenario when there
> was a cluster of 10 nodes, then the cluster was shut down, started first 5
> nodes, activated, made some updates, shut down 5 nodes, start up other 5
> nodes, activate, make some updates, start up first 5 nodes. Currently,
> there is no way to determine that there was an incompatible topology change
> which leads to data inconsistency.
> 3) When a cluster is shutting down node-by-node, we must track a node which
> has 'seen' a partition last time and not activate the cluster until all
> nodes are present. Otherwise, again, we may activate too early and see
> outdated values.
>
> I do not want to add any 'faster' hacks here because they will only make
> the issue above appear more likely. Besides, BT should be available in 2.2
> anyway, so no need to rush with hacks.
>
> --AG
>
> 2017-08-03 15:09 GMT+03:00 Yakov Zhdanov :
>
> > >Obvious connotation of "minimal set" is a set that cannot be decreased.
> >
> > >But lets consider the following case: user has a cluster of 50 nodes and
> > >decides to switch off 3 nodes for maintenance for a while. Ok, user just
> > >does it and then recreates this "minimal node set" to only 47 nodes.
> >
> > >So initial minimal node set was decreased - something counter-intuitive
> to
> > >me and may cause confusion as well.
> >
> > That was my point. If I have 50 nodes and 3 backups I can restart on 48,
> 49
> > and 50 without data loss. In case of 48 and 49 after cluster gets
> activated
> > missing backups are assigned and rebalancing starts.
> >
> > --Yakov
> >
>

Re: Cluster auto activation design proposal

2017-08-03 Thread Alexey Goncharuk

My understanding of Baseline Topology is the set of nodes which are
*expected* to be in the cluster.
Let me go a little bit further because BT (or whatever name we choose) may
and will solve more issues than just auto-activation:

1) More graceful control over rebalancing than just rebalance delay. If a
server is shut down for maintenance and there are enough backup nodes in
the cluster, there is no need to rebalance.
2) Guarantee that there will be no conflicting key-value mappings due to
incorrect cluster activation. For example, consider a scenario when there
was a cluster of 10 nodes, then the cluster was shut down, started first 5
nodes, activated, made some updates, shut down 5 nodes, start up other 5
nodes, activate, make some updates, start up first 5 nodes. Currently,
there is no way to determine that there was an incompatible topology change
which leads to data inconsistency.
3) When a cluster is shutting down node-by-node, we must track a node which
has 'seen' a partition last time and not activate the cluster until all
nodes are present. Otherwise, again, we may activate too early and see
outdated values.

I do not want to add any 'faster' hacks here because they will only make
the issue above appear more likely. Besides, BT should be available in 2.2
anyway, so no need to rush with hacks.

--AG

2017-08-03 15:09 GMT+03:00 Yakov Zhdanov :

> >Obvious connotation of "minimal set" is a set that cannot be decreased.
>
> >But lets consider the following case: user has a cluster of 50 nodes and
> >decides to switch off 3 nodes for maintenance for a while. Ok, user just
> >does it and then recreates this "minimal node set" to only 47 nodes.
>
> >So initial minimal node set was decreased - something counter-intuitive to
> >me and may cause confusion as well.
>
> That was my point. If I have 50 nodes and 3 backups I can restart on 48, 49
> and 50 without data loss. In case of 48 and 49 after cluster gets activated
> missing backups are assigned and rebalancing starts.
>
> --Yakov
>

Re: Cluster auto activation design proposal

2017-08-03 Thread Yakov Zhdanov

>Obvious connotation of "minimal set" is a set that cannot be decreased.

>But lets consider the following case: user has a cluster of 50 nodes and
>decides to switch off 3 nodes for maintenance for a while. Ok, user just
>does it and then recreates this "minimal node set" to only 47 nodes.

>So initial minimal node set was decreased - something counter-intuitive to
>me and may cause confusion as well.

That was my point. If I have 50 nodes and 3 backups I can restart on 48, 49
and 50 without data loss. In case of 48 and 49 after cluster gets activated
missing backups are assigned and rebalancing starts.

--Yakov

Re: Cluster auto activation design proposal

2017-08-03 Thread Yakov Zhdanov

>I think it is not just restarts, this set of nodes is minimally required
for the cluster to function, no?

I don't think so. Cluster can function if there is no data loss.

--Yakov

Re: Cluster auto activation design proposal

2017-08-03 Thread Dmitry Pavlov

Igniters, what about Target Node Set? Complete Node Set?

As when we reach this topology, we can activate cluster.

чт, 3 авг. 2017 г. в 12:58, Sergey Chugunov :

> Dmitriy,
>
> Obvious connotation of "minimal set" is a set that cannot be decreased.
>
> But lets consider the following case: user has a cluster of 50 nodes and
> decides to switch off 3 nodes for maintenance for a while. Ok, user just
> does it and then recreates this "minimal node set" to only 47 nodes.
>
> So initial minimal node set was decreased - something counter-intuitive to
> me and may cause confusion as well.
>
>
> On Thu, Aug 3, 2017 at 12:37 PM,  wrote:
>
> > Yakov,
> >
> > I think it is not just restarts, this set of nodes is minimally required
> > for the cluster to function, no?
> >
> > ⁣D.
> >
> > On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov <
> yzhda...@apache.org>
> > wrote:
> > >Ю> How about naming it "minimal node set" or "required node set"?
> > >
> > >Required for what? I would add restart if there are no confusion.
> > >
> > >--Yakov
> >
>

Re: Cluster auto activation design proposal

2017-08-03 Thread Sergey Chugunov

Dmitriy,

Obvious connotation of "minimal set" is a set that cannot be decreased.

But lets consider the following case: user has a cluster of 50 nodes and
decides to switch off 3 nodes for maintenance for a while. Ok, user just
does it and then recreates this "minimal node set" to only 47 nodes.

So initial minimal node set was decreased - something counter-intuitive to
me and may cause confusion as well.

On Thu, Aug 3, 2017 at 12:37 PM,  wrote:

> Yakov,
>
> I think it is not just restarts, this set of nodes is minimally required
> for the cluster to function, no?
>
> ⁣D.
>
> On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov 
> wrote:
> >Ю> How about naming it "minimal node set" or "required node set"?
> >
> >Required for what? I would add restart if there are no confusion.
> >
> >--Yakov
>

Re: Cluster auto activation design proposal

2017-08-03 Thread dsetrakyan

Yakov,

I think it is not just restarts, this set of nodes is minimally required for 
the cluster to function, no?

⁣D.

On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov  
wrote:
>Ю> How about naming it "minimal node set" or "required node set"?
>
>Required for what? I would add restart if there are no confusion.
>
>--Yakov

Re: Cluster auto activation design proposal

2017-08-03 Thread Sergey Chugunov

>From my standpoint name for the concept should emphasize that nodes from
the set constitute a target topology - the place where user wants to be.

If we go in a "node set" way, what about FixedNodeSet or BaseNodeSet?

"restart node set" also is a bit confusing because this concept works not
only to restart but to manage adding and removing nodes to/from cluster.

E.g. cluster admin decides to add ten more nodes to existing cluster:
he/she starts them one by one, nodes join the cluster but don't receive any
data as they are not in FixedNodeSet yet.
Then admin issues "change fixed node set" command or adds them to the set
in some other way and nodes become operational.
As one can see, no restarts are involved in the process.

Thanks,
Sergey.

On Thu, Aug 3, 2017 at 12:23 PM, Yakov Zhdanov  wrote:

> Ю> How about naming it "minimal node set" or "required node set"?
>
> Required for what? I would add restart if there are no confusion.
>
> --Yakov
>

Re: Cluster auto activation design proposal

2017-08-03 Thread Yakov Zhdanov

Ю> How about naming it "minimal node set" or "required node set"?

Required for what? I would add restart if there are no confusion.

--Yakov

Re: Cluster auto activation design proposal

2017-08-03 Thread dsetrakyan

How about naming it "minimal node set" or "required node set"?

⁣D.

On Aug 3, 2017, 11:15 AM, at 11:15 AM, Yakov Zhdanov  
wrote:
>> * Based on some sort of policies when the actual cluster topology
>differs
>too much from the baseline or when some critical condition happens
>(e.g.,
>when there are no more backups for a partition)
>
>Good point, Alex! I would even go further. If cluster is active and
>under
>load and nodes continue joining and leaving then we can have several
>BT's
>that are possible to restart on - the main condition is to have all the
>up
>to date data partitions. I.e. if you have 4 servers and 3 backups most
>probably you can have all the data with 2, 3 and, of course, 4 nodes.
>Makes
>sense?
>
>I would also think of different name. Topology (for me) also implies
>the
>version, but here only nodes carrying data are important. How about
>"restart nodes set"?
>
>--Yakov

Re: Cluster auto activation design proposal

2017-08-03 Thread Yakov Zhdanov

> * Based on some sort of policies when the actual cluster topology differs
too much from the baseline or when some critical condition happens (e.g.,
when there are no more backups for a partition)

Good point, Alex! I would even go further. If cluster is active and under
load and nodes continue joining and leaving then we can have several BT's
that are possible to restart on - the main condition is to have all the up
to date data partitions. I.e. if you have 4 servers and 3 backups most
probably you can have all the data with 2, 3 and, of course, 4 nodes. Makes
sense?

I would also think of different name. Topology (for me) also implies the
version, but here only nodes carrying data are important. How about
"restart nodes set"?

--Yakov

Re: Cluster auto activation design proposal

2017-08-02 Thread Dmitriy Setrakyan

First of all, this sounds a bit too scientific and will likely be
misunderstood and misused. Is there a chance to give it a better name and a
simpler use case and description?

Secondly, I do agree that it should be mandatory to update the BT. For
example, what if I want to add several nodes to the cluster for scalability
reasons? As Alexey G. suggested, perhaps we can have some policies that
will update the baseline topology automatically and will properly and
clearly report it to the user in logs.

D.

On Wed, Aug 2, 2017 at 9:51 AM, Alexey Goncharuk  wrote:

> I think we should be able to change the BT in the runtime, and a user
> should have several ways to do this:
>
>  * programmatically via the API suggested by Sergey
>  * Using management tools (console visor on Web Console)
>  * Based on some sort of policies when the actual cluster topology differs
> too much from the baseline or when some critical condition happens (e.g.,
> when there are no more backups for a partition)
>
>
>
> 2017-08-01 22:21 GMT+03:00 Denis Magda :
>
> > Sergey,
> >
> > Is it assumed that the baseline topology can be updated in runtime? Like,
> > initially I had a cluster of 10 nodes but in a couple of weeks it was
> > expanded to 15 nodes. How the baseline topology should be updated in this
> > way? Will it happen automatically?
> >
> > —
> > Denis
> >
> > > On Aug 1, 2017, at 7:51 AM, Sergey Chugunov  >
> > wrote:
> > >
> > > Hello Ignite developers,
> > >
> > > I would like to start a discussion about design of important
> improvement
> > > enabling automatic activation of cluster with durable store turned on
> > [1].
> > > Also it will help us to solve an issue with data divergence (e.g. this
> > may
> > > happen when half of the cluster goes down and updates are applied to
> > > another half, and than online and offline parts of the cluster switch).
> > >
> > > The idea is to introduce a *BaselineTopology *concept. Simplifying it
> is
> > > just a collection of nodes that are expected to be in the cluster.
> > >
> > > User establishes BaselineTopology (BT) on a cluster of desired
> > > configuration (I mean here number of nodes in the first place), after
> > that
> > > this topology is persisted.
> > >
> > > Once established BT represents a "frozen state" of topology which means
> > > that affinity function uses it instead of actual topology. As a result
> no
> > > rebalancing can happen until BT is reestablished.
> > >
> > > Having BT established it is easy to implement automatic activation:
> when
> > > nodes of starting cluster join it one by one, a special listener may
> > > trigger cluster activation when composition of nodes matches with the
> one
> > > described by BaselineTopology.
> > >
> > > API for BaselineTopology manipulation may look like this:
> > >
> > > *Ignite::activation::establishBaselineTopology();*
> > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > bltTop);*
> > >
> > > Both methods will establish BT and activate cluster once it is
> > established.
> > >
> > > The first one allows user to establish BT using current topology. If
> any
> > > changes happen to the topology during establishing process, user will
> be
> > > notified and allowed to proceed or abort the procedure.
> > >
> > > Second method allows to use some monitoring'n'management tools like
> > > WebConsole where user can prepare a list of nodes, using them create a
> BT
> > > and send to the cluster a command to finally establish it.
> > >
> > > From high level BaselineTopology entity contains only collection of
> > nodes:
> > >
> > > *BaselineTopology {*
> > > *  Collection nodes;*
> > > *}*
> > >
> > > *TopologyNode* here contains information about node - its consistent id
> > and
> > > set of user attributes used to calculate affinity function.
> > >
> > > In order to support data divergence prevention some kind of versioning
> > must
> > > be added to BT entity to refuse joining new node but we can clarify it
> > > later.
> > >
> > > Please provide your feedback/thoughts and ask any questions about
> > suggested
> > > improvement.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-5851
> >
> >
>

Re: Cluster auto activation design proposal

2017-08-02 Thread Alexey Goncharuk

I think we should be able to change the BT in the runtime, and a user
should have several ways to do this:

 * programmatically via the API suggested by Sergey
 * Using management tools (console visor on Web Console)
 * Based on some sort of policies when the actual cluster topology differs
too much from the baseline or when some critical condition happens (e.g.,
when there are no more backups for a partition)



2017-08-01 22:21 GMT+03:00 Denis Magda :

> Sergey,
>
> Is it assumed that the baseline topology can be updated in runtime? Like,
> initially I had a cluster of 10 nodes but in a couple of weeks it was
> expanded to 15 nodes. How the baseline topology should be updated in this
> way? Will it happen automatically?
>
> —
> Denis
>
> > On Aug 1, 2017, at 7:51 AM, Sergey Chugunov 
> wrote:
> >
> > Hello Ignite developers,
> >
> > I would like to start a discussion about design of important improvement
> > enabling automatic activation of cluster with durable store turned on
> [1].
> > Also it will help us to solve an issue with data divergence (e.g. this
> may
> > happen when half of the cluster goes down and updates are applied to
> > another half, and than online and offline parts of the cluster switch).
> >
> > The idea is to introduce a *BaselineTopology *concept. Simplifying it is
> > just a collection of nodes that are expected to be in the cluster.
> >
> > User establishes BaselineTopology (BT) on a cluster of desired
> > configuration (I mean here number of nodes in the first place), after
> that
> > this topology is persisted.
> >
> > Once established BT represents a "frozen state" of topology which means
> > that affinity function uses it instead of actual topology. As a result no
> > rebalancing can happen until BT is reestablished.
> >
> > Having BT established it is easy to implement automatic activation: when
> > nodes of starting cluster join it one by one, a special listener may
> > trigger cluster activation when composition of nodes matches with the one
> > described by BaselineTopology.
> >
> > API for BaselineTopology manipulation may look like this:
> >
> > *Ignite::activation::establishBaselineTopology();*
> > *Ignite::activation::establishBaselineTopology(BaselineTopology
> bltTop);*
> >
> > Both methods will establish BT and activate cluster once it is
> established.
> >
> > The first one allows user to establish BT using current topology. If any
> > changes happen to the topology during establishing process, user will be
> > notified and allowed to proceed or abort the procedure.
> >
> > Second method allows to use some monitoring'n'management tools like
> > WebConsole where user can prepare a list of nodes, using them create a BT
> > and send to the cluster a command to finally establish it.
> >
> > From high level BaselineTopology entity contains only collection of
> nodes:
> >
> > *BaselineTopology {*
> > *  Collection nodes;*
> > *}*
> >
> > *TopologyNode* here contains information about node - its consistent id
> and
> > set of user attributes used to calculate affinity function.
> >
> > In order to support data divergence prevention some kind of versioning
> must
> > be added to BT entity to refuse joining new node but we can clarify it
> > later.
> >
> > Please provide your feedback/thoughts and ask any questions about
> suggested
> > improvement.
> >
> > Thanks,
> > Sergey.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5851
>
>

Re: Cluster auto activation design proposal

2017-08-01 Thread Denis Magda

Sergey,

Is it assumed that the baseline topology can be updated in runtime? Like, 
initially I had a cluster of 10 nodes but in a couple of weeks it was expanded 
to 15 nodes. How the baseline topology should be updated in this way? Will it 
happen automatically?

—
Denis

> On Aug 1, 2017, at 7:51 AM, Sergey Chugunov  wrote:
> 
> Hello Ignite developers,
> 
> I would like to start a discussion about design of important improvement
> enabling automatic activation of cluster with durable store turned on [1].
> Also it will help us to solve an issue with data divergence (e.g. this may
> happen when half of the cluster goes down and updates are applied to
> another half, and than online and offline parts of the cluster switch).
> 
> The idea is to introduce a *BaselineTopology *concept. Simplifying it is
> just a collection of nodes that are expected to be in the cluster.
> 
> User establishes BaselineTopology (BT) on a cluster of desired
> configuration (I mean here number of nodes in the first place), after that
> this topology is persisted.
> 
> Once established BT represents a "frozen state" of topology which means
> that affinity function uses it instead of actual topology. As a result no
> rebalancing can happen until BT is reestablished.
> 
> Having BT established it is easy to implement automatic activation: when
> nodes of starting cluster join it one by one, a special listener may
> trigger cluster activation when composition of nodes matches with the one
> described by BaselineTopology.
> 
> API for BaselineTopology manipulation may look like this:
> 
> *Ignite::activation::establishBaselineTopology();*
> *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);*
> 
> Both methods will establish BT and activate cluster once it is established.
> 
> The first one allows user to establish BT using current topology. If any
> changes happen to the topology during establishing process, user will be
> notified and allowed to proceed or abort the procedure.
> 
> Second method allows to use some monitoring'n'management tools like
> WebConsole where user can prepare a list of nodes, using them create a BT
> and send to the cluster a command to finally establish it.
> 
> From high level BaselineTopology entity contains only collection of nodes:
> 
> *BaselineTopology {*
> *  Collection nodes;*
> *}*
> 
> *TopologyNode* here contains information about node - its consistent id and
> set of user attributes used to calculate affinity function.
> 
> In order to support data divergence prevention some kind of versioning must
> be added to BT entity to refuse joining new node but we can clarify it
> later.
> 
> Please provide your feedback/thoughts and ask any questions about suggested
> improvement.
> 
> Thanks,
> Sergey.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-5851

Cluster auto activation design proposal

2017-08-01 Thread Sergey Chugunov

Hello Ignite developers,

I would like to start a discussion about design of important improvement
enabling automatic activation of cluster with durable store turned on [1].
Also it will help us to solve an issue with data divergence (e.g. this may
happen when half of the cluster goes down and updates are applied to
another half, and than online and offline parts of the cluster switch).

The idea is to introduce a *BaselineTopology *concept. Simplifying it is
just a collection of nodes that are expected to be in the cluster.

User establishes BaselineTopology (BT) on a cluster of desired
configuration (I mean here number of nodes in the first place), after that
this topology is persisted.

Once established BT represents a "frozen state" of topology which means
that affinity function uses it instead of actual topology. As a result no
rebalancing can happen until BT is reestablished.

Having BT established it is easy to implement automatic activation: when
nodes of starting cluster join it one by one, a special listener may
trigger cluster activation when composition of nodes matches with the one
described by BaselineTopology.

API for BaselineTopology manipulation may look like this:

*Ignite::activation::establishBaselineTopology();*
*Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);*

Both methods will establish BT and activate cluster once it is established.

The first one allows user to establish BT using current topology. If any
changes happen to the topology during establishing process, user will be
notified and allowed to proceed or abort the procedure.

Second method allows to use some monitoring'n'management tools like
WebConsole where user can prepare a list of nodes, using them create a BT
and send to the cluster a command to finally establish it.

>From high level BaselineTopology entity contains only collection of nodes:

*BaselineTopology {*
*  Collection nodes;*
*}*

*TopologyNode* here contains information about node - its consistent id and
set of user attributes used to calculate affinity function.

In order to support data divergence prevention some kind of versioning must
be added to BT entity to refuse joining new node but we can clarify it
later.

Please provide your feedback/thoughts and ask any questions about suggested
improvement.

Thanks,
Sergey.

[1] https://issues.apache.org/jira/browse/IGNITE-5851

48 matches

Mail list logo