Re: Contribution to apache gossip

[Ankur Agrawal]

Hi

I have made some progress with the design and implementation for tickets
related to GOSSIP-70. I looked at all the 5 sub tickets for GOSSIP-70,
because thinking about them together helped me with clearly understanding
the requirement for each ticket. I am writing the details for each ticket
below as per my understanding, so that we are at same page before I start
writing the code.

*GOSSIP-94: Create latitude and longitude based targeting*
I have a basic implementation for this ready (this was simple enough).
However since this ticket builds over GOSSIP-93, I have a question about
this. The generateTargets method of OperationTargets class takes
List
living  as one of the arguments. Latitude and longitude based targeting
will make sense only if LocalMember class' properties map has key-values
latitude and longitude. So I thing a good solution might look like following

1. Create a subclass of LocalMember, called LocalMemberWithCoordinates
(pardon lack of a better name for now)
2. LocalMemberWithCoordinates has a constructor that takes co-ordinates as
one of the parameters
3. The function generateTargets takes List living
and List living as arguments (instead of
List living and List dead)

Please let me know if this makes sense.

*GOSSIP-95: Create a result merger*
The ResultMerger needs to work on List. It will be called with
this list of response objects by the co-ordinator, once the co-ordinator
hears back from all the executors. The response class that we have doesn't
have any structure. We will need to create a subclass, (lets call it
RWResponse for now). This subclass will have key value pair representing
the data returned by nodes for the request with key.


*GOSSIP-96: Create a multi-node request co-ordinator*
The request co-ordinator will issue requests to multiple nodes with futures
and block for n-results.
1. We can create ReadRequestMessages and WriteRequestMessages classes, as
subclasses of Message. These messages can be sent using gossipCore.send
method.
2. For the receiver side, we need to implement ReadRequestHandler and
WriteRequestHandler. Since the concrete implementation for this will be
given by the application using ApacheGossip, we can provide a abstract
implementation for now. The concrete implementations provided by the
application can inherit from the abstract implementation.
3. The Response objects created from the ticket GOSSIP-95 can be used to
return a response.


*GOSSIP-97: Implement model and message handler*
This is already covered above.

*GOSSIP-70: Build a GeoMetric/GridQuorum model*
Create a read and write function as mentioned by you earlier, with the
following signatures:
void write( String key, SharedGossipMessage message, Operationtargets c,
Consistency con)
Object read(String key, OperationTargets c, Consistency con)

Write Example
1. One getting a write request, we can use the generateTargets of
"OperationTargets c" object to get the list of members we want to send
request to.
2. Send a request to the co-ordinator with the list of target members,
message and key.
3. Apply the consistency model on the received responses.
4. Merge and send the results to the user.

Thanks
Ankur


On Fri, Aug 18, 2017 at 10:00 AM, Ankur Agrawal 
wrote:

> Hi Edward
>
> Yes I saw that. I have made some initial outline and notes about the
> implementation that I was planning to share with you, but unfortunately I
> got sidetracked by some assignments and exams.
> I am planning to start working on it from Wednesday again. I have free
> time this week starting Wednesday when I can make good progress.
>
> Please let me know if this timeline looks ok to you.
>
> Thanks
> Ankur
>
> On Fri, Aug 18, 2017 at 8:41 AM Edward Capriolo 
> wrote:
>
>> You may have noticed that I broke out the task into an umbrella task. I
>> submitted a PR with one possibly implementation of implementing a "geo"
>> node chooser and created some tickets to add others. I also hinted at some
>> ways of doing the other components.
>>
>> On Thu, Aug 10, 2017 at 10:48 PM, Edward Capriolo 
>> wrote:
>>
>>>
>>> On Thu, Aug 10, 2017 at 4:27 AM, Ankur Agrawal <
>>> ankuragrawal...@gmail.com> wrote:
>>>
 Hi Edward

 Thanks for sharing your notes.

 Most of what you have said makes sense to me as far as functionality is
 concerned. I need to take a stab at understanding the code flow again, and
 think through the entire flow, which might take a day or two. After that I
 will come up with a more detailed design, and maybe more questions :)

 Thanks
 Ankur

 On Wed, Aug 9, 2017 at 6:17 PM, Edward Capriolo 
 wrote:

>
>
> On Wed, Aug 9, 2017 at 6:32 PM, Ankur Agrawal <
> ankuragrawal...@gmail.com> wrote:
>
>> Hi Edward,
>>
>> Thanks for your response. I spent sometime understanding about the
>> Gossip protocol, and the apache-gossip 

Re: Contribution to apache gossip

[Edward Capriolo]

You may have noticed that I broke out the task into an umbrella task. I
submitted a PR with one possibly implementation of implementing a "geo"
node chooser and created some tickets to add others. I also hinted at some
ways of doing the other components.

On Thu, Aug 10, 2017 at 10:48 PM, Edward Capriolo 
wrote:

>
> On Thu, Aug 10, 2017 at 4:27 AM, Ankur Agrawal 
> wrote:
>
>> Hi Edward
>>
>> Thanks for sharing your notes.
>>
>> Most of what you have said makes sense to me as far as functionality is
>> concerned. I need to take a stab at understanding the code flow again, and
>> think through the entire flow, which might take a day or two. After that I
>> will come up with a more detailed design, and maybe more questions :)
>>
>> Thanks
>> Ankur
>>
>> On Wed, Aug 9, 2017 at 6:17 PM, Edward Capriolo 
>> wrote:
>>
>>>
>>>
>>> On Wed, Aug 9, 2017 at 6:32 PM, Ankur Agrawal >> > wrote:
>>>
 Hi Edward,

 Thanks for your response. I spent sometime understanding about the
 Gossip protocol, and the apache-gossip codebase.

 About Gossip-70, I am trying to understand the requirement.

 Does this look like a good blueprint to you:

- Each node is initialized with a n-dimensional co-ordinate.
- When a node creates its membership list, it gets the co-ordinates
of its members.
- Each node can draw a logical graph based on the co-ordinates of
its members.
- This topology can be used by the application that is using
gossip, to forward its updates/messages and to decide quorum based on a
formula.

 Do you think my understanding of basic requirements is correct?

 Thanks
 Ankur

 On Mon, Jul 31, 2017 at 7:58 AM, Edward Capriolo  wrote:

> On Sun, Jul 30, 2017 at 10:59 PM, Ankur Agrawal <
> ankuragrawal...@gmail.com>
> wrote:
>
> > Hi
> >
> > Not sure if my previous email one week ago did actually get
> delivered. So I
> > apologize for resenting in advance.
> >
> > My name is Ankur. I am a student, and I am keenly interested in, and
> want
> > to contribute to this project for implementation of Gossip protocol.
> >
> > I have some job experience in designing instance migration services
> for a
> > popular public cloud infrastructure, and distributed data backup and
> > recovery services. I think this project will help me learn a lot,
> and I can
> > also contribute while learning.
> >
> > I saw that GOSSIP-70 (https://issues.apache.org/jir
> a/browse/GOSSIP-70) is
> > still open. It was mentioned as a project in GSoC 2017 too. If this
> is
> > indeed open, and you guys think I can start with this, it will be
> great.
> > Please note that I am not participating in GSoC and I want to
> contribute on
> > a individual capacity.
> >
> > If there are other open issues where you are looking for
> contribution too,
> > do let me know, and I can get started with that.
> >
> > Also, I am wondering if there is any interest in implementation of
> Gossip
> > in other languages like python or C++.
> >
> > I look forward to hear from you guys.
> >
> > Thanks
> > Ankur
> >
>
> Thanks for your interest!
>
> "Also, I am wondering if there is any interest in implementation of
> Gossip
> in other languages like python or C++."
>
> Originally we planned to port Gossip to multiple languages but there
> has
> not been much interest in that. Currently the code base is evolving and
> porting would be challenging. I think that is a lot for us to bite off
> at
> this point.
>
> GOSSIP-70 is a fine place to start. We have not fleshed that ticket out
> enough. Starting a conversation in the ticket or on the mailing list
> would
> be a good thing to do before we start coding. I like the idea of using
> a
> formula to pick a quorum of nodes and doing an operation across them.
> This
> can be easily extracted but we need to figure out what the first step
> is
> and what a final target might be.
>
> Thanks,
> Edward
>

>>> Your understanding is mostly in line with my thinking.  I will started
>>> some high level design but do not have the interfaces at a point such that I
>>> was ready for a commit. I will share my notes.
>>>
>>> >> Each node is initialized with a n-dimensional co-ordinate.
>>>
>>> That is one route but we can go more simple then that. We need an
>>> interface:
>>>
>>> interface chooser {
>>>public List generateEndpoints(string key, List
>>> live, List dead, LocalNode me);
>>> }
>>>
>>> The most simple implementation is we can hash the node's id into a line,
>>> or circular queue. We can 

Re: Contribution to apache gossip

[Edward Capriolo]

On Sun, Jul 30, 2017 at 10:59 PM, Ankur Agrawal 
wrote:

> Hi
>
> Not sure if my previous email one week ago did actually get delivered. So I
> apologize for resenting in advance.
>
> My name is Ankur. I am a student, and I am keenly interested in, and want
> to contribute to this project for implementation of Gossip protocol.
>
> I have some job experience in designing instance migration services for a
> popular public cloud infrastructure, and distributed data backup and
> recovery services. I think this project will help me learn a lot, and I can
> also contribute while learning.
>
> I saw that GOSSIP-70 (https://issues.apache.org/jira/browse/GOSSIP-70) is
> still open. It was mentioned as a project in GSoC 2017 too. If this is
> indeed open, and you guys think I can start with this, it will be great.
> Please note that I am not participating in GSoC and I want to contribute on
> a individual capacity.
>
> If there are other open issues where you are looking for contribution too,
> do let me know, and I can get started with that.
>
> Also, I am wondering if there is any interest in implementation of Gossip
> in other languages like python or C++.
>
> I look forward to hear from you guys.
>
> Thanks
> Ankur
>

Thanks for your interest!

"Also, I am wondering if there is any interest in implementation of Gossip
in other languages like python or C++."

Originally we planned to port Gossip to multiple languages but there has
not been much interest in that. Currently the code base is evolving and
porting would be challenging. I think that is a lot for us to bite off at
this point.

GOSSIP-70 is a fine place to start. We have not fleshed that ticket out
enough. Starting a conversation in the ticket or on the mailing list would
be a good thing to do before we start coding. I like the idea of using a
formula to pick a quorum of nodes and doing an operation across them. This
can be easily extracted but we need to figure out what the first step is
and what a final target might be.

Thanks,
Edward

Re: Contribution to Apache Gossip

[chandresh pancholi]

Thank you for replying Gary. I have went through the example and got fair
bit if idea about code base.

I would prefer to start contributing by picking Jira tasks and later work
on features.

I have already written code for junit5 Jira ticket. I can raise PR if
allowed by assignee.

On Sep 8, 2016 8:48 PM, "Gary Dusbabek"  wrote:

> It might be a good idea for you to go through the example (
> https://github.com/apache/incubator-gossip/blob/master/
> src/main/java/org/apache/gossip/examples/GossipExample.java)
> to see if you have an itch to scratch.
>
> How would you make the experience better?
>
> Cheers,
>
> Gary.
>
> On Thu, Sep 8, 2016 at 12:56 AM, chandresh pancholi <
> chandreshpancholi...@gmail.com> wrote:
>
> > *Bump*
> > Please assign/suggest newbie jira ticket to get comfortable with
> codebase.
> >
> > Thanls
> >
> > On Tue, Sep 6, 2016 at 7:52 PM, chandresh pancholi <
> > chandreshpancholi...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would like to contribute to Apache gossip. I have been contributed to
> > > Apache Ignite and Concourse database previously and currently working
> on
> > > open source In house workflow engine at Flipkart Internet Pvt. Ltd.
> > >
> > >
> > > Thanks
> > >
> > > --
> > > Chandresh Pancholi
> > > Senior Software Engineer
> > > Flipkart.com
> > > Email-id:chandresh.panch...@flipkart.com
> > > Contact:08951803660
> > >
> >
> >
> >
> > --
> > Chandresh Pancholi
> > Senior Software Engineer
> > Flipkart.com
> > Email-id:chandresh.panch...@flipkart.com
> > Contact:08951803660
> >
>