Re: [RESULT][VOTE] Gobblin to enter Apache Incubator

[Olivier Lamy]

Hi,
My first task was to create the podling file in incubator svn the ask
mailing lists.
So now that's done then we will be able to exchange on mailing list when
they will have been created.
Cheers
Olivier

On 24 February 2017 at 00:06, Jean-Baptiste Onofré  wrote:

> Great !
>
> Here we are ;)
>
> @Olivier, can we sync together to create the resources ?
>
> Thanks !
> Regards
> JB
>
> On 02/23/2017 01:00 AM, Olivier Lamy wrote:
>
>> Hi,
>> The vote has passed with the following result:
>> +1 (binding): Jean-Baptiste, Ross, Colm, Jim, Raphael, Carl, Craig
>> +1 (non binding): Lian, Abhishek, Naresh
>>
>> Vote thread: https://s.apache.org/GOBBLIN_INCUBATION_VOTE
>> I will start all the process in the coming days.
>> Thanks to all the voters!
>>
>> Cheers
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: [RESULT][VOTE] Gobblin to enter Apache Incubator

[Jean-Baptiste Onofré]

Great !

Here we are ;)

@Olivier, can we sync together to create the resources ?

Thanks !
Regards
JB

On 02/23/2017 01:00 AM, Olivier Lamy wrote:

Hi,
The vote has passed with the following result:
+1 (binding): Jean-Baptiste, Ross, Colm, Jim, Raphael, Carl, Craig
+1 (non binding): Lian, Abhishek, Naresh

Vote thread: https://s.apache.org/GOBBLIN_INCUBATION_VOTE
I will start all the process in the coming days.
Thanks to all the voters!

Cheers



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [RESULT][VOTE] Gobblin to enter Apache Incubator

[Abhishek Tiwari]

Thanks Olivier.

You missed to count yourself in the binding votes :)

Regards,
Abhishek

On Wed, Feb 22, 2017 at 4:00 PM, Olivier Lamy  wrote:

> Hi,
> The vote has passed with the following result:
> +1 (binding): Jean-Baptiste, Ross, Colm, Jim, Raphael, Carl, Craig
> +1 (non binding): Lian, Abhishek, Naresh
>
> Vote thread: https://s.apache.org/GOBBLIN_INCUBATION_VOTE
> I will start all the process in the coming days.
> Thanks to all the voters!
>
> Cheers
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

[RESULT][VOTE] Gobblin to enter Apache Incubator

[Olivier Lamy]

Hi,
The vote has passed with the following result:
+1 (binding): Jean-Baptiste, Ross, Colm, Jim, Raphael, Carl, Craig
+1 (non binding): Lian, Abhishek, Naresh

Vote thread: https://s.apache.org/GOBBLIN_INCUBATION_VOTE
I will start all the process in the coming days.
Thanks to all the voters!

Cheers
-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: [VOTE] Gobblin to enter Apache Incubator

[Naresh Agarwal]

+1 (non binding). We had used Gobblin in past and found it quite powerful.
Looking forward to a exciting road-map ahead in Gobblin.

Thanks
Naresh

On Wed, Feb 22, 2017 at 12:31 AM, Abhishek Tiwari <
abhishektiwari.bt...@gmail.com> wrote:

> +1 (non-binding)
>
> Abhishek
>
> On Sun, Feb 19, 2017 at 3:01 PM, Olivier Lamy  wrote:
>
> > Hi Craig
> >
> > On 20 February 2017 at 00:23, Craig Russell 
> > wrote:
> >
> > > Hi Olivier,
> > >
> > > Can you also post the link to this proposal on
> https://wiki.apache.org/
> > > incubator/ProjectProposals ?
> > >
> >
> > Sure. done.
> >
> > Olivier
> >
> >
> > >
> > > Thanks,
> > >
> > > Craig
> > >
> > > > On Feb 16, 2017, at 9:33 PM, Olivier Lamy  wrote:
> > > >
> > > > Hello Everyone,
> > > > I would like to call a vote for accepting "Gobblin" for incubation in
> > the
> > > > Apache Incubator.
> > > > The full proposal is available below, and is also available in the
> > wiki:
> > > > https://wiki.apache.org/incubator/GobblinProposal
> > > >
> > >
> > > Craig L Russell
> > > Secretary, Apache Software Foundation
> > > c...@apache.org http://db.apache.org/jdo
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Olivier Lamy
> > http://twitter.com/olamy | http://linkedin.com/in/olamy
> >
>

Re: [VOTE] Gobblin to enter Apache Incubator

[Abhishek Tiwari]

+1 (non-binding)

Abhishek

On Sun, Feb 19, 2017 at 3:01 PM, Olivier Lamy  wrote:

> Hi Craig
>
> On 20 February 2017 at 00:23, Craig Russell 
> wrote:
>
> > Hi Olivier,
> >
> > Can you also post the link to this proposal on https://wiki.apache.org/
> > incubator/ProjectProposals ?
> >
>
> Sure. done.
>
> Olivier
>
>
> >
> > Thanks,
> >
> > Craig
> >
> > > On Feb 16, 2017, at 9:33 PM, Olivier Lamy  wrote:
> > >
> > > Hello Everyone,
> > > I would like to call a vote for accepting "Gobblin" for incubation in
> the
> > > Apache Incubator.
> > > The full proposal is available below, and is also available in the
> wiki:
> > > https://wiki.apache.org/incubator/GobblinProposal
> > >
> >
> > Craig L Russell
> > Secretary, Apache Software Foundation
> > c...@apache.org http://db.apache.org/jdo
> >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

Re: [VOTE] Gobblin to enter Apache Incubator

[Olivier Lamy]

Hi Craig

On 20 February 2017 at 00:23, Craig Russell 
wrote:

> Hi Olivier,
>
> Can you also post the link to this proposal on https://wiki.apache.org/
> incubator/ProjectProposals ?
>

Sure. done.

Olivier


>
> Thanks,
>
> Craig
>
> > On Feb 16, 2017, at 9:33 PM, Olivier Lamy  wrote:
> >
> > Hello Everyone,
> > I would like to call a vote for accepting "Gobblin" for incubation in the
> > Apache Incubator.
> > The full proposal is available below, and is also available in the wiki:
> > https://wiki.apache.org/incubator/GobblinProposal
> >
>
> Craig L Russell
> Secretary, Apache Software Foundation
> c...@apache.org http://db.apache.org/jdo
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: [VOTE] Gobblin to enter Apache Incubator

[Craig Russell]

Excellent proposal.

+1 (binding)

Craig

> On Feb 16, 2017, at 9:33 PM, Olivier Lamy  wrote:
> 
> Hello Everyone,
> I would like to call a vote for accepting "Gobblin" for incubation in the
> Apache Incubator.
> The full proposal is available below, and is also available in the wiki:
> https://wiki.apache.org/incubator/GobblinProposal
> 
> Please cast your vote:
> 
>  [ ] +1, bring Gobblin into Incubator
>  [ ] +0, I don't care either way,
>  [ ] -1, do not bring Gobblin into Incubator, because...
> 
> The vote will open at least for 72 hours and only votes from the Incubator
> PMC are binding.
> 
> I start with my vote:
> +1 (binding)
> 
> Cheers
> -- 
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
> 
> 
> = Apache Gobblin Proposal =

Craig L Russell
c...@apache.org



-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Craig Russell]

Hi Olivier,

Can you also post the link to this proposal on 
https://wiki.apache.org/incubator/ProjectProposals ?

Thanks,

Craig

> On Feb 16, 2017, at 9:33 PM, Olivier Lamy  wrote:
> 
> Hello Everyone,
> I would like to call a vote for accepting "Gobblin" for incubation in the
> Apache Incubator.
> The full proposal is available below, and is also available in the wiki:
> https://wiki.apache.org/incubator/GobblinProposal
> 

Craig L Russell
Secretary, Apache Software Foundation
c...@apache.org http://db.apache.org/jdo


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Carl Steinbach]

+1 (binding)

- Carl

On Sat, Feb 18, 2017 at 5:46 PM, Liang Chen <chenliang6...@gmail.com> wrote:

> +1 (non-binding)
>
> Regards
> Liang
>
>
>
> --
> View this message in context: http://apache-incubator-
> general.996316.n3.nabble.com/VOTE-Gobblin-to-enter-Apache-
> Incubator-tp53837p53858.html
> Sent from the Apache Incubator - General mailing list archive at
> Nabble.com.
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [VOTE] Gobblin to enter Apache Incubator

[Liang Chen]

+1 (non-binding)

Regards
Liang



--
View this message in context: 
http://apache-incubator-general.996316.n3.nabble.com/VOTE-Gobblin-to-enter-Apache-Incubator-tp53837p53858.html
Sent from the Apache Incubator - General mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Raphael Bircher]

Hi all

+1 (binding) It looks like a really good proposal

Regards Raphael

Am .02.2017, 06:33 Uhr, schrieb Olivier Lamy :


Hello Everyone,
I would like to call a vote for accepting "Gobblin" for incubation in the
Apache Incubator.
The full proposal is available below, and is also available in the wiki:
https://wiki.apache.org/incubator/GobblinProposal

Please cast your vote:

  [ ] +1, bring Gobblin into Incubator
  [ ] +0, I don't care either way,
  [ ] -1, do not bring Gobblin into Incubator, because...

The vote will open at least for 72 hours and only votes from the  
Incubator

PMC are binding.

I start with my vote:
+1 (binding)

Cheers



--
Mein Blog: https://raphaelbircher.blogspot.ch

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Jim Jagielski]

+1 (binding)

> On Feb 17, 2017, at 12:33 AM, Olivier Lamy  wrote:
> 
> Hello Everyone,
> I would like to call a vote for accepting "Gobblin" for incubation in the
> Apache Incubator.
> The full proposal is available below, and is also available in the wiki:
> https://wiki.apache.org/incubator/GobblinProposal
> 
> Please cast your vote:
> 
>  [ ] +1, bring Gobblin into Incubator
>  [ ] +0, I don't care either way,
>  [ ] -1, do not bring Gobblin into Incubator, because...
> 
> The vote will open at least for 72 hours and only votes from the Incubator
> PMC are binding.
> 


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Colm O hEigeartaigh]

+1 (binding)

Colm.

On Fri, Feb 17, 2017 at 7:43 AM, Ross Gardler <r...@gardler.org> wrote:

> +1 (binding)
>
> Not signing up to mentor, but I will be watching and hopefully helping
> from the sidelines.
>
> -Original Message-
> From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net]
> Sent: Thursday, February 16, 2017 11:27 PM
> To: general@incubator.apache.org
> Subject: Re: [VOTE] Gobblin to enter Apache Incubator
>
> +1 (binding)
>
> Happy to be mentor on it !
>
> Regards
> JB
>
> On 02/17/2017 06:33 AM, Olivier Lamy wrote:
> > Hello Everyone,
> > I would like to call a vote for accepting "Gobblin" for incubation in
> > the Apache Incubator.
> > The full proposal is available below, and is also available in the wiki:
> > https://wiki.apache.org/incubator/GobblinProposal
> >
> > Please cast your vote:
> >
> >   [ ] +1, bring Gobblin into Incubator
> >   [ ] +0, I don't care either way,
> >   [ ] -1, do not bring Gobblin into Incubator, because...
> >
> > The vote will open at least for 72 hours and only votes from the
> > Incubator PMC are binding.
> >
> > I start with my vote:
> > +1 (binding)
> >
> > Cheers
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

RE: [VOTE] Gobblin to enter Apache Incubator

[Ross Gardler]

+1 (binding)

Not signing up to mentor, but I will be watching and hopefully helping from the 
sidelines.

-Original Message-
From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net] 
Sent: Thursday, February 16, 2017 11:27 PM
To: general@incubator.apache.org
Subject: Re: [VOTE] Gobblin to enter Apache Incubator

+1 (binding)

Happy to be mentor on it !

Regards
JB

On 02/17/2017 06:33 AM, Olivier Lamy wrote:
> Hello Everyone,
> I would like to call a vote for accepting "Gobblin" for incubation in 
> the Apache Incubator.
> The full proposal is available below, and is also available in the wiki:
> https://wiki.apache.org/incubator/GobblinProposal
>
> Please cast your vote:
>
>   [ ] +1, bring Gobblin into Incubator
>   [ ] +0, I don't care either way,
>   [ ] -1, do not bring Gobblin into Incubator, because...
>
> The vote will open at least for 72 hours and only votes from the 
> Incubator PMC are binding.
>
> I start with my vote:
> +1 (binding)
>
> Cheers
>

--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org



-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [VOTE] Gobblin to enter Apache Incubator

[Jean-Baptiste Onofré]

+1 (binding)

Happy to be mentor on it !

Regards
JB

On 02/17/2017 06:33 AM, Olivier Lamy wrote:

Hello Everyone,
I would like to call a vote for accepting "Gobblin" for incubation in the
Apache Incubator.
The full proposal is available below, and is also available in the wiki:
https://wiki.apache.org/incubator/GobblinProposal

Please cast your vote:

  [ ] +1, bring Gobblin into Incubator
  [ ] +0, I don't care either way,
  [ ] -1, do not bring Gobblin into Incubator, because...

The vote will open at least for 72 hours and only votes from the Incubator
PMC are binding.

I start with my vote:
+1 (binding)

Cheers



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

[VOTE] Gobblin to enter Apache Incubator

[Olivier Lamy]

Hello Everyone,
I would like to call a vote for accepting "Gobblin" for incubation in the
Apache Incubator.
The full proposal is available below, and is also available in the wiki:
https://wiki.apache.org/incubator/GobblinProposal

Please cast your vote:

  [ ] +1, bring Gobblin into Incubator
  [ ] +0, I don't care either way,
  [ ] -1, do not bring Gobblin into Incubator, because...

The vote will open at least for 72 hours and only votes from the Incubator
PMC are binding.

I start with my vote:
+1 (binding)

Cheers
-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy


= Apache Gobblin Proposal =
== Abstract ==
Gobblin is a distributed data integration framework that simplifies common
aspects of big data integration such as data ingestion, replication,
organization and lifecycle management for both streaming and batch data
ecosystems.

== Proposal ==

Gobblin is a universal data integration framework. The framework has been
used to build a variety of big data applications such as ingestion,
replication, and data retention. The fundamental constructs provided by the
Gobblin framework are:

 1. An expandable set of connectors that allow data to be integrated from a
variety of sources and sinks. The range of connectors already available in
Gobblin are quite diverse and are an ever expanding set. To highlight just
a few examples, connectors exist for databases (e.g., MySQL, Oracle
Teradata, Couchbase etc.), web based technologies (REST APIs, FTP/SFTP
servers, Filers), scalable storage (HDFS, S3, Ambry etc,), streaming data
(Kafka, EventHubs etc.), and a variety of proprietary data sources and
sinks (e.g.Salesforce, Google Analytics, Google Webmaster etc.). Similarly,
Gobblin has a rich library of converters that allow for conversion of data
from one format to another as data moves across system boundaries (e.g.
AVRO in HDFS to JSON in another system).


 2. Gobblin has a well defined and customizable state management layer that
allows writing stateful applications. These are particularly useful when
solving problems like bulk incremental ingest and keeping several clusters
replicated in sync. The ability to record work that has been completed and
what remains in a scalable manner is critical to writing such diverse
applications successfully.


 3. Gobblin is agnostic to the underlying execution engine. It can be
tailored to run ontop of a variety of execution frameworks ranging from
multiple processes on a single node, to open source execution engines like
MapReduce, Spark or Samza, natively on top of raw containers like Yarn or
Mesos, and the public cloud like Amazon AWS or Microsoft Azure. We are
extending Gobblin to run on top of a self managed cluster when security is
vital.  This allows different applications that require different degrees
of scalability, latency or security to be customized to for their specific
needs. For example, highly latency sensitive applications can be executed
in a streaming environment while batch based execution might benefit
applications where the priority might be geared towards optimal container
utilization.

 4.Gobblin comes out of the box with several diagnosability features like
Gobblin metrics and error handling. Collectively, these features allow
Gobblin to operate at the scale of petabytes of data. To give just one
example, the ability to quarantine a few bad records from an isolated Kafka
topic without stopping the entire flow from continued execution is vital
when the number of Kafka topics range in the thousands and the collective
data handled is in the petabytes.

Gobblin thus provides crisply defined software constructs that can be used
to build a vast array of data integration applications customizable for
varied user needs. It has become a preferred technology for data
integration use-cases by many organizations worldwide (see a partial list
here).

== Background ==

Over the last decade, data integration has evolved use case by use case in
most companies. For example, at LinkedIn, when Kafka became a significant
part of the data ecosystem, a system called Camus was built to ingest this
data for analytics processing on Hadoop. Similarly, we had custom pipelines
to ingest data from Salesforce, Oracle and myriad other sources. This
pattern became the norm rather than the exception and one point, LinkedIn
was running at least fifteen different types of ingestion pipelines. This
fragmentation has several unfortunate implications. Operational costs scale
with the number of pipelines even if the myriad pipelines share a vasty
array of common features. Bug fixes and performance optimizations cannot be
shared across the pipelines. A common set of practices around debugging and
deployment does not emerge. Each pipeline operator will continue to invest
in his little silo of the data integration world completely oblivious to
the challenges of his fellow operator sitting five tables down.

These experiences were the genesis behind the design and