Migrating development to incubator

[Matteo Merli]

Hi All,

I hope everyone already subscribed this mailing list already.

I wanted to start the discussion on multiple points for onboarding into the
incubator.

First of all, which tools we want to keep using when moving the development
over at the ASF.

In particular, we need to chose about:
 1. Git repository and committer workflow
 2. Github issues vs JIRA
 3. Jenkins vs Travis for CI

The support for using Github tools for Apache project has been a recent
addition. Masakazu pointed to me that it was actually discussed at the
latest ApacheCon (https://youtu.be/yWurOHvm5WM?t=1078) and that right now
INFRA is supporting projects to use Github as the primary repository and
issues.

I think for us it would make sense to continue using Github tools that has
worked reasonably well in the past and it will be frictionless both from
our perspective as for potential contributors.

The other point would be to directly transfer the repository from
"yahoo/pulsar" to "apache/incubator-pulsar", instead of creating a new
repo.
The advantage would be to keep all the current issues/PRs, plus people that
are subscribed to the repository events.

I think that the logistic for this would be to give access to yahoo/pulsar
repo to ASF INFRA so that they can perform the switch.

About the SGA from Yahoo, Joe can you also update here whenever the grant
is submitted?

Finally, when repository and grant aspects are resolved, I would suggest to
make one last release (1.18), ASAP,  before moving the code over to the
ASF. The reason is that we have already accumulated lot of changes and
fixes in the current master and that it will take us some amount of time to
prepare well for an official release within the incubator.
We will need to sort lot of details and possibly make multiple iterations
before we can be ready for a release.

So, release what we have right now, and then concentrate in making a proper
release in the incubator (detached from the amount of "features" contained).

Any thoughts / opinions?

Please also raise any other point or question that I have missed.

Matteo

Proposal for Pulsar "PIP"

[Matteo Merli]

So far the discussion for large code changes (new features, improvements)
in the project has happened in a very informal way.

As a way to improve community involvement and also to have a better
internal
documentation I would like to propose to adopt the PIP model that many
projects
follow.

Here, PIP would stand for "Pulsar Improvement Proposal" and would consist
in creating design documents in the Wiki with a sequential id.

The document doesn't need to be a super-detailed specification, but should
explain all the design points, reasons for choosing a determined solution,
rejected alternatives and allow for other contributors to understand the
feature/change and to contribute as well to it.

The advantage would be to have a preliminary discussion and gather feedback
on the design itself and also have the proposals there as a reference.

In some cases the design has been discussed over "issues" or PRs but then
later it's more difficult to understand the whole picture, since the
information is fragmented.

Opinions?

Matteo

Re: Proposal for Pulsar "PIP"

[Matteo Merli]

It would be nice to have more design docs with details on the system
implementation. That would be a huge task though, volunteers are welcome! :)

Matteo


On Thu, Jun 8, 2017 at 2:05 PM Sahaya Andrews 
wrote:

> I like the proposal.
>
> Along with that should we also consider writing documents for the
> features already present?
>
> Andrews.
>
>
> On Thu, Jun 8, 2017 at 1:13 PM, Matteo Merli  wrote:
> > So far the discussion for large code changes (new features, improvements)
> > in the project has happened in a very informal way.
> >
> > As a way to improve community involvement and also to have a better
> > internal
> > documentation I would like to propose to adopt the PIP model that many
> > projects
> > follow.
> >
> > Here, PIP would stand for "Pulsar Improvement Proposal" and would consist
> > in creating design documents in the Wiki with a sequential id.
> >
> > The document doesn't need to be a super-detailed specification, but
> should
> > explain all the design points, reasons for choosing a determined
> solution,
> > rejected alternatives and allow for other contributors to understand the
> > feature/change and to contribute as well to it.
> >
> > The advantage would be to have a preliminary discussion and gather
> feedback
> > on the design itself and also have the proposals there as a reference.
> >
> > In some cases the design has been discussed over "issues" or PRs but then
> > later it's more difficult to understand the whole picture, since the
> > information is fragmented.
> >
> > Opinions?
> >
> > Matteo
>

Re: Migrating development to incubator

[Matteo Merli]

Any thought about the JIRA vs github issues for tracking?


On Thu, Jun 8, 2017 at 5:56 PM Joe F  wrote:

> The SGA has just been submitted.
>
> Joe
>
> On Wed, Jun 7, 2017 at 6:16 PM, Dave Fisher  wrote:
>
> > Hi Folks,
> >
> > Please respond to emails here and discuss the podling. In order to move
> to
> > the incubator we need to start talking.
> >
> > > On Jun 5, 2017, at 1:15 PM, Matteo Merli  wrote:
> > >
> > > Hi All,
> > >
> > > I hope everyone already subscribed this mailing list already.
> >
> > Several iCLAs were processed by the ASF secretary on Friday. Accounts
> > should be setup and I will add to the LDAP groups tomorrow.
> >
> > >
> > > I wanted to start the discussion on multiple points for onboarding into
> > the
> > > incubator.
> > >
> > > First of all, which tools we want to keep using when moving the
> > development
> > > over at the ASF.
> > >
> > > In particular, we need to chose about:
> > > 1. Git repository and committer workflow
> > > 2. Github issues vs JIRA
> >
> > The proposal requested a JIRA which I requested and is now available at
> > https://issues.apache.org/jira/browse/PULSAR/
> >
> > If the project wishes to switch to Github issues that would be fine.
> >
> > > 3. Jenkins vs Travis for CI
> > >
> > > The support for using Github tools for Apache project has been a recent
> > > addition. Masakazu pointed to me that it was actually discussed at the
> > > latest ApacheCon (https://youtu.be/yWurOHvm5WM?t=1078) and that right
> > now
> > > INFRA is supporting projects to use Github as the primary repository
> and
> > > issues.
> > >
> > > I think for us it would make sense to continue using Github tools that
> > has
> > > worked reasonably well in the past and it will be frictionless both
> from
> > > our perspective as for potential contributors.
> > >
> > > The other point would be to directly transfer the repository from
> > > "yahoo/pulsar" to "apache/incubator-pulsar", instead of creating a new
> > > repo.
> > > The advantage would be to keep all the current issues/PRs, plus people
> > that
> > > are subscribed to the repository events.
> > >
> > > I think that the logistic for this would be to give access to
> > yahoo/pulsar
> > > repo to ASF INFRA so that they can perform the switch.
> >
> > We need to have a discussion on this. We can discuss Consensus as part of
> > the discussion.
> >
> > >
> > > About the SGA from Yahoo, Joe can you also update here whenever the
> grant
> > > is submitted?
> > >
> > > Finally, when repository and grant aspects are resolved, I would
> suggest
> > to
> > > make one last release (1.18), ASAP,  before moving the code over to the
> > > ASF. The reason is that we have already accumulated lot of changes and
> > > fixes in the current master and that it will take us some amount of
> time
> > to
> > > prepare well for an official release within the incubator.
> > > We will need to sort lot of details and possibly make multiple
> iterations
> > > before we can be ready for a release.
> >
> > If you want to do a release on the old infrastructure then perhaps that
> is
> > first.
> >
> > >
> > > So, release what we have right now, and then concentrate in making a
> > proper
> > > release in the incubator (detached from the amount of "features"
> > contained).
> > >
> > > Any thoughts / opinions?
> > >
> > > Please also raise any other point or question that I have missed.
> > >
> > > Matteo
> >
> > Regards,
> > Dave
> >
> >
> >
>

Re: Migrating development to incubator

[Matteo Merli]

Joe, that is correct, that person would be someone from ASF INFRA team,
once we open a ticket to request it.
Is there any concern from Yahoo perspective to give admin permission on the
repo for the migration?



On Mon, Jun 12, 2017 at 11:12 AM Joe Francis 
wrote:

> From talking to the relevant folks here at Yahoo, it seems github requires
> that the person doing the transfer needs to have have admin rights on both
> the source and destination repos.
> Joe
>
> On Friday, June 9, 2017 1:07 PM, Dave Fisher 
> wrote:
>
>
>  Hi Joe,
>
> I see that the secretary has registered the SGA in the Foundation archives.
>
> I’ve updated the status page to include this fact along with the list of
> initial committers. So far 8 of the 16 initial committers have iCLAs on
> file.
>
> Regards,
> Dave
>
> > On Jun 8, 2017, at 5:56 PM, Joe F  wrote:
> >
> > The SGA has just been submitted.
> >
> > Joe
> >
> > On Wed, Jun 7, 2017 at 6:16 PM, Dave Fisher 
> wrote:
> >
> >> Hi Folks,
> >>
> >> Please respond to emails here and discuss the podling. In order to move
> to
> >> the incubator we need to start talking.
> >>
> >>> On Jun 5, 2017, at 1:15 PM, Matteo Merli  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I hope everyone already subscribed this mailing list already.
> >>
> >> Several iCLAs were processed by the ASF secretary on Friday. Accounts
> >> should be setup and I will add to the LDAP groups tomorrow.
> >>
> >>>
> >>> I wanted to start the discussion on multiple points for onboarding into
> >> the
> >>> incubator.
> >>>
> >>> First of all, which tools we want to keep using when moving the
> >> development
> >>> over at the ASF.
> >>>
> >>> In particular, we need to chose about:
> >>> 1. Git repository and committer workflow
> >>> 2. Github issues vs JIRA
> >>
> >> The proposal requested a JIRA which I requested and is now available at
> >> https://issues.apache.org/jira/browse/PULSAR/
> >>
> >> If the project wishes to switch to Github issues that would be fine.
> >>
> >>> 3. Jenkins vs Travis for CI
> >>>
> >>> The support for using Github tools for Apache project has been a recent
> >>> addition. Masakazu pointed to me that it was actually discussed at the
> >>> latest ApacheCon (https://youtu.be/yWurOHvm5WM?t=1078) and that right
> >> now
> >>> INFRA is supporting projects to use Github as the primary repository
> and
> >>> issues.
> >>>
> >>> I think for us it would make sense to continue using Github tools that
> >> has
> >>> worked reasonably well in the past and it will be frictionless both
> from
> >>> our perspective as for potential contributors.
> >>>
> >>> The other point would be to directly transfer the repository from
> >>> "yahoo/pulsar" to "apache/incubator-pulsar", instead of creating a new
> >>> repo.
> >>> The advantage would be to keep all the current issues/PRs, plus people
> >> that
> >>> are subscribed to the repository events.
> >>>
> >>> I think that the logistic for this would be to give access to
> >> yahoo/pulsar
> >>> repo to ASF INFRA so that they can perform the switch.
> >>
> >> We need to have a discussion on this. We can discuss Consensus as part
> of
> >> the discussion.
> >>
> >>>
> >>> About the SGA from Yahoo, Joe can you also update here whenever the
> grant
> >>> is submitted?
> >>>
> >>> Finally, when repository and grant aspects are resolved, I would
> suggest
> >> to
> >>> make one last release (1.18), ASAP,  before moving the code over to the
> >>> ASF. The reason is that we have already accumulated lot of changes and
> >>> fixes in the current master and that it will take us some amount of
> time
> >> to
> >>> prepare well for an official release within the incubator.
> >>> We will need to sort lot of details and possibly make multiple
> iterations
> >>> before we can be ready for a release.
> >>
> >> If you want to do a release on the old infrastructure then perhaps that
> is
> >> first.
> >>
> >>>
> >>> So, release what we have right now, and then concentrate in making a
> >> proper
> >>> release in the incubator (detached from the amount of "features"
> >> contained).
> >>>
> >>> Any thoughts / opinions?
> >>>
> >>> Please also raise any other point or question that I have missed.
> >>>
> >>> Matteo
> >>
> >> Regards,
> >> Dave
> >>
> >>
> >>
>
>
>

Re: Proposal for Pulsar "PIP"

[Matteo Merli]

> Is there specific criteria for determining whether documents are
necessary or not?

That's a good question. I don't have a precise answer for it. As Andrews
stated, it should
allow any contributor to understand the new feature/changes, enough to be
able to
do a meaningful code review and also serve as design documentation for
future
contributors.

So, if after the discussion, some details are changed, the wiki should be
updated to
reflect the final state.

If one thinks that additional context is not required, he should go ahead
with a PR. And
later we can discuss whether a design doc for that change would be
beneficial or not.

For candidates, and to give an example, I was thinking to start with a
"Pulsar native proxy"
 proposal that I'm working on. Also, I think other good candidates would be
the
non-persistent topics from Rajan, or the delivery throttling.



On Thu, Jun 8, 2017 at 10:21 PM Sahaya Andrews 
wrote:

> On Thu, Jun 8, 2017 at 7:08 PM, Nozomi Kurihara 
> wrote:
> > Hi Matteo,
> >
> > ➢ The document doesn't need to be a super-detailed specification, but
> should
> > ➢ explain all the design points, reasons for choosing a determined
> solution,
> > ➢ rejected alternatives and allow for other contributors to
> understand the
> > ➢ feature/change and to contribute as well to it.
> > ➢
> > I agree with creating such design documents in the Wiki.
> >
> > ➢ for large code changes (new features, improvements)
> > ➢
> > Is there specific criteria for determining whether documents are
> necessary or not?
>
> One way to approach is how quickly someone new can understand the
> architecture or a feature, so they can contribute back.
>
> Coming up with a list of such modules/features is not difficult.
>
> Andrews.
>
> >
> > Thanks,
> > Nozomi Kurihara
> >
> > 2017/06/09 5:13 に、"Matteo Merli"  を書き込みました:
> >
> > So far the discussion for large code changes (new features,
> improvements)
> > in the project has happened in a very informal way.
> >
> > As a way to improve community involvement and also to have a better
> > internal
> > documentation I would like to propose to adopt the PIP model that
> many
> > projects
> > follow.
> >
> > Here, PIP would stand for "Pulsar Improvement Proposal" and would
> consist
> > in creating design documents in the Wiki with a sequential id.
> >
> > The document doesn't need to be a super-detailed specification, but
> should
> > explain all the design points, reasons for choosing a determined
> solution,
> > rejected alternatives and allow for other contributors to understand
> the
> > feature/change and to contribute as well to it.
> >
> > The advantage would be to have a preliminary discussion and gather
> feedback
> > on the design itself and also have the proposals there as a
> reference.
> >
> > In some cases the design has been discussed over "issues" or PRs but
> then
> > later it's more difficult to understand the whole picture, since the
> > information is fragmented.
> >
> > Opinions?
> >
> > Matteo
> >
> >
> >
> >
>

Re: Proposal for Pulsar "PIP"

[Matteo Merli]

> The project will need to have a minimal website.

A colleague of mine, a technical writer, is going through the existing
documentation, editing, expanding it and re-organizing it a bit. He's also
setting up the script and tools to generate a static website with links to
each version documentation.

A pull-request will be ready in the next few days. We were also waiting for
the repo to be moved over to the ASF before starting with the website
(because some nomenclature will change, copyright noticies, etc..).


On Mon, Jun 12, 2017 at 2:11 PM Dave Fisher  wrote:

> Hi -
>
> The project will need to have a minimal website. I would be willing to be
> the test case about documentation. If it is enough for me to understand how
> to do development and use along with understanding the architecture then it
> will be good.
>
> I would recommend that a Wiki be used for shared design. There are many
> approaches and some choices for the Wiki. Apache offers Confluence and
> MOIN. Then there is Github. I am familiar with Confluence, but don’t let
> that influence the project’s choice unduly.
>
> Here to help.
>
> Regards,
> Dave
>
> > On Jun 12, 2017, at 12:55 PM, Matteo Merli 
> wrote:
> >
> >> Is there specific criteria for determining whether documents are
> > necessary or not?
> >
> > That's a good question. I don't have a precise answer for it. As Andrews
> > stated, it should
> > allow any contributor to understand the new feature/changes, enough to be
> > able to
> > do a meaningful code review and also serve as design documentation for
> > future
> > contributors.
> >
> > So, if after the discussion, some details are changed, the wiki should be
> > updated to
> > reflect the final state.
> >
> > If one thinks that additional context is not required, he should go ahead
> > with a PR. And
> > later we can discuss whether a design doc for that change would be
> > beneficial or not.
> >
> > For candidates, and to give an example, I was thinking to start with a
> > "Pulsar native proxy"
> > proposal that I'm working on. Also, I think other good candidates would
> be
> > the
> > non-persistent topics from Rajan, or the delivery throttling.
> >
> >
> >
> > On Thu, Jun 8, 2017 at 10:21 PM Sahaya Andrews  >
> > wrote:
> >
> >> On Thu, Jun 8, 2017 at 7:08 PM, Nozomi Kurihara  >
> >> wrote:
> >>> Hi Matteo,
> >>>
> >>> ➢ The document doesn't need to be a super-detailed specification, but
> >> should
> >>> ➢ explain all the design points, reasons for choosing a determined
> >> solution,
> >>> ➢ rejected alternatives and allow for other contributors to
> >> understand the
> >>> ➢ feature/change and to contribute as well to it.
> >>> ➢
> >>> I agree with creating such design documents in the Wiki.
> >>>
> >>> ➢ for large code changes (new features, improvements)
> >>> ➢
> >>> Is there specific criteria for determining whether documents are
> >> necessary or not?
> >>
> >> One way to approach is how quickly someone new can understand the
> >> architecture or a feature, so they can contribute back.
> >>
> >> Coming up with a list of such modules/features is not difficult.
> >>
> >> Andrews.
> >>
> >>>
> >>> Thanks,
> >>> Nozomi Kurihara
> >>>
> >>> 2017/06/09 5:13 に、"Matteo Merli"  を書き込みました:
> >>>
> >>>So far the discussion for large code changes (new features,
> >> improvements)
> >>>in the project has happened in a very informal way.
> >>>
> >>>As a way to improve community involvement and also to have a better
> >>>internal
> >>>documentation I would like to propose to adopt the PIP model that
> >> many
> >>>projects
> >>>follow.
> >>>
> >>>Here, PIP would stand for "Pulsar Improvement Proposal" and would
> >> consist
> >>>in creating design documents in the Wiki with a sequential id.
> >>>
> >>>The document doesn't need to be a super-detailed specification, but
> >> should
> >>>explain all the design points, reasons for choosing a determined
> >> solution,
> >>>rejected alternatives and allow for other contributors to understand
> >> the
> >>>feature/change and to contribute as well to it.
> >>>
> >>>The advantage would be to have a preliminary discussion and gather
> >> feedback
> >>>on the design itself and also have the proposals there as a
> >> reference.
> >>>
> >>>In some cases the design has been discussed over "issues" or PRs but
> >> then
> >>>later it's more difficult to understand the whole picture, since the
> >>>information is fragmented.
> >>>
> >>>Opinions?
> >>>
> >>>Matteo
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Re: Proposal for Pulsar "PIP"

[Matteo Merli]

The documentations sources are in Markdown. From there, the proposed
solution would be to use Jekill to generate HTML, plus a set of other
scripts to automate the documentation for Javadoc, doxygen, REST API, CLI
tools, protobuf definition, etc.

> The Apache website infrastructure is hosted in SVN.

Some time back I was reading that some project are using GIT based
publishing. That would work by committing the "generated" static website
into a "asf-site" branch in the project repository.
I would prefer git to avoid dealing with multiple source control system,
though it would not be a big problem in any case.

https://blogs.apache.org/infra/entry/git_based_websites_available



On Mon, Jun 12, 2017 at 4:30 PM Dave Fisher  wrote:

> The Apache website infrastructure is hosted in SVN. There are certain
> requirements for links.
>
> It would be good to discuss. For example is the content in Markdown, XML,
> HTML or what? It is not a big deal to conform. I have some experience from
> moving OpenOffice.org to Apache.
>
> Regards.
> Dave
>
> > On Jun 12, 2017, at 4:21 PM, Matteo Merli  wrote:
> >
> >> The project will need to have a minimal website.
> >
> > A colleague of mine, a technical writer, is going through the existing
> > documentation, editing, expanding it and re-organizing it a bit. He's
> also
> > setting up the script and tools to generate a static website with links
> to
> > each version documentation.
> >
> > A pull-request will be ready in the next few days. We were also waiting
> for
> > the repo to be moved over to the ASF before starting with the website
> > (because some nomenclature will change, copyright noticies, etc..).
> >
> >
> > On Mon, Jun 12, 2017 at 2:11 PM Dave Fisher 
> wrote:
> >
> >> Hi -
> >>
> >> The project will need to have a minimal website. I would be willing to
> be
> >> the test case about documentation. If it is enough for me to understand
> how
> >> to do development and use along with understanding the architecture
> then it
> >> will be good.
> >>
> >> I would recommend that a Wiki be used for shared design. There are many
> >> approaches and some choices for the Wiki. Apache offers Confluence and
> >> MOIN. Then there is Github. I am familiar with Confluence, but don’t let
> >> that influence the project’s choice unduly.
> >>
> >> Here to help.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Jun 12, 2017, at 12:55 PM, Matteo Merli 
> >> wrote:
> >>>
> >>>> Is there specific criteria for determining whether documents are
> >>> necessary or not?
> >>>
> >>> That's a good question. I don't have a precise answer for it. As
> Andrews
> >>> stated, it should
> >>> allow any contributor to understand the new feature/changes, enough to
> be
> >>> able to
> >>> do a meaningful code review and also serve as design documentation for
> >>> future
> >>> contributors.
> >>>
> >>> So, if after the discussion, some details are changed, the wiki should
> be
> >>> updated to
> >>> reflect the final state.
> >>>
> >>> If one thinks that additional context is not required, he should go
> ahead
> >>> with a PR. And
> >>> later we can discuss whether a design doc for that change would be
> >>> beneficial or not.
> >>>
> >>> For candidates, and to give an example, I was thinking to start with a
> >>> "Pulsar native proxy"
> >>> proposal that I'm working on. Also, I think other good candidates would
> >> be
> >>> the
> >>> non-persistent topics from Rajan, or the delivery throttling.
> >>>
> >>>
> >>>
> >>> On Thu, Jun 8, 2017 at 10:21 PM Sahaya Andrews <
> sahaya.andr...@gmail.com
> >>>
> >>> wrote:
> >>>
> >>>> On Thu, Jun 8, 2017 at 7:08 PM, Nozomi Kurihara <
> nkuri...@yahoo-corp.jp
> >>>
> >>>> wrote:
> >>>>> Hi Matteo,
> >>>>>
> >>>>> ➢ The document doesn't need to be a super-detailed specification, but
> >>>> should
> >>>>> ➢ explain all the design points, reasons for choosing a
> determined
> >>>> solution,
> >>>>> ➢ rejected alternatives and allow for other contributors to
> >>>> understand the
>

Re: Proposal for Pulsar "PIP"

[Matteo Merli]

Thank you for starting the discussion there.

Matteo

On Mon, Jun 12, 2017 at 5:07 PM Dave Fisher  wrote:

> Hi -
>
> > On Jun 12, 2017, at 4:59 PM, Matteo Merli  wrote:
> >
> > The documentations sources are in Markdown. From there, the proposed
> > solution would be to use Jekill to generate HTML, plus a set of other
> > scripts to automate the documentation for Javadoc, doxygen, REST API, CLI
> > tools, protobuf definition, etc.
> >
> >> The Apache website infrastructure is hosted in SVN.
> >
> > Some time back I was reading that some project are using GIT based
> > publishing. That would work by committing the "generated" static website
> > into a "asf-site" branch in the project repository.
> > I would prefer git to avoid dealing with multiple source control system,
> > though it would not be a big problem in any case.
> >
> > https://blogs.apache.org/infra/entry/git_based_websites_available
>
> I’ve included you on an email to us...@infra.apache.org to ask about the
> status. This will naturally lead into discussions about GitHub transfer.
>
> Regards,
> Dave
> >
> >
> >
> > On Mon, Jun 12, 2017 at 4:30 PM Dave Fisher 
> wrote:
> >
> >> The Apache website infrastructure is hosted in SVN. There are certain
> >> requirements for links.
> >>
> >> It would be good to discuss. For example is the content in Markdown,
> XML,
> >> HTML or what? It is not a big deal to conform. I have some experience
> from
> >> moving OpenOffice.org to Apache.
> >>
> >> Regards.
> >> Dave
> >>
> >>> On Jun 12, 2017, at 4:21 PM, Matteo Merli  wrote:
> >>>
> >>>> The project will need to have a minimal website.
> >>>
> >>> A colleague of mine, a technical writer, is going through the existing
> >>> documentation, editing, expanding it and re-organizing it a bit. He's
> >> also
> >>> setting up the script and tools to generate a static website with links
> >> to
> >>> each version documentation.
> >>>
> >>> A pull-request will be ready in the next few days. We were also waiting
> >> for
> >>> the repo to be moved over to the ASF before starting with the website
> >>> (because some nomenclature will change, copyright noticies, etc..).
> >>>
> >>>
> >>> On Mon, Jun 12, 2017 at 2:11 PM Dave Fisher 
> >> wrote:
> >>>
> >>>> Hi -
> >>>>
> >>>> The project will need to have a minimal website. I would be willing to
> >> be
> >>>> the test case about documentation. If it is enough for me to
> understand
> >> how
> >>>> to do development and use along with understanding the architecture
> >> then it
> >>>> will be good.
> >>>>
> >>>> I would recommend that a Wiki be used for shared design. There are
> many
> >>>> approaches and some choices for the Wiki. Apache offers Confluence and
> >>>> MOIN. Then there is Github. I am familiar with Confluence, but don’t
> let
> >>>> that influence the project’s choice unduly.
> >>>>
> >>>> Here to help.
> >>>>
> >>>> Regards,
> >>>> Dave
> >>>>
> >>>>> On Jun 12, 2017, at 12:55 PM, Matteo Merli 
> >>>> wrote:
> >>>>>
> >>>>>> Is there specific criteria for determining whether documents are
> >>>>> necessary or not?
> >>>>>
> >>>>> That's a good question. I don't have a precise answer for it. As
> >> Andrews
> >>>>> stated, it should
> >>>>> allow any contributor to understand the new feature/changes, enough
> to
> >> be
> >>>>> able to
> >>>>> do a meaningful code review and also serve as design documentation
> for
> >>>>> future
> >>>>> contributors.
> >>>>>
> >>>>> So, if after the discussion, some details are changed, the wiki
> should
> >> be
> >>>>> updated to
> >>>>> reflect the final state.
> >>>>>
> >>>>> If one thinks that additional context is not required, he should go
> >> ahead
> >>>>> with a PR. And
> >>>>> later we can discuss whether a design doc for that change would be
&g

Release 1.18

[Matteo Merli]

Joe and team,

how do things look for the release 1.18? Any other outstanding PR needs to
go in?

>From my side, I have a couple of outstanding PRs, though they could be
easily pushed to 1.19.

Matteo

Re: Release 1.18

[Matteo Merli]

I forgot. There are still the 3 bookkeeper changes to be merged.

On Tue, Jun 13, 2017 at 5:07 PM Matteo Merli  wrote:

> Joe and team,
>
> how do things look for the release 1.18? Any other outstanding PR needs to
> go in?
>
> From my side, I have a couple of outstanding PRs, though they could be
> easily pushed to 1.19.
>
> Matteo
>

Re: Release 1.18

[Matteo Merli]

Thanks Nozomi for pointing that out. We have merged it already.

On Tue, Jun 13, 2017 at 11:36 PM Nozomi Kurihara 
wrote:

> Yuki opened a Pull-Request ( https://github.com/yahoo/pulsar/pull/473 )
> which fixes the bug on WebSocket proxy.
> Could you include it in the 1.18 release?
>
> Nozomi Kurihara
>
> 2017/06/14 9:08 に、"Matteo Merli"  を書き込みました:
>
> I forgot. There are still the 3 bookkeeper changes to be merged.
>
> On Tue, Jun 13, 2017 at 5:07 PM Matteo Merli 
> wrote:
>
> > Joe and team,
> >
> > how do things look for the release 1.18? Any other outstanding PR
> needs to
> > go in?
> >
> > From my side, I have a couple of outstanding PRs, though they could
> be
> > easily pushed to 1.19.
> >
> > Matteo
> >
>
>
>

Re: Migrating development to incubator

[Matteo Merli]

That would be transferring the repository organization ownership and
renaming from yahoo/pulsar into apache/incubator-pulsar.

Everything in the repo will be moved. Then we would need to add write
permissions for the committer to write.

On Thu, Jun 15, 2017 at 5:42 PM Dave Fisher  wrote:

> Hi Joe,
>
> I will create an Infra JIRA for this task. Who is the Yahoo contact who
> can grant access?
>
> We want to move source code and issues. Anything else?
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Jun 15, 2017, at 2:32 PM, Joe F  wrote:
> >
> > Mentors,
> >
> > I have got confirmation that  Yahoo can give admin rights to the pulsar
> > repo to someone in ASF infra team. Let me know who, and I will get it
> going
> >
> > Joe
> >
> > On Mon, Jun 12, 2017 at 12:45 PM, Matteo Merli 
> > wrote:
> >
> >> Joe, that is correct, that person would be someone from ASF INFRA team,
> >> once we open a ticket to request it.
> >> Is there any concern from Yahoo perspective to give admin permission on
> the
> >> repo for the migration?
> >>
> >>
> >>
> >> On Mon, Jun 12, 2017 at 11:12 AM Joe Francis  >
> >> wrote:
> >>
> >>> From talking to the relevant folks here at Yahoo, it seems github
> >> requires
> >>> that the person doing the transfer needs to have have admin rights on
> >> both
> >>> the source and destination repos.
> >>> Joe
> >>>
> >>>On Friday, June 9, 2017 1:07 PM, Dave Fisher  >
> >>> wrote:
> >>>
> >>>
> >>> Hi Joe,
> >>>
> >>> I see that the secretary has registered the SGA in the Foundation
> >> archives.
> >>>
> >>> I’ve updated the status page to include this fact along with the list
> of
> >>> initial committers. So far 8 of the 16 initial committers have iCLAs on
> >>> file.
> >>>
> >>> Regards,
> >>> Dave
> >>>
> >>>> On Jun 8, 2017, at 5:56 PM, Joe F  wrote:
> >>>>
> >>>> The SGA has just been submitted.
> >>>>
> >>>> Joe
> >>>>
> >>>> On Wed, Jun 7, 2017 at 6:16 PM, Dave Fisher 
> >>> wrote:
> >>>>
> >>>>> Hi Folks,
> >>>>>
> >>>>> Please respond to emails here and discuss the podling. In order to
> >> move
> >>> to
> >>>>> the incubator we need to start talking.
> >>>>>
> >>>>>> On Jun 5, 2017, at 1:15 PM, Matteo Merli  wrote:
> >>>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I hope everyone already subscribed this mailing list already.
> >>>>>
> >>>>> Several iCLAs were processed by the ASF secretary on Friday. Accounts
> >>>>> should be setup and I will add to the LDAP groups tomorrow.
> >>>>>
> >>>>>>
> >>>>>> I wanted to start the discussion on multiple points for onboarding
> >> into
> >>>>> the
> >>>>>> incubator.
> >>>>>>
> >>>>>> First of all, which tools we want to keep using when moving the
> >>>>> development
> >>>>>> over at the ASF.
> >>>>>>
> >>>>>> In particular, we need to chose about:
> >>>>>> 1. Git repository and committer workflow
> >>>>>> 2. Github issues vs JIRA
> >>>>>
> >>>>> The proposal requested a JIRA which I requested and is now available
> >> at
> >>>>> https://issues.apache.org/jira/browse/PULSAR/
> >>>>>
> >>>>> If the project wishes to switch to Github issues that would be fine.
> >>>>>
> >>>>>> 3. Jenkins vs Travis for CI
> >>>>>>
> >>>>>> The support for using Github tools for Apache project has been a
> >> recent
> >>>>>> addition. Masakazu pointed to me that it was actually discussed at
> >> the
> >>>>>> latest ApacheCon (https://youtu.be/yWurOHvm5WM?t=1078) and that
> >> right
> >>>>> now
> >>>>>> INFRA is supporting projects to use Github as the primary repository
> >>> and
> >>>>>> issues.
> >>

Re: Migrating development to incubator

[Matteo Merli]

> I was thinking this would be a copy rather than a rename.
> If it is just a rename then that could interfere with the stated desire
to do one more release.

The release 1.18 should be out in the next few hours. After that we'll
focus on aligning to the Apache incubator release process.

Renaming the old repository has the advantage of keeping the current issues
/ PR, people watching the repo and getting notifications about changes on
it. It's a pretty smooth transition from that point of view.

To create new tags in the 1.18 release, Yahoo can always fork the
repository back (eg: yahoo/incubator-pulsar) and add tags in there.

On Fri, Jun 16, 2017 at 11:59 AM Dave Fisher  wrote:

> > On Jun 16, 2017, at 9:48 AM, Matteo Merli 
> wrote:
> >
> > That would be transferring the repository organization ownership and
> > renaming from yahoo/pulsar into apache/incubator-pulsar.
>
> I was thinking this would be a copy rather than a rename.
>
> If it is just a rename then that could interfere with the stated desire to
> do one more release.
>
> >
> > Everything in the repo will be moved. Then we would need to add write
> > permissions for the committer to write.
>
> Yes the Infra admin would be granting write permissions to the initial
> committers and mentors.
>
> I think we will all need to associate a GitHub account with our Apache ID
> through id.apache.org.
>
> Regards,
> Dave
>
> >
> > On Thu, Jun 15, 2017 at 5:42 PM Dave Fisher 
> wrote:
> >
> >> Hi Joe,
> >>
> >> I will create an Infra JIRA for this task. Who is the Yahoo contact who
> >> can grant access?
> >>
> >> We want to move source code and issues. Anything else?
> >>
> >> Regards,
> >> Dave
> >>
> >> Sent from my iPhone
> >>
> >>> On Jun 15, 2017, at 2:32 PM, Joe F  wrote:
> >>>
> >>> Mentors,
> >>>
> >>> I have got confirmation that  Yahoo can give admin rights to the pulsar
> >>> repo to someone in ASF infra team. Let me know who, and I will get it
> >> going
> >>>
> >>> Joe
> >>>
> >>> On Mon, Jun 12, 2017 at 12:45 PM, Matteo Merli  >
> >>> wrote:
> >>>
> >>>> Joe, that is correct, that person would be someone from ASF INFRA
> team,
> >>>> once we open a ticket to request it.
> >>>> Is there any concern from Yahoo perspective to give admin permission
> on
> >> the
> >>>> repo for the migration?
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jun 12, 2017 at 11:12 AM Joe Francis
>  >>>
> >>>> wrote:
> >>>>
> >>>>> From talking to the relevant folks here at Yahoo, it seems github
> >>>> requires
> >>>>> that the person doing the transfer needs to have have admin rights on
> >>>> both
> >>>>> the source and destination repos.
> >>>>> Joe
> >>>>>
> >>>>>   On Friday, June 9, 2017 1:07 PM, Dave Fisher <
> dave2w...@comcast.net
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>> Hi Joe,
> >>>>>
> >>>>> I see that the secretary has registered the SGA in the Foundation
> >>>> archives.
> >>>>>
> >>>>> I’ve updated the status page to include this fact along with the list
> >> of
> >>>>> initial committers. So far 8 of the 16 initial committers have iCLAs
> on
> >>>>> file.
> >>>>>
> >>>>> Regards,
> >>>>> Dave
> >>>>>
> >>>>>> On Jun 8, 2017, at 5:56 PM, Joe F  wrote:
> >>>>>>
> >>>>>> The SGA has just been submitted.
> >>>>>>
> >>>>>> Joe
> >>>>>>
> >>>>>> On Wed, Jun 7, 2017 at 6:16 PM, Dave Fisher 
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi Folks,
> >>>>>>>
> >>>>>>> Please respond to emails here and discuss the podling. In order to
> >>>> move
> >>>>> to
> >>>>>>> the incubator we need to start talking.
> >>>>>>>
> >>>>>>>> On Jun 5, 2017, at 1:15 PM, Matteo Merli 
> wrote:
> >>>>>>>

Re: Migrating development to incubator

[Matteo Merli]

Dave,

on our end, the release 1.18 went out on Friday, so we're ready for the
transfer to happen.

Matteo

On Fri, Jun 16, 2017 at 2:00 PM Dave Fisher  wrote:

> The INFRA JIRA issue is submitted:
> https://issues.apache.org/jira/browse/INFRA-14376
>
> Regards,
> Dave
>
> > On Jun 16, 2017, at 12:18 PM, Matteo Merli 
> wrote:
> >
> >> I was thinking this would be a copy rather than a rename.
> >> If it is just a rename then that could interfere with the stated desire
> > to do one more release.
> >
> > The release 1.18 should be out in the next few hours. After that we'll
> > focus on aligning to the Apache incubator release process.
> >
> > Renaming the old repository has the advantage of keeping the current
> issues
> > / PR, people watching the repo and getting notifications about changes on
> > it. It's a pretty smooth transition from that point of view.
> >
> > To create new tags in the 1.18 release, Yahoo can always fork the
> > repository back (eg: yahoo/incubator-pulsar) and add tags in there.
> >
> > On Fri, Jun 16, 2017 at 11:59 AM Dave Fisher 
> wrote:
> >
> >>> On Jun 16, 2017, at 9:48 AM, Matteo Merli 
> >> wrote:
> >>>
> >>> That would be transferring the repository organization ownership and
> >>> renaming from yahoo/pulsar into apache/incubator-pulsar.
> >>
> >> I was thinking this would be a copy rather than a rename.
> >>
> >> If it is just a rename then that could interfere with the stated desire
> to
> >> do one more release.
> >>
> >>>
> >>> Everything in the repo will be moved. Then we would need to add write
> >>> permissions for the committer to write.
> >>
> >> Yes the Infra admin would be granting write permissions to the initial
> >> committers and mentors.
> >>
> >> I think we will all need to associate a GitHub account with our Apache
> ID
> >> through id.apache.org.
> >>
> >> Regards,
> >> Dave
> >>
> >>>
> >>> On Thu, Jun 15, 2017 at 5:42 PM Dave Fisher 
> >> wrote:
> >>>
> >>>> Hi Joe,
> >>>>
> >>>> I will create an Infra JIRA for this task. Who is the Yahoo contact
> who
> >>>> can grant access?
> >>>>
> >>>> We want to move source code and issues. Anything else?
> >>>>
> >>>> Regards,
> >>>> Dave
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On Jun 15, 2017, at 2:32 PM, Joe F  wrote:
> >>>>>
> >>>>> Mentors,
> >>>>>
> >>>>> I have got confirmation that  Yahoo can give admin rights to the
> pulsar
> >>>>> repo to someone in ASF infra team. Let me know who, and I will get it
> >>>> going
> >>>>>
> >>>>> Joe
> >>>>>
> >>>>> On Mon, Jun 12, 2017 at 12:45 PM, Matteo Merli <
> matteo.me...@gmail.com
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> Joe, that is correct, that person would be someone from ASF INFRA
> >> team,
> >>>>>> once we open a ticket to request it.
> >>>>>> Is there any concern from Yahoo perspective to give admin permission
> >> on
> >>>> the
> >>>>>> repo for the migration?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Jun 12, 2017 at 11:12 AM Joe Francis
> >>  >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> From talking to the relevant folks here at Yahoo, it seems github
> >>>>>> requires
> >>>>>>> that the person doing the transfer needs to have have admin rights
> on
> >>>>>> both
> >>>>>>> the source and destination repos.
> >>>>>>> Joe
> >>>>>>>
> >>>>>>>  On Friday, June 9, 2017 1:07 PM, Dave Fisher <
> >> dave2w...@comcast.net
> >>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi Joe,
> >>>>>>>
> >>>>>>> I see that the secretary has registered the SGA in the Foundation
> &

Fwd: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Matteo Merli]

The repo location has now been transferred to
https://github.com/apache/incubator-pulsar

I think the next step would be to do the "reporeq" at
https://reporeq.apache.org
And that should be done by someone in IPMC :)
I am not sure wether it was already done though.

For all the committers: you should go to id.apache.org and enter your
github account there, so that you can link that account to the apache
account.

Matteo


-- Forwarded message -
From: Daniel Takamori (JIRA) 
Date: Wed, Jun 21, 2017 at 2:11 PM
Subject: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration
To: 


Daniel Takamori

*commented* on [image: Github Integration] INFRA-14376


Re: Pulsar GitHub Integration

Imported the repo to Apache and set the hooks, now your PMC can go to
https://reporeq.apache.org to clone the Apache repo and we can move on from
there :)

 Add Comment


This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

Re: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Matteo Merli]

Andrews,

once the setup is finalized, all the committers will have write access if
the github account is linked to the Apache account.



On Wed, Jun 21, 2017 at 3:32 PM Sahaya Andrews 
wrote:

> Hi Daniel,
>
>   Could you add pulsar repo write access to the following GitHub account?
> These folks were part of admin account before moving the repo.
> *apache id - GitHub id *
> andrews  - saandrews
> joef  - joefk
> rdhabalia - rdhabalia
>
> Thanks,
> Andrews.
>
> On Wed, Jun 21, 2017 at 3:08 PM, Sahaya Andrews 
> wrote:
>
> > I have updated my GitHub id in profile.
> >
> > What does the "reporeq" step meant to do?
> >
> > On Wed, Jun 21, 2017 at 2:56 PM, Matteo Merli  wrote:
> >
> >> The repo location has now been transferred to
> >> https://github.com/apache/incubator-pulsar
> >>
> >> I think the next step would be to do the "reporeq" at
> >> https://reporeq.apache.org
> >> And that should be done by someone in IPMC :)
> >> I am not sure wether it was already done though.
> >>
> >> For all the committers: you should go to id.apache.org and enter your
> >> github account there, so that you can link that account to the apache
> >> account.
> >>
> >> Matteo
> >>
> >>
> >> -- Forwarded message -
> >> From: Daniel Takamori (JIRA) 
> >> Date: Wed, Jun 21, 2017 at 2:11 PM
> >> Subject: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration
> >> To: 
> >>
> >>
> >> Daniel Takamori
> >> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=pono>
> >> *commented* on [image: Github Integration] INFRA-14376
> >> <https://issues.apache.org/jira/browse/INFRA-14376>
> >>
> >> Re: Pulsar GitHub Integration
> >> <https://issues.apache.org/jira/browse/INFRA-14376>
> >> Imported the repo to Apache and set the hooks, now your PMC can go to
> >> https://reporeq.apache.org to clone the Apache repo and we can move on
> >> from there :)
> >>
> >> <https://issues.apache.org/jira/browse/INFRA-14376#add-comment> Add
> >> Comment <https://issues.apache.org/jira/browse/INFRA-14376#add-comment>
> >>
> >> This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
> >>
> >
> >
>

Renaming package names and license headers

[Matteo Merli]

I'm going to start to send out PRs to update the license headers and to
rename the Package names.

 I believe that the renaming should not pose a big trouble on the currently
opened PRs, since git is able to follow that without conflicts.

For the license headers, there might be some conflicts marked, but they
would be trivial to fix.

Matteo

Re: Renaming package names and license headers

[Matteo Merli]

Ok, got it :)

I've closed the PR.

For the record, the steps I did to change the headers were:

 1. Modify "src/license-header.txt" with standard ASF header
 2. mvn license:format



Matteo


On Thu, Jun 22, 2017 at 10:51 AM Dave Fisher  wrote:

> Hi Matteo,
>
> No - the license header changes must be done by someone from Yahoo. That's
> ASF policy that the changes are done from the SGA org.
>
> Package names are ok.
>
> Regards,
> Dave
>
> > On Jun 22, 2017, at 10:48 AM, Matteo Merli  wrote:
> >
> > I'm going to start to send out PRs to update the license headers and to
> > rename the Package names.
> >
> > I believe that the renaming should not pose a big trouble on the
> currently
> > opened PRs, since git is able to follow that without conflicts.
> >
> > For the license headers, there might be some conflicts marked, but they
> > would be trivial to fix.
> >
> > Matteo
>
>

Re: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Matteo Merli]

I think the gitbox integration is still not active. If I go on
https://gitbox.apache.org/setup/

I can see that :

According to LDAP, you will have access to the following repositories:

bookkeeper:
No repositories for the bookkeeper project served from gitbox yet...
incubator:
No repositories for the incubator project served from gitbox yet...
pulsar:
No repositories for the pulsar project served from gitbox yet...



On Thu, Jun 22, 2017 at 6:03 PM Rajan Dhabalia 
wrote:

> Hi Dave,
>
> > Everyone who has provided an iCLA and has setup their Apache ID to point
> to their GitHub ID should have Write access as you see.
> Actually, I have verified that my Apache ID is pointing to GitHub-ID
> (username) and I have also submitted the iCLA earlier. I think it's same
> for Andrews as well. So, not sure if write-access is not provided due to
> other incomplete process for which we might have to wait for some more
> time.
>
> Thanks,
> Rajan
>
>
> On Thu, Jun 22, 2017 at 5:49 PM, Dave Fisher 
> wrote:
>
> > Hi Andrews,
> >
> > > On Jun 22, 2017, at 3:40 PM, Sahaya Andrews 
> > wrote:
> > >
> > > When can we expect the setup to be finalized? We are still waiting to
> > > get write access. Some of us who were part of the dev group in the old
> > > repo got write access automatically. But the rest who were part of
> > > admin group did not :)
> >
> > Everyone who has provided an iCLA and has setup their Apache ID to point
> > to their GitHub ID should have Write access as you see.
> >
> > Daniel - would you comment on Admin privilege setup with Gitbox
> > integration. What can the podling expect? How do we control the group? Do
> > we do JIRA tickets? I know the Gitbox integration is new and the
> > documentation that you pointed me to is out of date and not on point with
> > the new.
> >
> > Regards,
> > Dave
> >
> > >
> > > Andrews.
> > >
> > > On Wed, Jun 21, 2017 at 4:36 PM, Matteo Merli 
> wrote:
> > >> Andrews,
> > >>
> > >> once the setup is finalized, all the committers will have write access
> > if
> > >> the github account is linked to the Apache account.
> > >>
> > >>
> > >>
> > >> On Wed, Jun 21, 2017 at 3:32 PM Sahaya Andrews <
> > sahaya.andr...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Daniel,
> > >>>
> > >>>  Could you add pulsar repo write access to the following GitHub
> > account?
> > >>> These folks were part of admin account before moving the repo.
> > >>> *apache id - GitHub id *
> > >>> andrews  - saandrews
> > >>> joef  - joefk
> > >>> rdhabalia - rdhabalia
> > >>>
> > >>> Thanks,
> > >>> Andrews.
> > >>>
> > >>> On Wed, Jun 21, 2017 at 3:08 PM, Sahaya Andrews <
> > sahaya.andr...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I have updated my GitHub id in profile.
> > >>>>
> > >>>> What does the "reporeq" step meant to do?
> > >>>>
> > >>>> On Wed, Jun 21, 2017 at 2:56 PM, Matteo Merli 
> > wrote:
> > >>>>
> > >>>>> The repo location has now been transferred to
> > >>>>> https://github.com/apache/incubator-pulsar
> > >>>>>
> > >>>>> I think the next step would be to do the "reporeq" at
> > >>>>> https://reporeq.apache.org
> > >>>>> And that should be done by someone in IPMC :)
> > >>>>> I am not sure wether it was already done though.
> > >>>>>
> > >>>>> For all the committers: you should go to id.apache.org and enter
> > your
> > >>>>> github account there, so that you can link that account to the
> apache
> > >>>>> account.
> > >>>>>
> > >>>>> Matteo
> > >>>>>
> > >>>>>
> > >>>>> -- Forwarded message -
> > >>>>> From: Daniel Takamori (JIRA) 
> > >>>>> Date: Wed, Jun 21, 2017 at 2:11 PM
> > >>>>> Subject: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration
> > >>>>> To: 
> > >>>>>
> > >>>>>
> > >>>>> Daniel Takamori
> > >>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=pono>
> > >>>>> *commented* on [image: Github Integration] INFRA-14376
> > >>>>> <https://issues.apache.org/jira/browse/INFRA-14376>
> > >>>>>
> > >>>>> Re: Pulsar GitHub Integration
> > >>>>> <https://issues.apache.org/jira/browse/INFRA-14376>
> > >>>>> Imported the repo to Apache and set the hooks, now your PMC can go
> to
> > >>>>> https://reporeq.apache.org to clone the Apache repo and we can
> move
> > on
> > >>>>> from there :)
> > >>>>>
> > >>>>> <https://issues.apache.org/jira/browse/INFRA-14376#add-comment>
> Add
> > >>>>> Comment <https://issues.apache.org/jira/browse/INFRA-14376#add-
> > comment>
> > >>>>>
> > >>>>> This message was sent by Atlassian JIRA
> (v6.4.14#64029-sha1:ae256fe)
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> >
> >
>

Re: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Matteo Merli]

Dave,

I think we need to open an INFRA ticket with the type "GitBox request" to
have that started.

Can you create that? Or can I do that? (I don't know whether that request
needs to come from an IPMC member).

Thanks,
Matteo

On Thu, Jun 22, 2017 at 11:46 PM Sahaya Andrews 
wrote:

> Thank you everyone for taking time to respond. I guess we will wait
> till the gitbox integration to complete.
>
> On Thu, Jun 22, 2017 at 6:06 PM, Matteo Merli  wrote:
> > I think the gitbox integration is still not active. If I go on
> > https://gitbox.apache.org/setup/
> >
> > I can see that :
> >
> > According to LDAP, you will have access to the following repositories:
> >
> > bookkeeper:
> > No repositories for the bookkeeper project served from gitbox yet...
> > incubator:
> > No repositories for the incubator project served from gitbox yet...
> > pulsar:
> > No repositories for the pulsar project served from gitbox yet...
> >
> >
> >
> > On Thu, Jun 22, 2017 at 6:03 PM Rajan Dhabalia 
> > wrote:
> >>
> >> Hi Dave,
> >>
> >> > Everyone who has provided an iCLA and has setup their Apache ID to
> point
> >> to their GitHub ID should have Write access as you see.
> >> Actually, I have verified that my Apache ID is pointing to GitHub-ID
> >> (username) and I have also submitted the iCLA earlier. I think it's same
> >> for Andrews as well. So, not sure if write-access is not provided due to
> >> other incomplete process for which we might have to wait for some more
> >> time.
> >>
> >> Thanks,
> >> Rajan
> >>
> >>
> >> On Thu, Jun 22, 2017 at 5:49 PM, Dave Fisher 
> >> wrote:
> >>
> >> > Hi Andrews,
> >> >
> >> > > On Jun 22, 2017, at 3:40 PM, Sahaya Andrews <
> sahaya.andr...@gmail.com>
> >> > wrote:
> >> > >
> >> > > When can we expect the setup to be finalized? We are still waiting
> to
> >> > > get write access. Some of us who were part of the dev group in the
> old
> >> > > repo got write access automatically. But the rest who were part of
> >> > > admin group did not :)
> >> >
> >> > Everyone who has provided an iCLA and has setup their Apache ID to
> point
> >> > to their GitHub ID should have Write access as you see.
> >> >
> >> > Daniel - would you comment on Admin privilege setup with Gitbox
> >> > integration. What can the podling expect? How do we control the group?
> >> > Do
> >> > we do JIRA tickets? I know the Gitbox integration is new and the
> >> > documentation that you pointed me to is out of date and not on point
> >> > with
> >> > the new.
> >> >
> >> > Regards,
> >> > Dave
> >> >
> >> > >
> >> > > Andrews.
> >> > >
> >> > > On Wed, Jun 21, 2017 at 4:36 PM, Matteo Merli 
> >> > > wrote:
> >> > >> Andrews,
> >> > >>
> >> > >> once the setup is finalized, all the committers will have write
> >> > >> access
> >> > if
> >> > >> the github account is linked to the Apache account.
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Wed, Jun 21, 2017 at 3:32 PM Sahaya Andrews <
> >> > sahaya.andr...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Hi Daniel,
> >> > >>>
> >> > >>>  Could you add pulsar repo write access to the following GitHub
> >> > account?
> >> > >>> These folks were part of admin account before moving the repo.
> >> > >>> *apache id - GitHub id *
> >> > >>> andrews  - saandrews
> >> > >>> joef  - joefk
> >> > >>> rdhabalia - rdhabalia
> >> > >>>
> >> > >>> Thanks,
> >> > >>> Andrews.
> >> > >>>
> >> > >>> On Wed, Jun 21, 2017 at 3:08 PM, Sahaya Andrews <
> >> > sahaya.andr...@gmail.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> I have updated my GitHub id in profile.
> >> > >>>>
> >> > >>>> What does the "reporeq" step meant to do?
> >> > &

Apache rat and license headers

[Matteo Merli]

I have added the apache-rat maven plugin and double checked the license
headers status.

There were a few headers missing here and there (some config files, etc..).
I have added
the exclusions for generated files and for imported source files that have
valid licenses (eg: we have some LZ4 related code which is under BSD
2-clause license and I believe it's safe to import in source distribution,
given that we mention it in the NOTICE file).

Dave, is it fine if the PR is coming from non-Yahoo employee this time?

There is just one point where I am not sure. We have one file that contains
the implementation for CRC32-C in software. This is used as a fallback in
case the processor doesn't support SSE4.2 instructions.
https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/lib/checksum/crc32c_sw.cc

The license is not any "standard" one, though it looks very permissive.
IANAL but as far as I can understand this just requires to maintain the
header notice and not mis-represent the origin of this code.

---
/* crc32c.c -- compute CRC-32C using the Intel crc32 instruction
 * Copyright (C) 2013 Mark Adler
 * Version 1.1  1 Aug 2013  Mark Adler
 */

/*
 This software is provided 'as-is', without any express or implied
 warranty.  In no event will the author be held liable for any damages
 arising from the use of this software.
 Permission is granted to anyone to use this software for any purpose,
 including commercial applications, and to alter it and redistribute it
 freely, subject to the following restrictions:
 1. The origin of this software must not be misrepresented; you must not
 claim that you wrote the original software. If you use this software
 in a product, an acknowledgment in the product documentation would be
 appreciated but is not required.
 2. Altered source versions must be plainly marked as such, and must not be
 misrepresented as being the original software.
 3. This notice may not be removed or altered from any source distribution.
 Mark Adler
 mad...@alumni.caltech.edu
 */
--


What are our options here? Should we ask for advice on the legal ML ?




--
Matteo Merli

Re: Apache rat and license headers

[Matteo Merli]

On Mon, Jun 26, 2017 at 11:49 AM, Dave Fisher  wrote:

> You should open up a LEGAL lira to discuss on the legal-discuss email list.
>
> I think that we will need to put some information in the NOTICE, but some
> will consider the source note as required to be enough.
>
> Another question is if the special license must be in the LICENSE file.
>


Thanks, I'll follow up on this.


--
Matteo Merli

Re: [GitHub] incubator-pulsar issue #523: Max Payload size is validated before compressio...

[Matteo Merli]

> I am curious about SLA vs. message size. If Pulsar has a 5ms SLA is that
going to be feasible if there are very large messages?

Of course it's not feasible :)

The numbers published were related to 1KB entries. Of course there can be a
huge number of different use cases with different trades off. Typically
most users don't care about latency of bigger messages, so we never really
tested it, but at least at lower transfer rate, the latency shouldn't be
too bad even for 1MB messages.



--
Matteo Merli


On Mon, Jun 26, 2017 at 1:42 PM, Sahaya Andrews 
wrote:

> As the message size grows, we won't be able to guarantee 5ms latency
> since the underlying storage itself will take longer to sync the data.
>
> Andrews.
>
> On Mon, Jun 26, 2017 at 1:31 PM, Dave Fisher 
> wrote:
> > Hi -
> >
> > If you are negotiating the buffer size then you should also negotiate if
> this is a compressed or uncompressed size.
> >
> > I am curious about SLA vs. message size. If Pulsar has a 5ms SLA is that
> going to be feasible if there are very large messages?
> >
> > Regards,
> > Dave
> >
> >> On Jun 26, 2017, at 1:26 PM, merlimat  wrote:
> >>
> >> Github user merlimat commented on the issue:
> >>
> >>https://github.com/apache/incubator-pulsar/issues/523
> >>
> >>> If it's configurable then client may start sending msg with
> unreasonable size which will be rejected at broker due to
> frame-size/storage-limitation, So, client will not know up-front.
> >>
> >>Max-frame size could be negotiated between client and broker in
> Connect/Connected handshake
> >>
> >>
> >> ---
> >> If your project is set up for it, you can reply to this email and have
> your
> >> reply appear on GitHub as well. If your project does not have this
> feature
> >> enabled and wishes so, or if the feature is enabled but not working,
> please
> >> contact infrastructure at infrastruct...@apache.org or file a JIRA
> ticket
> >> with INFRA.
> >> ---
> >
>

[DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

I have created the wiki page with a first proposal (as we discussed
earlier).

https://github.com/apache/incubator-pulsar/wiki/PIP-1

Please add feedback in this thread (or if people prefer we can create a
GitHub issue to have the discussion on the proposal).

Other than the proposal itself, it would be good to have some feedback on
the format as well.

Matteo


--
Matteo Merli

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

Good point,

I think we should support SSL, though it might get terminated at the proxy
layer.

Whether the connection between proxy and broker it's encrypted would be
kind of a separated issue, but in any case we cannot just forward the
encrypted connection because that would break the SSL certificate
validation.



--
Matteo Merli


On Thu, Jun 29, 2017 at 2:25 PM, Joe Francis 
wrote:

>  SSL will be supported?
> Joe
>
> On Thursday, June 29, 2017 1:51 PM, Matteo Merli 
> wrote:
>
>
>  I have created the wiki page with a first proposal (as we discussed
> earlier).
>
> https://github.com/apache/incubator-pulsar/wiki/PIP-1
>
> Please add feedback in this thread (or if people prefer we can create a
> GitHub issue to have the discussion on the proposal).
>
> Other than the proposal itself, it would be good to have some feedback on
> the format as well.
>
> Matteo
>
>
> --
> Matteo Merli
> 
>
>
>
>

Fwd: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration

[Matteo Merli]

The GitBox integration is complete.

I can see the "pulsar committers" teams in the Apache org in GitHub.

All the committers should now link their GitHub with the Apache account and
they will be automatically granted write permissions on the repository.
I can see 4 of us so far are enabled.

Visit https://gitbox.apache.org/setup/ to link the accounts.

Notice that it might take a while to reflect the permission (I think
there's a cron job that periodically syncs up the accounts permissions).

--
Matteo Merli


-- Forwarded message --
From: Chris Thistlethwaite (JIRA) 
Date: Thu, Jun 29, 2017 at 1:01 PM
Subject: [jira] [Commented] (INFRA-14376) Pulsar GitHub Integration
To: mme...@apache.org


Chris Thistlethwaite
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cthistle>
*commented* on [image: Github Integration] INFRA-14376
<https://issues.apache.org/jira/browse/INFRA-14376>

Re: Pulsar GitHub Integration
<https://issues.apache.org/jira/browse/INFRA-14376>
GitBox has been enabled

g...@github.com:apache/incubator-pulsar.git

https://gitbox.apache.org/repos/asf?p=incubator-pulsar.git;a=summary
[image: Add Comment]
<https://issues.apache.org/jira/browse/INFRA-14376#add-comment> Add Comment
<https://issues.apache.org/jira/browse/INFRA-14376#add-comment>

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
[image: Atlassian logo]

Re: Avoid duplicate notification

[Matteo Merli]

I think you can setup a filter in gmail and attach a label + skip the inbox
for the github notifications to the mailing list.

Example of gmail filter:

from:(g...@git.apache.org) to:(dev@pulsar.incubator.apache.org,)

Send to Pulsar/Git folder

--
Matteo Merli


On Thu, Jun 29, 2017 at 5:16 PM, Sahaya Andrews  wrote:

> I and probably the rest of us gets email notification from GitHub as
> well as from our dev mailing list for every commit/comment. Is there a
> way to avoid such duplicate and get one notification instead? The
> GitHub notification seems to be more readable.
>
> Andrews.
>

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Thu, Jun 29, 2017 at 4:06 PM, Dave Fisher  wrote:

> Does Apache Zookeeper meet these needs?
>

Dave, do you mean with respect to SSL?

--
Matteo Merli

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Thu, Jun 29, 2017 at 5:18 PM, Maurice Barnum 
wrote:

> this seems like it should be a solved problem.from a quick look, for
> example, it seems like nginx can do this already.
> https://www.nginx.com/resources/admin-guide/tcp-load-balancing/
>

That would only work for the initial topic lookup.

After that, the client discover that topic X is server on broker Y and it
will try to directly connect to it.

In the same way, when you connect the 2nd time, the proxy cannot just be a
simple TCP proxy, because you need to
specify which particular broker you want to connect to. So, I'm my
proposal, this is done in the Connect/Connected
phase, so the proxy knows which target it needs to connect to and after
that it gets out of the way by doing simple
buffer forwarding.

--
Matteo Merli

Re: Podling Report Reminder - July 2017

[Matteo Merli]

I'll try to get a draft ready by tomorrow.

--
Matteo Merli


On Thu, Jun 29, 2017 at 5:51 PM,  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 19 July 2017, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, July 05).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/July2017
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Thu, Jun 29, 2017 at 6:33 PM, Dave Fisher  wrote:

> I mean do you think it would meet the needs of a proxy including SSL?
>
> I'll look into this more as this proxy design intrigues me.
>


So, ZooKeeper is more of a distributed coordination service and it doesn't
really work as a proxy. We use it in Pulsar to coordinate brokers and
storage nodes and to store metadata.
One design trait is that we don't want to expose ZK service to our users,
since it's a very critical piece of the infrastructure (if ZK is down, the
Pulsar cluster cannot operate).

Here there is a high-level diagram that shows where ZK is being used in
Pulsar
https://github.com/apache/incubator-pulsar/blob/master/docs/Architecture.md#architecture

As Maurice commented, there are many ways to to do HTTP or TCP proxy, from
nginx to Apache TrafficServer, but these won't work to proxy to stateful
backend services.
This is a common problem for cloud deployment. For Kafka they have the same
issue and the solution they offer is to use a REST proxy to expose to the
outside world (but that has a huge performance penalty, especially if you
need to guarantee the message ordering).

For the SSL part, unless you have an L4 proxy such as a VIP, the SSL needs
to be terminated at that layer.
I think this fits well anyway for most deployment and has the advantage of
offloading the SSL portion to the proxy compared to the broker.


Matteo


--
Matteo Merli

Re: Avoid duplicate notification

[Matteo Merli]

Andrews, I think one of the the main reasons to email to dev list is to
archive all the conversations and have them searchable.


On Fri, Jun 30, 2017 at 10:53 AM Sahaya Andrews  wrote:

> Thanks Matteo & Masakazu. I have setup the filter, but wondering if we
> can turn off notification from GitHub to dev alias. But other non
> committers might want to get the commit updates.
>
> Andrews.
>
> On Fri, Jun 30, 2017 at 4:29 AM, Masakazu Kitajo 
> wrote:
> > I think there is a way to stop the duplicate emails. I'm also a committer
> > of Apache Traffic Server and the project uses GitHub but I only receive
> > emails from g...@git.apache.org for Pulsar project.
> >
> > Masakazu
> >
> > On Fri, Jun 30, 2017 at 9:19 AM, Matteo Merli 
> > wrote:
> >
> >> I think you can setup a filter in gmail and attach a label + skip the
> inbox
> >> for the github notifications to the mailing list.
> >>
> >> Example of gmail filter:
> >>
> >> from:(g...@git.apache.org) to:(dev@pulsar.incubator.apache.org,)
> >>
> >> Send to Pulsar/Git folder
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
> >> On Thu, Jun 29, 2017 at 5:16 PM, Sahaya Andrews 
> >> wrote:
> >>
> >> > I and probably the rest of us gets email notification from GitHub as
> >> > well as from our dev mailing list for every commit/comment. Is there a
> >> > way to avoid such duplicate and get one notification instead? The
> >> > GitHub notification seems to be more readable.
> >> >
> >> > Andrews.
> >> >
> >>
>

Re: Avoid duplicate notification

[Matteo Merli]

Anyway, I think we're getting double notifications, one from gitbox and the
other from the previous apache git repo notifications.
One is going to dev@pulsar.incubator.apache.org and the other to
d...@pulsar.apache.org
But they're just aliases for the same mailing list :)

I think we should ask INFRA to turn one of them off.



--
Matteo Merli


On Fri, Jun 30, 2017 at 12:12 PM, Matteo Merli 
wrote:

> Andrews, I think one of the the main reasons to email to dev list is to
> archive all the conversations and have them searchable.
>
>
> On Fri, Jun 30, 2017 at 10:53 AM Sahaya Andrews 
> wrote:
>
>> Thanks Matteo & Masakazu. I have setup the filter, but wondering if we
>> can turn off notification from GitHub to dev alias. But other non
>> committers might want to get the commit updates.
>>
>> Andrews.
>>
>> On Fri, Jun 30, 2017 at 4:29 AM, Masakazu Kitajo 
>> wrote:
>> > I think there is a way to stop the duplicate emails. I'm also a
>> committer
>> > of Apache Traffic Server and the project uses GitHub but I only receive
>> > emails from g...@git.apache.org for Pulsar project.
>> >
>> > Masakazu
>> >
>> > On Fri, Jun 30, 2017 at 9:19 AM, Matteo Merli 
>> > wrote:
>> >
>> >> I think you can setup a filter in gmail and attach a label + skip the
>> inbox
>> >> for the github notifications to the mailing list.
>> >>
>> >> Example of gmail filter:
>> >>
>> >> from:(g...@git.apache.org) to:(dev@pulsar.incubator.apache.org,)
>> >>
>> >> Send to Pulsar/Git folder
>> >>
>> >> --
>> >> Matteo Merli
>> >> 
>> >>
>> >> On Thu, Jun 29, 2017 at 5:16 PM, Sahaya Andrews 
>> >> wrote:
>> >>
>> >> > I and probably the rest of us gets email notification from GitHub as
>> >> > well as from our dev mailing list for every commit/comment. Is there
>> a
>> >> > way to avoid such duplicate and get one notification instead? The
>> >> > GitHub notification seems to be more readable.
>> >> >
>> >> > Andrews.
>> >> >
>> >>
>>
>

Re: Dependencies and Licenses

[Matteo Merli]

Yes, I have been following the same LEGAL-303 Jira.

At this point, RocksDB is a required dependency. It would not be hard to
have it optional and by default not included/required and provide
instruction (with licenssing warning) on the website on how to add support
for it.

The difference would be mostly on the performance side.
(One concern there might be on people that download the distribution and do
a performance test with the default configuration and finds "not-stellar"
results, especially in terms of low latency)

I'm also interested in see what will be the solution that other projects
will adopt (like Flink, Samza and Kafka) that depend heavily on RocksDb and
for which there is no current comparable replacement, unfortunately.


So, short answer, we can easily turn it off, but I'd hope to see a common
solution for all the Apache projects on this issue.

Matteo


--
Matteo Merli


On Fri, Jun 30, 2017 at 12:12 PM, Dave Fisher  wrote:

> Hi -
>
> I was exploring the project in GitHub and I noticed a reference to
> RocksDB. As part of an Apache project it is important to make sure that all
> of the required dependencies have licenses that are compatible with the
> Apache License 2.0 and also that those licenses do not in anyway interfere
> with downstream consumer's rights to make use of the project code however
> they wish.
>
> There is a legal affairs committee which handles these questions. Resolved
> licenses are found here [1]. The mailing list is legal-disc...@apache.org.
> Each question is typically handled via JIRA. RocksDB has just been declared
> Category X which means it is not suitable. The discussion occurred here
> [2]. It did included that some projects are using RocksDB and this use may
> be grandfathered.
>
> Is RocksDB required for Pulsar? If so then we need to have a discussion.
>
> Regards,
> Dave
>
> [1] https://www.apache.org/legal/resolved.html
> [2] https://issues.apache.org/jira/browse/LEGAL-303
>
>
>
>

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Fri, Jun 30, 2017 at 2:46 PM, Maurice Barnum 
wrote:

> it's a bit unfortunate the proxy would have to decode enough of the wire
> protocol to find the lookup commands and responses to build the "routing
> map".  i'd still be tempted to do it with nginx and a lua module. if you're
> careful, you can avoid deserializing responses unless you have an
> outstanding lookup request, but you'll still have to keep track of the
> message framing.
>

But the deserialization is only required for topic and partitions metadata
lookups.
In general, and that will be true with the proxy as well, the TCP
connection used for lookup is different from the TCP connections used to
send/receive data. That's because the lookups all happen on the
"serviceUrl" hostname, while the send/receive is done to the direct broker.

With the proxy, on the connection where the actual data is being sent, we
just need to deserialize the initial Connect/Connected commands, and after
that remove the deserialization logic. That makes the proxy very cheap in
terms of memory allocated (GC rate is only 1 object per buffer transferred
each way) and CPU.


--
Matteo Merli

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Fri, Jun 30, 2017 at 12:41 PM, Dave Fisher  wrote:

> So I am clear the problem is having an SSL endpoint that authenticates the
> client and allows the messages to flow through to the correct broker.
>


The main problem this proposal is trying to solve is how to expose Pulsar,
which is a stateful service [1], through a stateless frontend.

The reason to have a stateless frontend, is that it's much easier to give
access to this service from outside the current cluster/datacenter. If you
have a stateless service, you can easily use multiple strategies (VIP, DNS,
) to expose multiple instances that are composing the service and you
don't need to connect to specific nodes. Any node, as routed by the
VIP/DNS/.. mechanism will work.

One example of this is when deploying in a cloud environment, especially
with some container orchestration mechanism like kubernetes. If you want to
connect and publish messages from outside the Kubernetes cluster (or from
outside the cloud region alltogether), having the stateless service,
exposed to a cloud specific load balancer, makes it a lot easier to deploy
and to control the access from the outside world.

For the SSL part, right now in Pulsar you might want to use TLS/SSL for few
reasons:
 1. Transport encryption
 2. Broker authentication (validate the broker we're talking to is
legitimate)
 3. Client authentication (extract the client principal to be used for
authorization)

All these 3 should continue to work even when a proxy is introduced. In
particular, for the client authentication, the proxy is responsible of
validating the authentication and then carry the client principal over to
the broker.


>> Can I assume that if the broker disconnects that the requirement is for
the client to reconnect and then at that point get the new broker?

Correct, that's the behavior of the client library and it will not change.
Whenever the connection breaks, the client library internally tries to
re-do the topic lookup (because now it might have moved to a different
broker) and reconnect, with exponential backoff, until it succeeds.


[1]  brokers don't keep state on disk, but still a topic is only assign to
a single broker at a give point in time.


--
Matteo Merli

Re: Podling report draft - please review

[Matteo Merli]

On Fri, Jun 30, 2017 at 4:43 PM, Joe F  wrote:

>   [ ] Working towards first release
>

I would suggest to check this mark, since we're indeed working on this. Not
there yet, but working towards it.

--
Matteo Merli

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

I have updated the Wiki page with a section about TLS :
https://github.com/apache/incubator-pulsar/wiki/PIP-1#tls-encryption-and-authentication

Also, I just posted a PR with the implementation:

https://github.com/apache/incubator-pulsar/pull/548



--
Matteo Merli


On Fri, Jun 30, 2017 at 5:09 PM, Dave Fisher  wrote:

>
> > On Jun 30, 2017, at 4:39 PM, Matteo Merli 
> wrote:
> >
> > On Fri, Jun 30, 2017 at 12:41 PM, Dave Fisher 
> wrote:
> >
> >> So I am clear the problem is having an SSL endpoint that authenticates
> the
> >> client and allows the messages to flow through to the correct broker.
> >>
> >
> >
> > The main problem this proposal is trying to solve is how to expose
> Pulsar,
> > which is a stateful service [1], through a stateless frontend.
>
> One thought is make the proxy into a WebSocket frontend but then pass
> through to the broker using the TCP protocol.
>
> WebSocket would fit a pattern that everyone is used to scaling. It could
> be setup through Tomcat with connections through a VIP.
>
> Think about it.
>
> > The reason to have a stateless frontend, is that it's much easier to give
> > access to this service from outside the current cluster/datacenter. If
> you
> > have a stateless service, you can easily use multiple strategies (VIP,
> DNS,
> > ) to expose multiple instances that are composing the service and you
> > don't need to connect to specific nodes. Any node, as routed by the
> > VIP/DNS/.. mechanism will work.
> >
> > One example of this is when deploying in a cloud environment, especially
> > with some container orchestration mechanism like kubernetes. If you want
> to
> > connect and publish messages from outside the Kubernetes cluster (or from
> > outside the cloud region alltogether), having the stateless service,
> > exposed to a cloud specific load balancer, makes it a lot easier to
> deploy
> > and to control the access from the outside world.
>
> Yes and this could work with WebSocket.
>
> >
> > For the SSL part, right now in Pulsar you might want to use TLS/SSL for
> few
> > reasons:
> > 1. Transport encryption
> > 2. Broker authentication (validate the broker we're talking to is
> > legitimate)
> > 3. Client authentication (extract the client principal to be used for
> > authorization)
> >
> > All these 3 should continue to work even when a proxy is introduced. In
> > particular, for the client authentication, the proxy is responsible of
> > validating the authentication and then carry the client principal over to
> > the broker.
> >
> >
> >>> Can I assume that if the broker disconnects that the requirement is for
> > the client to reconnect and then at that point get the new broker?
> >
> > Correct, that's the behavior of the client library and it will not
> change.
> > Whenever the connection breaks, the client library internally tries to
> > re-do the topic lookup (because now it might have moved to a different
> > broker) and reconnect, with exponential backoff, until it succeeds.
> >
> >
> > [1]  brokers don't keep state on disk, but still a topic is only assign
> to
> > a single broker at a give point in time.
> >
>
> Great.
>
> Regards,
> Dave
>
> >
> > --
> > Matteo Merli
> > 
>
>

Re: [DISCUSS] PIP-1 Pulsar proxy component

[Matteo Merli]

On Fri, Jun 30, 2017 at 5:09 PM, Dave Fisher  wrote:

> One thought is make the proxy into a WebSocket frontend but then pass
> through to the broker using the TCP protocol.
>
> WebSocket would fit a pattern that everyone is used to scaling. It could
> be setup through Tomcat with connections through a VIP.
>

The problem is still related to the stateful nature of the brokers (a topic
is only served at a particular point in time), so the VIP cannot just route
to any random broker.
So, even if we change the client to always connect through the WebSocket
service, it would still need some way to indicate which broker it needs to
connect to.

Side note: Pulsar already provides a WebSocket service, and that is used to
expose an "easier" interface that you can use from whatever language. The
problem is that it's not meant to be super-performant (the exchanges
between client and server happen in JSON and the payloads are serialized in
base64..


Matteo


--
Matteo Merli

Re: [GitHub] rdhabalia commented on a change in pull request #514: Static link dependency libraries for C++ client lib and Python wrapper

[Matteo Merli]

On Wed, Jul 5, 2017 at 1:56 PM,  wrote:

> rdhabalia commented on a change in pull request #514: Static link
> dependency libraries for C++ client lib and Python wrapper
> URL: https://github.com/apache/incubator-pulsar/pull/514#discussi
> on_r125755406
>
> +find_library(LOG4CXX_LIBRARY_PATH NAMES liblog4cxx.a)
>
>  Review comment:
>`liblog4cxx.a` is this correct?
>


( I don't know why this comment doesn't appear on the github PR page)

liblog4cxx.a is to force cmake to link against the static library (that
allows to remove the dependency on log4cxx, since it's like bundling the
library inside libpulsar.so)

Re: [GitHub] rdhabalia commented on a change in pull request #514: Static link dependency libraries for C++ client lib and Python wrapper

[Matteo Merli]

Hi Dave,

So far, we have never released a binary package for C++ client library, and
I still think we shouldn't, because it gets really complicated to have
binaries working for multiple OSs.

We don't have any binaries in the repo and this is more about the flags we
use to compile the C++ client lib.

If we link the pulsar shared library (libpulsar.so) against static
libraries (eg: liblog4cxx.a instead of liblog4cxx.so), that the
libpulsar.so will be self-contained. At that point, someone can just
package libpulsar.so without having to worry about the dependencies. This
is especially important for Python wrapper, since by statically linking we
can just create a Python wheel file that contains the wrapper and the c++
client lib with all the dependencies in a single shared package.

After the discussion with Andrews, I'll make the change in the PR so
enable/disable the static linking based on a flag in the CMake script.

Matteo

--
Matteo Merli


On Thu, Jul 6, 2017 at 8:03 AM, Dave Fisher  wrote:

> Hi -
>
> Is liblog4cxx.a a binary in the repository? What is the provenance of this
> file?
>
> Regards,
> Dave
>
> > On Jul 5, 2017, at 10:03 PM, Matteo Merli 
> wrote:
> >
> > On Wed, Jul 5, 2017 at 1:56 PM,  wrote:
> >
> >> rdhabalia commented on a change in pull request #514: Static link
> >> dependency libraries for C++ client lib and Python wrapper
> >> URL: https://github.com/apache/incubator-pulsar/pull/514#discussi
> >> on_r125755406
> >>
> >> +find_library(LOG4CXX_LIBRARY_PATH NAMES liblog4cxx.a)
> >>
> >> Review comment:
> >>   `liblog4cxx.a` is this correct?
> >>
> >
> >
> > ( I don't know why this comment doesn't appear on the github PR page)
> >
> > liblog4cxx.a is to force cmake to link against the static library (that
> > allows to remove the dependency on log4cxx, since it's like bundling the
> > library inside libpulsar.so)
>
>

Re: Apache rat and license headers

[Matteo Merli]

I was about to ask on the Legal-discuss forum, but then I have looked a bit
more and verified that the license for the CRC32-C code is the ZLib
license, which according to https://www.apache.org/legal/resolved.html
belongs to the Category-A and can be included in the source distribution.

I think we should just need to list it in the NOTICE file and that should
be all.

--
Matteo Merli


On Mon, Jun 26, 2017 at 12:14 PM, Matteo Merli 
wrote:

> On Mon, Jun 26, 2017 at 11:49 AM, Dave Fisher 
> wrote:
>
>> You should open up a LEGAL lira to discuss on the legal-discuss email
>> list.
>>
>> I think that we will need to put some information in the NOTICE, but some
>> will consider the source note as required to be enough.
>>
>> Another question is if the special license must be in the LICENSE file.
>>
>
>
> Thanks, I'll follow up on this.
>
>
> --
> Matteo Merli
> 
>

Re: Apache rat and license headers

[Matteo Merli]

Ok, Thanks for the clarification.

I still need to read thoroughly the release process documentation. Then
I'll try to give it a shot at fixing the LICENSE and NOTICE files.

--
Matteo Merli


On Mon, Jul 10, 2017 at 2:28 PM, Dave Fisher  wrote:

> Hi Matteo,
>
> The ZLIB license needs to go into the LICENSE file. Only put something
> into NOTICE for required copyright and/or other attribution.
>
> Thanks,
> Dave
>
> Sent from my iPhone
>
> > On Jul 10, 2017, at 2:19 PM, Matteo Merli 
> wrote:
> >
> > I was about to ask on the Legal-discuss forum, but then I have looked a
> bit
> > more and verified that the license for the CRC32-C code is the ZLib
> > license, which according to https://www.apache.org/legal/resolved.html
> > belongs to the Category-A and can be included in the source distribution.
> >
> > I think we should just need to list it in the NOTICE file and that should
> > be all.
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Jun 26, 2017 at 12:14 PM, Matteo Merli 
> > wrote:
> >
> >> On Mon, Jun 26, 2017 at 11:49 AM, Dave Fisher 
> >> wrote:
> >>
> >>> You should open up a LEGAL lira to discuss on the legal-discuss email
> >>> list.
> >>>
> >>> I think that we will need to put some information in the NOTICE, but
> some
> >>> will consider the source note as required to be enough.
> >>>
> >>> Another question is if the special license must be in the LICENSE file.
> >>>
> >>
> >>
> >> Thanks, I'll follow up on this.
> >>
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
>
>

Re: Jenkins build is back to stable : pulsar-master » Pulsar Broker #5

[Matteo Merli]

I have created this Jenkins job at
https://builds.apache.org/job/pulsar-master/

So far it's for testing purposes. It's only building the Java portion, we
need a way to build the C++ and python part as well.

--
Matteo Merli


On Tue, Jul 11, 2017 at 1:04 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <https://builds.apache.org/job/pulsar-master/org.apache.
> pulsar$pulsar-broker/5/display/redirect>
>
>

Re: Apache rat and license headers

[Matteo Merli]

Ok, I've been reading a bit of material, including :

http://incubator.apache.org/guides/releasemanagement.html
http://www.apache.org/dev/licensing-howto.html


My current understanding is that we need to attach 2 different LICENSE and
NOTICE files to the source and binary artifacts that will be included in
the release.

So far, we have been releasing 2 artifacts :
 1. pulsar-x.y-src.tar.gz --> All the sources (Java, C++, Python)
 2. pulsar-x.y-bin.tar.gz --> Java binaries (No C++ and Python client
libraries binaries)

In addition, the JAR were being pushed to Sonatype and synced with Maven
Central.


The src and bin packages differs in what it is included :
 * SRC --> Pulsar sources, including bundling a few source files with
compatible licenses
 * BIN --> Compiled Pulsar Jars and scripts, plus all runtime dependencies


For the SRC artifact:
 * LICENSE: include ASF license + license of additional bundled sources
 * NOTICE: Standard ASF notice (no additional notices for the sources we
are bundling)

For the BIN artifact:
 * LICENSE:
  - Include ASF License
  - Include Licenses for all bundled dependency jars, adding them in a
folder like:
licences/LICENSE-HdrHistogram.txt
and then mentioning in LICENSE file:
Eg: "HdrHistogram -- HdrHistogram-*.jar -- BSD 2-Clause License"
  - Include licenses

 * NOTICE:
 - Include Pulsar ASF copyright
 - Include copyright lines for all the bundled dependencies
 - Include additional content from NOTICE files from the bundled
dependencies

In both artifacts, include a DISCLAIMER file with the ASF incubator
disclaimer text.


Is that a fair understanding or am I missing something?

Matteo


--
Matteo Merli


On Mon, Jul 10, 2017 at 3:00 PM, Matteo Merli  wrote:

> Ok, Thanks for the clarification.
>
> I still need to read thoroughly the release process documentation. Then
> I'll try to give it a shot at fixing the LICENSE and NOTICE files.
>
> --
> Matteo Merli
> 
>
> On Mon, Jul 10, 2017 at 2:28 PM, Dave Fisher 
> wrote:
>
>> Hi Matteo,
>>
>> The ZLIB license needs to go into the LICENSE file. Only put something
>> into NOTICE for required copyright and/or other attribution.
>>
>> Thanks,
>> Dave
>>
>> Sent from my iPhone
>>
>> > On Jul 10, 2017, at 2:19 PM, Matteo Merli 
>> wrote:
>> >
>> > I was about to ask on the Legal-discuss forum, but then I have looked a
>> bit
>> > more and verified that the license for the CRC32-C code is the ZLib
>> > license, which according to https://www.apache.org/legal/resolved.html
>> > belongs to the Category-A and can be included in the source
>> distribution.
>> >
>> > I think we should just need to list it in the NOTICE file and that
>> should
>> > be all.
>> >
>> > --
>> > Matteo Merli
>> > 
>> >
>> > On Mon, Jun 26, 2017 at 12:14 PM, Matteo Merli 
>> > wrote:
>> >
>> >> On Mon, Jun 26, 2017 at 11:49 AM, Dave Fisher 
>> >> wrote:
>> >>
>> >>> You should open up a LEGAL lira to discuss on the legal-discuss email
>> >>> list.
>> >>>
>> >>> I think that we will need to put some information in the NOTICE, but
>> some
>> >>> will consider the source note as required to be enough.
>> >>>
>> >>> Another question is if the special license must be in the LICENSE
>> file.
>> >>>
>> >>
>> >>
>> >> Thanks, I'll follow up on this.
>> >>
>> >>
>> >> --
>> >> Matteo Merli
>> >> 
>> >>
>>
>>
>

Re: Apache rat and license headers

[Matteo Merli]

Yes, we need to bundle them in order to have a downloadable artifact that
you can untar and start the service.
Otherwise there's no way to have a binary distribution.

I'm going through the list of the flattened dependencies (Kind of the same
exercise I did to compile the list for the incubator proposal).

--
Matteo Merli


On Wed, Jul 12, 2017 at 10:53 AM, Joe F  wrote:

> We will have to recursively analyze dependencies. Do we really have to
> bundle the runtime dependency bits in the bin distribution?
>
> (quoting from http://www.apache.org/dev/licensing-howto.html)
>
> > a number of other "permissive" licenses which are approved
> <http://www.apache.org/legal/resolved.html#category-a> for use by the ASF
> Legal Affairs Committee. Some of these may require additions to NOTICE --
> if in doubt, ask
> <http://www.apache.org/legal/resolved.html#asking-questions>.
>
> >If the dependency supplies a NOTICE file, its contents must be analyzed
> and the relevant portions bubbled up into the top-level NOTICE file.
>
> >Dependencies of dependencies (including so-called "transitive
> dependencies") are no different from first-order dependencies for the
> purposes of assembling LICENSE and NOTICE: LICENSE and NOTICE need only be
> modified to accommodate them *if and only if their bits are bundled*.
>
> Joe
>
> On Wed, Jul 12, 2017 at 10:39 AM, Matteo Merli  wrote:
>
> > Ok, I've been reading a bit of material, including :
> >
> > http://incubator.apache.org/guides/releasemanagement.html
> > http://www.apache.org/dev/licensing-howto.html
> >
> >
> > My current understanding is that we need to attach 2 different LICENSE
> and
> > NOTICE files to the source and binary artifacts that will be included in
> > the release.
> >
> > So far, we have been releasing 2 artifacts :
> >  1. pulsar-x.y-src.tar.gz --> All the sources (Java, C++, Python)
> >  2. pulsar-x.y-bin.tar.gz --> Java binaries (No C++ and Python client
> > libraries binaries)
> >
> > In addition, the JAR were being pushed to Sonatype and synced with Maven
> > Central.
> >
> >
> > The src and bin packages differs in what it is included :
> >  * SRC --> Pulsar sources, including bundling a few source files with
> > compatible licenses
> >  * BIN --> Compiled Pulsar Jars and scripts, plus all runtime
> dependencies
> >
> >
> > For the SRC artifact:
> >  * LICENSE: include ASF license + license of additional bundled sources
> >  * NOTICE: Standard ASF notice (no additional notices for the sources we
> > are bundling)
> >
> > For the BIN artifact:
> >  * LICENSE:
> >   - Include ASF License
> >   - Include Licenses for all bundled dependency jars, adding them in
> a
> > folder like:
> > licences/LICENSE-HdrHistogram.txt
> > and then mentioning in LICENSE file:
> > Eg: "HdrHistogram -- HdrHistogram-*.jar -- BSD 2-Clause License"
> >   - Include licenses
> >
> >  * NOTICE:
> >  - Include Pulsar ASF copyright
> >  - Include copyright lines for all the bundled dependencies
> >  - Include additional content from NOTICE files from the bundled
> > dependencies
> >
> > In both artifacts, include a DISCLAIMER file with the ASF incubator
> > disclaimer text.
> >
> >
> > Is that a fair understanding or am I missing something?
> >
> > Matteo
> >
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Jul 10, 2017 at 3:00 PM, Matteo Merli  wrote:
> >
> > > Ok, Thanks for the clarification.
> > >
> > > I still need to read thoroughly the release process documentation. Then
> > > I'll try to give it a shot at fixing the LICENSE and NOTICE files.
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Mon, Jul 10, 2017 at 2:28 PM, Dave Fisher 
> > > wrote:
> > >
> > >> Hi Matteo,
> > >>
> > >> The ZLIB license needs to go into the LICENSE file. Only put something
> > >> into NOTICE for required copyright and/or other attribution.
> > >>
> > >> Thanks,
> > >> Dave
> > >>
> > >> Sent from my iPhone
> > >>
> > >> > On Jul 10, 2017, at 2:19 PM, Matteo Merli 
> > >> wrote:
> > >> >
> > >> > I was about to ask on the Legal-discuss forum, but then I have
> looked
> > a
> > >> bit
> > >> > more and verified tha

Re: Apache rat and license headers

[Matteo Merli]

On Wed, Jul 12, 2017 at 11:30 AM, Dave Fisher  wrote:

> Instead of going through Sonatype we will need to go through the Apache
> Distribution and release on Apache infrastructure. These can then be pushed
> out through Maven Central and Sonatype.
>
> Other of the other mentors may be a better help on this detail or you kn
> ow how to HipChat with Infrastructure and get guidance.


Oh, sure. I was describing the artifacts we were releasing from the
yahoo/pulsar project. :)

Before we discuss the exact packaging and whether or not it is within
> policy to have a single package for the convenience binary we need to first
> analyze the dependencies. If a dependency is optional and has a class X
> license then we can provide instructions and a script for the user to
> retrieve.
>
> We cannot include Category X in required dependencies.
>

Sure. I'm right now going through the list of dependencies and marking all
the licenses so that we know where we are.

I think the only Cat-X is RocksDB and of course we'll have to deal with it
before we can be ready for a release.

We should open a separate thread to discuss the various options that we can
take for that (eg: using BK without DbLedgerStorage by default, provide
LevelDb default implementation, ... )

Re: Apache rat and license headers

[Matteo Merli]

>
>
>> We cannot include Category X in required dependencies.
>
>
The other thing about RocksDb in Cat-X is that I've been monitoring the dev
lists of TLPs which are major users of RocksDb (Kafka, Samza, Flink) and
couldn't find any discussion in how they're going to handle it, and in the
meantime they're creating new releases :).

I haven't seen any update on the grandfathering police on LEGAL-303, just
recommendation that projects should "move hastily in removing RocksDb"

Re: [GitHub] rdhabalia commented on a change in pull request #425: Add GrowablePriorityLongPairQueue and tests

[Matteo Merli]

On Wed, Jul 12, 2017 at 12:28 PM, Dave Fisher  wrote:

> Hi -
>
> I have two questions here:
>
> (1) Is the project switching now to the com.apache.pulsar?
>

This Pull Request was created before the repository was switched over to
Apache and the content of the PR was not updated since. It will need to get
fixed before merging (actually I can see that Rajan already fixed it).


>
> (2) Are we going to need to revisit the license conversion eliminating the
> Yahoo copyright?
>
> What’s the plan?
>

The build is already validating that all files are carrying the ASF header
(unless explicit exclusions for generated files or bundled sources). If
someone submits a PR missing the headers, the build will fail.

As of now, there are no Yahoo copyrighted files merged in "master" branch.

Re: Apache rat and license headers

[Matteo Merli]

I have created a PR with the changes described earlier, leaving for now
RocksDB in there until we decide how to fix it.

https://github.com/apache/incubator-pulsar/pull/563

I also have uploaded an example of the generated artifacts, for easier
review of the final
result, at :
https://www.dropbox.com/sh/6oevjcdc94sx2tq/AACLOwP4OEgfY-eG_n2_lNNMa?dl=0


Dave, please take a look when you have time.

Thank,
Matteo

--
Matteo Merli


On Wed, Jul 12, 2017 at 12:22 PM, Dave Fisher  wrote:

>
> > On Jul 12, 2017, at 12:06 PM, Matteo Merli 
> wrote:
> >
> > On Wed, Jul 12, 2017 at 11:30 AM, Dave Fisher 
> wrote:
> >
> >> Instead of going through Sonatype we will need to go through the Apache
> >> Distribution and release on Apache infrastructure. These can then be
> pushed
> >> out through Maven Central and Sonatype.
> >>
> >> Other of the other mentors may be a better help on this detail or you kn
> >> ow how to HipChat with Infrastructure and get guidance.
> >
> >
> > Oh, sure. I was describing the artifacts we were releasing from the
> > yahoo/pulsar project. :)
> >
> > Before we discuss the exact packaging and whether or not it is within
> >> policy to have a single package for the convenience binary we need to
> first
> >> analyze the dependencies. If a dependency is optional and has a class X
> >> license then we can provide instructions and a script for the user to
> >> retrieve.
> >>
> >> We cannot include Category X in required dependencies.
> >>
> >
> > Sure. I'm right now going through the list of dependencies and marking
> all
> > the licenses so that we know where we are.
> >
> > I think the only Cat-X is RocksDB and of course we'll have to deal with
> it
> > before we can be ready for a release.
> >
> > We should open a separate thread to discuss the various options that we
> can
> > take for that (eg: using BK without DbLedgerStorage by default, provide
> > LevelDb default implementation, … )
>
> There will guidance coming soon from the VP, Legal Affairs regarding this
> license since many projects are using RocksDB.
>
> We can discuss as it comes.
>
> Regards,
> Dave
>
>

[PROPOSAL] Created a Slack channel for Pulsar

[Matteo Merli]

What do you all think about creating a Pulsar dedicated Slack channel for
quick and informal discussion between developers and to eventually better
support users and contributors ramping up on the code base?

I know that a handful of Apache projects are doing that already, and there
are ways to have a bot that sends a daily digest of the conversation to the
mailing lists.

The ASF has already a Slack org, but it's restricted to committers only. I
think it would be beneficial to have it with open signup so that external
people can join as well.

Thoughts?

Matteo

--
Matteo Merli

Re: [PROPOSAL] Created a Slack channel for Pulsar

[Matteo Merli]

Well, the subject of the thread should have started with "Creating..." :)

On Wed, Jul 12, 2017 at 6:57 PM Yuki Shiga  wrote:

> I think it would be nice!
> :+1:
>
> Best,
> --
> Yuki
> Email: yush...@yahoo-corp.jp
>
> 2017/07/13 10:37, "Matteo Merli"  wrote:
>
> What do you all think about creating a Pulsar dedicated Slack channel
> for
> quick and informal discussion between developers and to eventually
> better
> support users and contributors ramping up on the code base?
>
> I know that a handful of Apache projects are doing that already, and
> there
> are ways to have a bot that sends a daily digest of the conversation
> to the
> mailing lists.
>
> The ASF has already a Slack org, but it's restricted to committers
> only. I
> think it would be beneficial to have it with open signup so that
> external
> people can join as well.
>
> Thoughts?
>
> Matteo
>
> --
> Matteo Merli
> 
>
>
>

Re: [PROPOSAL] Created a Slack channel for Pulsar

[Matteo Merli]

I have created the Slack instance at

https://apache-pulsar.slack.com

The admin is set to this mailing-list, and here is where the digest of the
conversations will be posted daily.

People can visit this page to self-invite:

https://apache-pulsar.herokuapp.com/

We'll put it as well on the website, so that users will be able to join as
well.

Matteo

--
Matteo Merli


On Thu, Jul 13, 2017 at 5:13 AM, Sebastián Schepens <
sebastian.schep...@mercadolibre.com.invalid> wrote:

> +1
> Sure
>
> On Thu, Jul 13, 2017, 05:18 Hiroyuki Sakai  wrote:
>
> > +1
> > me,too.
> >
> >
> > on 2017/07/13 10:37, "Matteo Merli"  wrote:
> >
> > What do you all think about creating a Pulsar dedicated Slack channel
> > for
> > quick and informal discussion between developers and to eventually
> > better
> > support users and contributors ramping up on the code base?
> >
> > I know that a handful of Apache projects are doing that already, and
> > there
> > are ways to have a bot that sends a daily digest of the conversation
> > to the
> > mailing lists.
> >
> > The ASF has already a Slack org, but it's restricted to committers
> > only. I
> > think it would be beneficial to have it with open signup so that
> > external
> > people can join as well.
> >
> > Thoughts?
> >
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
> >
> >
>

Re: How should be the rule about merging?

[Matteo Merli]

I know some projects have explicit ByLaws statements with all the rules for
the project decisions and day-to-day operations.

Eg:  http://bookkeeper.apache.org/bylaws.html

I don't know when we should start thinking about formalizing that.

For commits, I would follow a similar rule as specified in BookKeeper:

 * Review by 2 committers
 * If the contributor is a committer, then just one additional reviewer
 * For non-trivial changes (one-liners, quick test fixes), leave the PR
open for at least 24h to give chance to anyone to weight in.

Once the conditions are met, any committer is allowed to "push the button"
and merge the PR.

I also would encourage anyone, committers in particulars, but also
non-committers, to review all the PRs. The more the people, the more we can
share the workload. Even if someone is not familiar with the particular
code that is being modified, it provides several benefits:

 * The reviewer gets more acquainted with the code base
 * If the change is cryptical, it's a good way to force to add more
comments, documentation or context in the PR/commit log message
 * The more the eyes looking at the change, more chances to figure out bugs.





--
Matteo Merli


On Thu, Jul 13, 2017 at 2:04 PM, Sahaya Andrews  wrote:

> I would prefer option 1, to review first and then commit. I think it
> would be useful to have 2 reviews as well before commit.
>
> Andrews
>
> On Thu, Jul 13, 2017 at 11:21 AM, Dave Fisher 
> wrote:
> > Hi Yuki,
> >
> > There are two approaches to code review that the project can choose.
> >
> > (1) Review Then Commit (RTC) - you are proposing a version of that
> policy.
> >
> > (2) Commit Then Review (CTR) - in this policy anyone can do the merge
> without review. It would then be an option to request a review if the
> committer felt it was a large change and they wanted others to comment.
> >
> > Thanks for bringing up this topic.
> >
> > Regards,
> > Dave
> >
> >> On Jul 13, 2017, at 1:17 AM, Yuki Shiga  wrote:
> >>
> >> I think we don’t have any rules about merging of PR and we should
> decide it.
> >> I consider we can merge  PR when 2 reviewers approve the changes and
> Travis CI build passes.
> >>
> >> What do you think ?
> >>
> >> Best,
> >> --
> >> Yuki
> >> Email: yush...@yahoo-corp.jp
> >
>

[DISCUSS] Make RocksDb dependency optional

[Matteo Merli]

As we have already talked about, the RocksDb library has been put in the
Category-X (Forbidden) by the ASF, due to licensing problems.

With my very limited understanding of the issue, it seems the problem
resides in the patent clauses that's added the license.

https://www.apache.org/legal/resolved.html#category-x

Rocks DB License

The Rocks DB license includes a specification of a PATENTS file that passes
along risk to downstream consumers of our software imbalanced in favor of
the licensor, not the licensee, thereby violating our Apache legal policy
of being a universal donor <https://s.apache.org/4Uzg>. The terms of
ROCKSDB license are not a subset of those found in the ALv2, and they
cannot be sublicensed as ALv2.


Pulsar is using RocksDb since that is being pulled in from the Yahoo branch
of BookKeeper, which includes an alternative LedgerStorage implementation
that stores the indexes for the entries in a RocksDB databases instead of
using individual index files as it's default for BookKeeper. That is
important to store many tens of thousands of ledgers per single Bookie node.

To be able to release "Apache Pulsar" we need to fix the dependency
problem. These are the options that I have been thinking of (feel free to
propose alternatives! ):

 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)

 2. Provide an additional key-value storage implementation for the
DbLedgerStorage (eg: LevelDb) and make that the default

 3. Wait until there's more clarity on the subject (that would mean to
delay the Pulsar release)

As I see it, I prefer option #1 for now, because I don't see any comparable
replacement for RocksDB. LevelDb works fine for small workloads but show
several problems at higher load, and if we include it, we would need to
support it.

The main concern I have is related to the out-of-the-box performance. We
need to document how to enable the RocksDb configuration and we should make
clear that the published results were accomplished with that config.

Additionally, since the ultimate goal for the DbLedgerStorage was to be
merged into BookKeeper, it would be good to understand what are the options
on that side.
For example, will it be fine to have RocksDb as an optional dependency
(needed to compile from source, but not used or included in binary
distribution) ?
That way we could include the KeyValueRocksDb java class that uses the API
and compile it. Then, since the dependency is optional, it would require
the user to explicitly include RocksDB to have it working.


What do you think?
Matteo


--
Matteo Merli

Re: [DISCUSS] Make RocksDb dependency optional

[Matteo Merli]

There have been some updates on this story:

1. This morning a mail was sent out to all PMCs instructing to remove
RocksDb before Aug 31st. That was only for projects that had released code
using RocksDb and all the other project were prevented from starting using
it.
https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088663&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16088663

2. This afternoon in an update from Facebook, they announced they're going
to switch to dual license RocksDb in ALv2 + GPL.
https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088730&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16088730


In light of this, we should continue as we are right now and we should be
able to keep using RocksDB.





On Fri, Jul 14, 2017 at 3:26 PM Matteo Merli  wrote:

> As we have already talked about, the RocksDb library has been put in the
> Category-X (Forbidden) by the ASF, due to licensing problems.
>
> With my very limited understanding of the issue, it seems the problem
> resides in the patent clauses that's added the license.
>
> https://www.apache.org/legal/resolved.html#category-x
> 
> Rocks DB License
>
> The Rocks DB license includes a specification of a PATENTS file that
> passes along risk to downstream consumers of our software imbalanced in
> favor of the licensor, not the licensee, thereby violating our Apache legal
> policy of being a universal donor <https://s.apache.org/4Uzg>. The terms
> of ROCKSDB license are not a subset of those found in the ALv2, and they
> cannot be sublicensed as ALv2.
> 
>
> Pulsar is using RocksDb since that is being pulled in from the Yahoo
> branch of BookKeeper, which includes an alternative LedgerStorage
> implementation that stores the indexes for the entries in a RocksDB
> databases instead of using individual index files as it's default for
> BookKeeper. That is important to store many tens of thousands of ledgers
> per single Bookie node.
>
> To be able to release "Apache Pulsar" we need to fix the dependency
> problem. These are the options that I have been thinking of (feel free to
> propose alternatives! ):
>
>  1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>
>  2. Provide an additional key-value storage implementation for the
> DbLedgerStorage (eg: LevelDb) and make that the default
>
>  3. Wait until there's more clarity on the subject (that would mean to
> delay the Pulsar release)
>
> As I see it, I prefer option #1 for now, because I don't see any
> comparable replacement for RocksDB. LevelDb works fine for small workloads
> but show several problems at higher load, and if we include it, we would
> need to support it.
>
> The main concern I have is related to the out-of-the-box performance. We
> need to document how to enable the RocksDb configuration and we should make
> clear that the published results were accomplished with that config.
>
> Additionally, since the ultimate goal for the DbLedgerStorage was to be
> merged into BookKeeper, it would be good to understand what are the options
> on that side.
> For example, will it be fine to have RocksDb as an optional dependency
> (needed to compile from source, but not used or included in binary
> distribution) ?
> That way we could include the KeyValueRocksDb java class that uses the API
> and compile it. Then, since the dependency is optional, it would require
> the user to explicitly include RocksDB to have it working.
>
>
> What do you think?
> Matteo
>
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] Make RocksDb dependency optional

[Matteo Merli]

FYI: License change in RocksDB is already happening:
https://github.com/facebook/rocksdb/pull/2589

--
Matteo Merli


On Sat, Jul 15, 2017 at 3:56 PM, Matteo Merli  wrote:

> There have been some updates on this story:
>
> 1. This morning a mail was sent out to all PMCs instructing to remove
> RocksDb before Aug 31st. That was only for projects that had released code
> using RocksDb and all the other project were prevented from starting using
> it.
> https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088663&;
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16088663
>
> 2. This afternoon in an update from Facebook, they announced they're going
> to switch to dual license RocksDb in ALv2 + GPL.
> https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16088730&;
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16088730
>
>
> In light of this, we should continue as we are right now and we should be
> able to keep using RocksDB.
>
>
>
>
>
> On Fri, Jul 14, 2017 at 3:26 PM Matteo Merli  wrote:
>
>> As we have already talked about, the RocksDb library has been put in the
>> Category-X (Forbidden) by the ASF, due to licensing problems.
>>
>> With my very limited understanding of the issue, it seems the problem
>> resides in the patent clauses that's added the license.
>>
>> https://www.apache.org/legal/resolved.html#category-x
>> 
>> Rocks DB License
>>
>> The Rocks DB license includes a specification of a PATENTS file that
>> passes along risk to downstream consumers of our software imbalanced in
>> favor of the licensor, not the licensee, thereby violating our Apache legal
>> policy of being a universal donor <https://s.apache.org/4Uzg>. The terms
>> of ROCKSDB license are not a subset of those found in the ALv2, and they
>> cannot be sublicensed as ALv2.
>> 
>>
>> Pulsar is using RocksDb since that is being pulled in from the Yahoo
>> branch of BookKeeper, which includes an alternative LedgerStorage
>> implementation that stores the indexes for the entries in a RocksDB
>> databases instead of using individual index files as it's default for
>> BookKeeper. That is important to store many tens of thousands of ledgers
>> per single Bookie node.
>>
>> To be able to release "Apache Pulsar" we need to fix the dependency
>> problem. These are the options that I have been thinking of (feel free to
>> propose alternatives! ):
>>
>>  1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
>> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
>>
>>  2. Provide an additional key-value storage implementation for the
>> DbLedgerStorage (eg: LevelDb) and make that the default
>>
>>  3. Wait until there's more clarity on the subject (that would mean to
>> delay the Pulsar release)
>>
>> As I see it, I prefer option #1 for now, because I don't see any
>> comparable replacement for RocksDB. LevelDb works fine for small workloads
>> but show several problems at higher load, and if we include it, we would
>> need to support it.
>>
>> The main concern I have is related to the out-of-the-box performance. We
>> need to document how to enable the RocksDb configuration and we should make
>> clear that the published results were accomplished with that config.
>>
>> Additionally, since the ultimate goal for the DbLedgerStorage was to be
>> merged into BookKeeper, it would be good to understand what are the options
>> on that side.
>> For example, will it be fine to have RocksDb as an optional dependency
>> (needed to compile from source, but not used or included in binary
>> distribution) ?
>> That way we could include the KeyValueRocksDb java class that uses the
>> API and compile it. Then, since the dependency is optional, it would
>> require the user to explicitly include RocksDB to have it working.
>>
>>
>> What do you think?
>> Matteo
>>
>>
>> --
>> Matteo Merli
>> 
>>
>

Re: [DISCUSS] Make RocksDb dependency optional

[Matteo Merli]

Dave,
I think RocksDB is cutting a new release 5.5.4 with Apache License right
now :

https://github.com/facebook/rocksdb/commits/5.5.fb

Thankfully they're moving quickly to solve the issue!

Matteo

--
Matteo Merli


On Tue, Jul 18, 2017 at 9:17 AM, Dave Fisher  wrote:

> Hi -
>
> Current guidance is to wait for the relicensed version of RocksDB before
> it can be allowed. Projects currently have until August 31 to resolve the
> issues in Apache releases. Pulsar does not yet have an Apache release.
>
> It becomes a question of timing within RocksDB, Bookkeeper and this
> podling. If the podling gets to the point of releasing before the proper
> licensed version is available then following Matteo’s suggestion is
> probably needed. That wouldn’t stop the project from publishing benchmark
> data based on. RocksDB.
>
> Regards,
> Dave
>
> > On Jul 16, 2017, at 12:02 PM, Dave Fisher  wrote:
> >
> > Hi -
> >
> > Please see https://issues.apache.org/jira/plugins/servlet/mobile#
> issue/LEGAL-303/comment/16088730
> >
> > It looks like the RocksDB community is resolving the issue!
> >
> > Regards,
> > Dave
> >
> >
> > Sent from my iPhone
> >
> >> On Jul 14, 2017, at 3:26 PM, Matteo Merli  wrote:
> >>
> >> As we have already talked about, the RocksDb library has been put in the
> >> Category-X (Forbidden) by the ASF, due to licensing problems.
> >>
> >> With my very limited understanding of the issue, it seems the problem
> >> resides in the patent clauses that's added the license.
> >>
> >> https://www.apache.org/legal/resolved.html#category-x
> >> 
> 
> >> Rocks DB License
> >>
> >> The Rocks DB license includes a specification of a PATENTS file that
> passes
> >> along risk to downstream consumers of our software imbalanced in favor
> of
> >> the licensor, not the licensee, thereby violating our Apache legal
> policy
> >> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
> >> ROCKSDB license are not a subset of those found in the ALv2, and they
> >> cannot be sublicensed as ALv2.
> >> 
> 
> >>
> >> Pulsar is using RocksDb since that is being pulled in from the Yahoo
> branch
> >> of BookKeeper, which includes an alternative LedgerStorage
> implementation
> >> that stores the indexes for the entries in a RocksDB databases instead
> of
> >> using individual index files as it's default for BookKeeper. That is
> >> important to store many tens of thousands of ledgers per single Bookie
> node.
> >>
> >> To be able to release "Apache Pulsar" we need to fix the dependency
> >> problem. These are the options that I have been thinking of (feel free
> to
> >> propose alternatives! ):
> >>
> >> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
> >> explicitly exclude the RocksDb dependency from BookKeeper (yahoo branch)
> >>
> >> 2. Provide an additional key-value storage implementation for the
> >> DbLedgerStorage (eg: LevelDb) and make that the default
> >>
> >> 3. Wait until there's more clarity on the subject (that would mean to
> >> delay the Pulsar release)
> >>
> >> As I see it, I prefer option #1 for now, because I don't see any
> comparable
> >> replacement for RocksDB. LevelDb works fine for small workloads but show
> >> several problems at higher load, and if we include it, we would need to
> >> support it.
> >>
> >> The main concern I have is related to the out-of-the-box performance. We
> >> need to document how to enable the RocksDb configuration and we should
> make
> >> clear that the published results were accomplished with that config.
> >>
> >> Additionally, since the ultimate goal for the DbLedgerStorage was to be
> >> merged into BookKeeper, it would be good to understand what are the
> options
> >> on that side.
> >> For example, will it be fine to have RocksDb as an optional dependency
> >> (needed to compile from source, but not used or included in binary
> >> distribution) ?
> >> That way we could include the KeyValueRocksDb java class that uses the
> API
> >> and compile it. Then, since the dependency is optional, it would require
> >> the user to explicitly include RocksDB to have it working.
> >>
> >>
> >> What do you think?
> >> Matteo
> >>
> >>
> >> --
> >> Matteo Merli
> >> 
>
>

Pulsar web site is online!

[Matteo Merli]

The new website is now online at

https://pulsar.incubator.apache.org/

For devs:
 * All the contents is now under ./site/ directory

 * Any change to the docs merged to master will appear quickly online

 * Refer to
https://github.com/apache/incubator-pulsar/blob/master/site/README.md for
information on how to test locally the changes to site


Please submit PRs for anything out of place you see.

Matteo

--
Matteo Merli

Re: Pulsar web site is online!

[Matteo Merli]

>
> 1. Under "Client Libraries", the links below C++ returns 404 except
> "Python docs".
>
> I don't understand why the build didn't include the Javadoc and Doxygen.
That should be something on the Makefile.


> 2. There seems to be some encoding issue: e.g:
> https://pulsar.incubator.apache.org/docs/latest/getting-started/
> ConceptsAndArchitecture/#Architectureoverview,
> I see texts like "Pulsar’s key features include:"
>
>
Ok, I didn't see any encoding problem before (I was trying on MacOS with
Chrome and Safari)

I'm trying to fix it. I'll send out a PR shortly.


Dave & Taylor: Thanks for the links instructions. I'll add that as well

Re: svn commit: r20484 - /dev/incubator/pulsar/

[Matteo Merli]

I was just testing to see whether the SVN permission were working now.

--
Matteo Merli


On Tue, Jul 18, 2017 at 4:23 PM,  wrote:

> Author: mmerli
> Date: Tue Jul 18 23:23:44 2017
> New Revision: 20484
>
> Log:
> Creating staging dist directory for Pulsar
>
> Added:
> dev/incubator/pulsar/
>
>

Re: Pulsar web site is online!

[Matteo Merli]

The fix for the encoding and the missing javadoc pages is in PR
https://github.com/apache/incubator-pulsar/pull/582

Once the the build passes, it should be ready to merge, please take a look
as well.
On Tue, Jul 18, 2017 at 4:56 PM Nozomi Kurihara 
wrote:

> Great job!
>
> ➢ When I go to japanese under documentation, it does not look Japanese
> ➢ to me:) Can you also check?
>
> Some encoding issue occurs in the Japanese documents.
> We will check it.
>
> Thanks,
> Nozomi
>
> 2017/07/19 4:40 に、"Sahaya Andrews"  を書き込みました:
>
> Nice! Looks good.
>
> 1. Under "Client Libraries", the links below C++ returns 404 except
> "Python docs".
>
> 2. There seems to be some encoding issue: e.g:
>
> https://pulsar.incubator.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Architectureoverview
> ,
> I see texts like "Pulsar’s key features include:"
>
> Nozomi & Yuki,
>
>   When I go to japanese under documentation, it does not look Japanese
> to me:) Can you also check?
>
> Andrews.
>
> On Tue, Jul 18, 2017 at 12:30 PM, P. Taylor Goetz 
> wrote:
> > Nice job!
> >
> > -Taylor
> >
> >> On Jul 18, 2017, at 3:06 PM, Sijie Guo  wrote:
> >>
> >> Awesome! Great job, Matteo and the community! Congrats!
> >>
> >> - Sijie
> >>
> >> On Wed, Jul 19, 2017 at 2:57 AM, Matteo Merli 
> wrote:
> >>
> >>> The new website is now online at
> >>>
> >>> https://pulsar.incubator.apache.org/
> >>>
> >>> For devs:
> >>> * All the contents is now under ./site/ directory
> >>>
> >>> * Any change to the docs merged to master will appear quickly
> online
> >>>
> >>> * Refer to
> >>>
> https://github.com/apache/incubator-pulsar/blob/master/site/README.md for
> >>> information on how to test locally the changes to site
> >>>
> >>>
> >>> Please submit PRs for anything out of place you see.
> >>>
> >>> Matteo
> >>>
> >>> --
> >>> Matteo Merli
> >>> 
> >>>
> >
>
>
>

Re: Pulsar web site is online!

[Matteo Merli]

Yes, the encoding issues should have been fixed by the build last night.
(It's still strange that was not showing up on certain machines compared to
others.. even using the same browsers.. )

Taylor, Dave: We also added the standard ASF links in the "Community"
drop-down. Please take a look.

The Javadoc link was fixed and the C++ Doxygen and Python generated docs
are still broken, though a fix is on the way:
https://github.com/apache/incubator-pulsar/pull/585

--
Matteo Merli


On Wed, Jul 19, 2017 at 1:20 AM, Nozomi Kurihara 
wrote:

> Thanks!
> I confirmed the Japanese document is recovered.
>
> Nozomi
>
> 2017/07/19 9:01 に、"Matteo Merli"  を書き込みました:
>
> The fix for the encoding and the missing javadoc pages is in PR
> https://github.com/apache/incubator-pulsar/pull/582
>
> Once the the build passes, it should be ready to merge, please take a
> look
> as well.
> On Tue, Jul 18, 2017 at 4:56 PM Nozomi Kurihara <
> nkuri...@yahoo-corp.jp>
> wrote:
>
> > Great job!
> >
> > ➢ When I go to japanese under documentation, it does not look
> Japanese
> > ➢ to me:) Can you also check?
> >
> > Some encoding issue occurs in the Japanese documents.
> > We will check it.
> >
> > Thanks,
> > Nozomi
> >
> > 2017/07/19 4:40 に、"Sahaya Andrews"  を書き込みました:
> >
> > Nice! Looks good.
> >
> > 1. Under "Client Libraries", the links below C++ returns 404
> except
> > "Python docs".
> >
> > 2. There seems to be some encoding issue: e.g:
> >
> > https://pulsar.incubator.apache.org/docs/latest/getting-started/
> ConceptsAndArchitecture/#Architectureoverview
> > ,
> > I see texts like "Pulsar’s key features include:"
> >
> > Nozomi & Yuki,
> >
> >   When I go to japanese under documentation, it does not look
> Japanese
> > to me:) Can you also check?
> >
> > Andrews.
> >
> > On Tue, Jul 18, 2017 at 12:30 PM, P. Taylor Goetz <
> ptgo...@gmail.com>
>     > wrote:
> > > Nice job!
> > >
> > > -Taylor
> > >
> > >> On Jul 18, 2017, at 3:06 PM, Sijie Guo 
> wrote:
> > >>
> > >> Awesome! Great job, Matteo and the community! Congrats!
> > >>
> > >> - Sijie
> > >>
> > >> On Wed, Jul 19, 2017 at 2:57 AM, Matteo Merli <
> mme...@apache.org>
> > wrote:
> > >>
> > >>> The new website is now online at
> > >>>
> > >>> https://pulsar.incubator.apache.org/
> > >>>
> > >>> For devs:
> > >>> * All the contents is now under ./site/ directory
> > >>>
> > >>> * Any change to the docs merged to master will appear quickly
> > online
> > >>>
> > >>> * Refer to
> > >>>
> > https://github.com/apache/incubator-pulsar/blob/master/
> site/README.md for
> > >>> information on how to test locally the changes to site
> > >>>
> > >>>
> > >>> Please submit PRs for anything out of place you see.
> > >>>
> > >>> Matteo
> > >>>
> > >>> --
> > >>> Matteo Merli
> > >>> 
> > >>>
> > >
> >
> >
> >
>
>
>

Re: [DISCUSS] Make RocksDb dependency optional

[Matteo Merli]

I have just created a PR to upgrade RocksDB and resolve the license issue.

https://github.com/apache/incubator-pulsar/pull/587

We don't need to wait on BK release, since we can force which RocksDB
version we want to include.



--
Matteo Merli


On Tue, Jul 18, 2017 at 2:11 PM, Matteo Merli  wrote:

> Dave,
> I think RocksDB is cutting a new release 5.5.4 with Apache License right
> now :
>
> https://github.com/facebook/rocksdb/commits/5.5.fb
>
> Thankfully they're moving quickly to solve the issue!
>
> Matteo
>
> --
> Matteo Merli
> 
>
> On Tue, Jul 18, 2017 at 9:17 AM, Dave Fisher 
> wrote:
>
>> Hi -
>>
>> Current guidance is to wait for the relicensed version of RocksDB before
>> it can be allowed. Projects currently have until August 31 to resolve the
>> issues in Apache releases. Pulsar does not yet have an Apache release.
>>
>> It becomes a question of timing within RocksDB, Bookkeeper and this
>> podling. If the podling gets to the point of releasing before the proper
>> licensed version is available then following Matteo’s suggestion is
>> probably needed. That wouldn’t stop the project from publishing benchmark
>> data based on. RocksDB.
>>
>> Regards,
>> Dave
>>
>> > On Jul 16, 2017, at 12:02 PM, Dave Fisher 
>> wrote:
>> >
>> > Hi -
>> >
>> > Please see https://issues.apache.org/jira/plugins/servlet/mobile#issue/
>> LEGAL-303/comment/16088730
>> >
>> > It looks like the RocksDB community is resolving the issue!
>> >
>> > Regards,
>> > Dave
>> >
>> >
>> > Sent from my iPhone
>> >
>> >> On Jul 14, 2017, at 3:26 PM, Matteo Merli  wrote:
>> >>
>> >> As we have already talked about, the RocksDb library has been put in
>> the
>> >> Category-X (Forbidden) by the ASF, due to licensing problems.
>> >>
>> >> With my very limited understanding of the issue, it seems the problem
>> >> resides in the patent clauses that's added the license.
>> >>
>> >> https://www.apache.org/legal/resolved.html#category-x
>> >> 
>> 
>> >> Rocks DB License
>> >>
>> >> The Rocks DB license includes a specification of a PATENTS file that
>> passes
>> >> along risk to downstream consumers of our software imbalanced in favor
>> of
>> >> the licensor, not the licensee, thereby violating our Apache legal
>> policy
>> >> of being a universal donor <https://s.apache.org/4Uzg>. The terms of
>> >> ROCKSDB license are not a subset of those found in the ALv2, and they
>> >> cannot be sublicensed as ALv2.
>> >> 
>> 
>> >>
>> >> Pulsar is using RocksDb since that is being pulled in from the Yahoo
>> branch
>> >> of BookKeeper, which includes an alternative LedgerStorage
>> implementation
>> >> that stores the indexes for the entries in a RocksDB databases instead
>> of
>> >> using individual index files as it's default for BookKeeper. That is
>> >> important to store many tens of thousands of ledgers per single Bookie
>> node.
>> >>
>> >> To be able to release "Apache Pulsar" we need to fix the dependency
>> >> problem. These are the options that I have been thinking of (feel free
>> to
>> >> propose alternatives! ):
>> >>
>> >> 1. Configure out of the box Pulsar to use InterleavedLedgerStorage and
>> >> explicitly exclude the RocksDb dependency from BookKeeper (yahoo
>> branch)
>> >>
>> >> 2. Provide an additional key-value storage implementation for the
>> >> DbLedgerStorage (eg: LevelDb) and make that the default
>> >>
>> >> 3. Wait until there's more clarity on the subject (that would mean to
>> >> delay the Pulsar release)
>> >>
>> >> As I see it, I prefer option #1 for now, because I don't see any
>> comparable
>> >> replacement for RocksDB. LevelDb works fine for small workloads but
>> show
>> >> several problems at higher load, and if we include it, we would need to
>> >> support it.
>> >>
>> >> The main concern I have is related to the out-of-the-box performance.
>> We
>> >> need to document how to enable the RocksDb configuration and we should
>> make
>> >> clear that the published results were accomplished with that config.
>> >>
>> >> Additionally, since the ultimate goal for the DbLedgerStorage was to be
>> >> merged into BookKeeper, it would be good to understand what are the
>> options
>> >> on that side.
>> >> For example, will it be fine to have RocksDb as an optional dependency
>> >> (needed to compile from source, but not used or included in binary
>> >> distribution) ?
>> >> That way we could include the KeyValueRocksDb java class that uses the
>> API
>> >> and compile it. Then, since the dependency is optional, it would
>> require
>> >> the user to explicitly include RocksDB to have it working.
>> >>
>> >>
>> >> What do you think?
>> >> Matteo
>> >>
>> >>
>> >> --
>> >> Matteo Merli
>> >> 
>>
>>
>

Re: Github documentation links

[Matteo Merli]

Yes, the README.md in root directory needs to be updated to point to the
website.

It couldn't be done in the same commit, because it took time to get the
website online after the commit.

I'm changing it right now, and I'll also add basic info on how to build
from source, since the README is what is included in the source
distribution and it will not be anymore the project "homepage".

--
Matteo Merli


On Fri, Jul 21, 2017 at 2:22 PM, Sahaya Andrews  wrote:

> Did we link our GitHub page "Documentation" section to the new website
> or is it still work in progress? They are returning 404.
> e.g: https://github.com/apache/incubator-pulsar/blob/master/
> docs/Documentation.md
>

Re: Pulsar web site is online!

[Matteo Merli]

Hi Dave,

sorry for late reply

> You might take a look at https://www.apache.org/foundation/marks/linking

I went through this and so far we don't have any external links / logos. Is
there any specific point you were thinking about?

> There are links to GitHub.com/yahoo/pulsar
<http://github.com/yahoo/pulsar> that are no longer valid in the docs - on
InstanceSetup for example.

Thanks for catching, will fix these.

> We need to discuss where the podling should place Apache releases. Is
Github.com <http://github.com/> acceptable or is the project required to
use Apache Dist and the Mirrors?
> Also - Pulsar team it would be good to show on the downloads page that
version 1.18 was the release just prior to moving to the Apache Software
Foundation.

I think we should stick with the regular Apache process, and maybe just use
github releases page as a place where to have the release notes and links
to the actual download page.

I agree we need to mark the 1.18 release as the "last release before
entering incubation". I'll add the disclaimer on the webpage.

So, we're still kind of bridging the gap from old-repo and releases model
to the new one. I think we should try to get an Apache release ready ASAP,
so that it will be easier to normalize all links, processes and
documentation.

Matteo


--
Matteo Merli


On Wed, Jul 19, 2017 at 12:40 PM, Dave Fisher  wrote:

> Hi Matteo,
>
> On Jul 19, 2017, at 10:35 AM, Matteo Merli  wrote:
>
> Yes, the encoding issues should have been fixed by the build last night.
> (It's still strange that was not showing up on certain machines compared to
> others.. even using the same browsers.. )
>
> Taylor, Dave: We also added the standard ASF links in the "Community"
> drop-down. Please take a look.
>
>
> Looks good. The “Apache” header might be redundant, but that’s up to the
> Pulsar team.
>
> You might take a look at https://www.apache.org/foundation/marks/linking
>
>
> The Javadoc link was fixed and the C++ Doxygen and Python generated docs
> are still broken, though a fix is on the way:
> https://github.com/apache/incubator-pulsar/pull/585
>
>
> There are links to GitHub.com/yahoo/pulsar that are no longer valid in
> the docs - on InstanceSetup for example.
>
> Mentors - See pages https://pulsar.incubator.apache.org/docs/
> latest/deployment/InstanceSetup/ and https://pulsar.incubator.apache.org/
> download/
>
> We need to discuss where the podling should place Apache releases. Is
> Github.com acceptable or is the project required to use Apache Dist and
> the Mirrors?
>
> Also - Pulsar team it would be good to show on the downloads page that
> version 1.18 was the release just prior to moving to the Apache Software
> Foundation.
>
> Regards,
> Dave
>
>
> --
> Matteo Merli
> 
>
> On Wed, Jul 19, 2017 at 1:20 AM, Nozomi Kurihara 
> wrote:
>
> Thanks!
> I confirmed the Japanese document is recovered.
>
> Nozomi
>
> 2017/07/19 9:01 に、"Matteo Merli"  を書き込みました:
>
>The fix for the encoding and the missing javadoc pages is in PR
>https://github.com/apache/incubator-pulsar/pull/582
>
>Once the the build passes, it should be ready to merge, please take a
> look
>as well.
>On Tue, Jul 18, 2017 at 4:56 PM Nozomi Kurihara <
> nkuri...@yahoo-corp.jp>
>wrote:
>
> Great job!
>
> ➢ When I go to japanese under documentation, it does not look
>
> Japanese
>
> ➢ to me:) Can you also check?
>
> Some encoding issue occurs in the Japanese documents.
> We will check it.
>
> Thanks,
> Nozomi
>
> 2017/07/19 4:40 に、"Sahaya Andrews"  を書き込みました:
>
>Nice! Looks good.
>
>1. Under "Client Libraries", the links below C++ returns 404
>
> except
>
>"Python docs".
>
>2. There seems to be some encoding issue: e.g:
>
> https://pulsar.incubator.apache.org/docs/latest/getting-started/
>
> ConceptsAndArchitecture/#Architectureoverview
>
> ,
>I see texts like "Pulsar’s key features include:"
>
>Nozomi & Yuki,
>
>  When I go to japanese under documentation, it does not look
>
> Japanese
>
>to me:) Can you also check?
>
>Andrews.
>
>On Tue, Jul 18, 2017 at 12:30 PM, P. Taylor Goetz <
>
> ptgo...@gmail.com>
>
> wrote:
>
> Nice job!
>
> -Taylor
>
> On Jul 18, 2017, at 3:06 PM, Sijie Guo 
>
> wrote:
>
>
> Awesome! Great job, Matteo and the community! Congrats!
>
> - Sijie
>
> On Wed, Jul 19, 2017 at 2:57 AM, Matteo Merli <
>
> mme...@apache.org>
>
> wrote:
>
>
> The new website is now online at
>
> https://pulsar.incubator.apache.org/
>
> For devs:
> * All the contents is now under ./site/ directory
>
> * Any change to the docs merged to master will appear quickly
>
> online
>
>
> * Refer to
>
> https://github.com/apache/incubator-pulsar/blob/master/
>
> site/README.md for
>
> information on how to test locally the changes to site
>
>
> Please submit PRs for anything out of place you see.
>
> Matteo
>
> --
> Matteo Merli
> 
>
>
>
>
>
>
>
>
>
>

[DISCUSS] Pulsar release

[Matteo Merli]

(Starting a new thread to detach from the general website discussion)

I think that it would be good to try to release a new Pulsar version ASAP,
following the Apache release process and requirements.

That would take the project out of the current limbo where we are in the
incubator, but we can only point users to Yahoo releases of Pulsar.

Once the 1st Apache release is ready, it would be easier to create new
releases, since most of comments and feedback from the IPMC are related to
release process and content of the artifacts. Once we establish that, the
rest will be downhill.

Questions:

 * Release Manager
   Does for podlings work in the same way as for TLPs? Should we design a
release manager?

 * Pending changes
What are the pending changes that *absolutely* need to go in
1.19-incubating release?

 * Versioning number
   Since we're going from "pulsar" to "apache-pulsar" artifacts, as Dave
suggested it could be the right moment to change versions. The options I
see are :

- 1.19-incubating This is keeping the continuity in versioning and also
signals that no major changes have been introduced.
- 2.0-incubating I think should be reserved for breaking changes
- 1.0-incubating I wouldn't go back in time

Any suggestions?

Open items:
 * Formalize the release procedure
It should be documented in the Wiki so that each committer should be
able to perform a release.

For all committers, please go through these documents to familiarize with
the process, and provide feedback on the Pulsar release instruction wiki:

   - https://incubator.apache.org/guides/releasemanagement.html
   - http://www.apache.org/legal/release-policy.html
   - http://www.apache.org/dev/release-distribution.html
   - http://www.apache.org/dev/release-publishing.html
   - http://www.apache.org/dev/publishing-maven-artifacts.html


Matteo

--
Matteo Merli

Re: [DISCUSS] Pulsar release

[Matteo Merli]

On Mon, Jul 24, 2017 at 11:39 AM, Dave Fisher  wrote:
>
> >   Does for podlings work in the same way as for TLPs? Should we design a
> > release manager?
>
> Yes. I see you created a KEYS file. If someone else is Release Manager
> then they can add their key as well.
>
> BTW - Is your Key Signed and in the Web of Trust?
>

Dave, I have created the key as per http://apache.org/dev/openpgp.html
and I have summarized the steps into a wiki page at
https://github.com/apache/incubator-pulsar/wiki/Create-GPG-keys-to-sign-release-artifacts

I didn't see how to sign the key itself and didn't get a chance to exchange
it.

Re: [DISCUSS] Pulsar release

[Matteo Merli]

Bay area. Who can I get to sign it? Can any other Apache committer (or PMC
from other projects) do it?

--
Matteo Merli


On Mon, Jul 24, 2017 at 12:27 PM, Dave Fisher  wrote:

>
> > On Jul 24, 2017, at 12:23 PM, Matteo Merli  wrote:
> >
> > On Mon, Jul 24, 2017 at 11:39 AM, Dave Fisher 
> wrote:
> >>
> >>>  Does for podlings work in the same way as for TLPs? Should we design a
> >>> release manager?
> >>
> >> Yes. I see you created a KEYS file. If someone else is Release Manager
> >> then they can add their key as well.
> >>
> >> BTW - Is your Key Signed and in the Web of Trust?
> >>
> >
> > Dave, I have created the key as per http://apache.org/dev/openpgp.html
> > and I have summarized the steps into a wiki page at
> > https://github.com/apache/incubator-pulsar/wiki/Create-
> GPG-keys-to-sign-release-artifacts
> >
> > I didn't see how to sign the key itself and didn't get a chance to
> exchange
> > it.
>
> Someone else would need to sign your key after physically validating your
> IDs. This is not an absolute requirement. Where in the world are you
> located?
>
> Regards,
> Dave
>

Re: [DISCUSS] Pulsar release

[Matteo Merli]

Thanks Dave, I think I'll try with some of my colleagues here. I'll let you
know if I can make it.



--
Matteo Merli


On Mon, Jul 24, 2017 at 1:06 PM, Dave Fisher  wrote:

>
> > On Jul 24, 2017, at 12:33 PM, Matteo Merli 
> wrote:
> >
> > Bay area. Who can I get to sign it? Can any other Apache committer (or
> PMC
> > from other projects) do it?
>
> Yes, if they have participated in a key signing.
>
> I am in the North Bay myself and recently met someone to sign their key.
> We could make arrangements. Try me on Slack later.
>
> Regards,
> Dave
>
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Jul 24, 2017 at 12:27 PM, Dave Fisher 
> wrote:
> >
> >>
> >>> On Jul 24, 2017, at 12:23 PM, Matteo Merli  wrote:
> >>>
> >>> On Mon, Jul 24, 2017 at 11:39 AM, Dave Fisher 
> >> wrote:
> >>>>
> >>>>> Does for podlings work in the same way as for TLPs? Should we design
> a
> >>>>> release manager?
> >>>>
> >>>> Yes. I see you created a KEYS file. If someone else is Release Manager
> >>>> then they can add their key as well.
> >>>>
> >>>> BTW - Is your Key Signed and in the Web of Trust?
> >>>>
> >>>
> >>> Dave, I have created the key as per http://apache.org/dev/openpgp.html
> >>> and I have summarized the steps into a wiki page at
> >>> https://github.com/apache/incubator-pulsar/wiki/Create-
> >> GPG-keys-to-sign-release-artifacts
> >>>
> >>> I didn't see how to sign the key itself and didn't get a chance to
> >> exchange
> >>> it.
> >>
> >> Someone else would need to sign your key after physically validating
> your
> >> IDs. This is not an absolute requirement. Where in the world are you
> >> located?
> >>
> >> Regards,
> >> Dave
> >>
>
>

Re: [DISCUSS] Pulsar release

[Matteo Merli]

I have created a wiki page with the WIP instructions for the release

https://github.com/apache/incubator-pulsar/wiki/Release-process

I also have created an INFRA ticket to create the Nexus project so that we
can publish maven artifacts.
https://issues.apache.org/jira/browse/INFRA-14694


Any comments on the questions asked?

If there are no concerns/objections I could volunteer to be the release
manager for this release and then we could start a rotation across the
committers for the subsequent releases.

Matteo

--
Matteo Merli


On Fri, Jul 21, 2017 at 4:23 PM, Matteo Merli  wrote:

> (Starting a new thread to detach from the general website discussion)
>
> I think that it would be good to try to release a new Pulsar version ASAP,
> following the Apache release process and requirements.
>
> That would take the project out of the current limbo where we are in the
> incubator, but we can only point users to Yahoo releases of Pulsar.
>
> Once the 1st Apache release is ready, it would be easier to create new
> releases, since most of comments and feedback from the IPMC are related to
> release process and content of the artifacts. Once we establish that, the
> rest will be downhill.
>
> Questions:
>
>  * Release Manager
>Does for podlings work in the same way as for TLPs? Should we design a
> release manager?
>
>  * Pending changes
> What are the pending changes that *absolutely* need to go in
> 1.19-incubating release?
>
>  * Versioning number
>Since we're going from "pulsar" to "apache-pulsar" artifacts, as Dave
> suggested it could be the right moment to change versions. The options I
> see are :
>
> - 1.19-incubating This is keeping the continuity in versioning and
> also signals that no major changes have been introduced.
> - 2.0-incubating I think should be reserved for breaking changes
> - 1.0-incubating I wouldn't go back in time
>
> Any suggestions?
>
> Open items:
>  * Formalize the release procedure
> It should be documented in the Wiki so that each committer should be
> able to perform a release.
>
> For all committers, please go through these documents to familiarize with
> the process, and provide feedback on the Pulsar release instruction wiki:
>
>- https://incubator.apache.org/guides/releasemanagement.html
>- http://www.apache.org/legal/release-policy.html
>- http://www.apache.org/dev/release-distribution.html
>- http://www.apache.org/dev/release-publishing.html
>- http://www.apache.org/dev/publishing-maven-artifacts.html
>
>
> Matteo
>
> --
> Matteo Merli
> 
>

Re: [DISCUSS] Pulsar release

[Matteo Merli]

Ok, my question was also on the line: "is anyone ok in doing a release now"
? :)

> Instruction seems to be detailed. Though I will only be able to add
> meaningful comment when I attempt these(volunteer for a release
> manager) :)

Sure, the wiki is a living document. Anyone should feel free to update and
correct!

> Slightly off topic question: Are we not using Jenkins for the build?

Ideally we should try to move to Jenkins. The Java unit tests execution is
much more reliable compared to Travis.

There are few minor things to resolve for Jenkins, but the only major
problem is how to compile C++.

Now, we don't include the C++ build in our releases, but we should make
sure everything compiles. I haven't found a way to install
all the required dependency on Jenkins so that we can compile the C++
client.

WIP Jenkins build: https://builds.apache.org/job/pulsar-master/

If anyone has time to look into it.. welcome ;)


So my take is, for now use Travis, since it's already in place and working
and move to Jenkins once the build is setup there.

Matteo

--
Matteo Merli


On Mon, Jul 24, 2017 at 3:41 PM, Sahaya Andrews  wrote:

> Instruction seems to be detailed. Though I will only be able to add
> meaningful comment when I attempt these(volunteer for a release
> manager) :)
>
> Slightly off topic question: Are we not using Jenkins for the build?
>
> Andrews.
>
> On Mon, Jul 24, 2017 at 3:20 PM, Dave Fisher 
> wrote:
> >
> >> On Jul 24, 2017, at 3:09 PM, Matteo Merli  wrote:
> >>
> >> I have created a wiki page with the WIP instructions for the release
> >>
> >> https://github.com/apache/incubator-pulsar/wiki/Release-process
> >>
> >> I also have created an INFRA ticket to create the Nexus project so that
> we
> >> can publish maven artifacts.
> >> https://issues.apache.org/jira/browse/INFRA-14694
> >>
> >>
> >> Any comments on the questions asked?
> >
> > In item #3 of the release process two additional checks are needed to
> help with voting.
> >
> > (a) The output for the RAT report needs to be examined to make sure that
> all of the source has the proper headers. That any excluded files are
> properly explained.
> > (b) The two release artifacts need to have hashes generated so that
> users can be sure that they downloaded the genuine package. I believe that
> MD5 and SHA256 are now typical ...
> >
> >>
> >> If there are no concerns/objections I could volunteer to be the release
> >> manager for this release and then we could start a rotation across the
> >> committers for the subsequent releases.
> >
> > Rotation and a well documented build release process is excellent
> practice.
> >
> > Regards,
> > Dave
> >
> >>
> >> Matteo
> >>
> >> --
> >> Matteo Merli
> >> 
> >>
> >> On Fri, Jul 21, 2017 at 4:23 PM, Matteo Merli 
> wrote:
> >>
> >>> (Starting a new thread to detach from the general website discussion)
> >>>
> >>> I think that it would be good to try to release a new Pulsar version
> ASAP,
> >>> following the Apache release process and requirements.
> >>>
> >>> That would take the project out of the current limbo where we are in
> the
> >>> incubator, but we can only point users to Yahoo releases of Pulsar.
> >>>
> >>> Once the 1st Apache release is ready, it would be easier to create new
> >>> releases, since most of comments and feedback from the IPMC are
> related to
> >>> release process and content of the artifacts. Once we establish that,
> the
> >>> rest will be downhill.
> >>>
> >>> Questions:
> >>>
> >>> * Release Manager
> >>>   Does for podlings work in the same way as for TLPs? Should we design
> a
> >>> release manager?
> >>>
> >>> * Pending changes
> >>>What are the pending changes that *absolutely* need to go in
> >>> 1.19-incubating release?
> >>>
> >>> * Versioning number
> >>>   Since we're going from "pulsar" to "apache-pulsar" artifacts, as Dave
> >>> suggested it could be the right moment to change versions. The options
> I
> >>> see are :
> >>>
> >>>- 1.19-incubating This is keeping the continuity in versioning and
> >>> also signals that no major changes have been introduced.
> >>>- 2.0-incubating I think should be reserved for breaking changes
> >>>- 1.0-incubating I wouldn't go back in time
> >>>
> >>> Any suggestions?
> >>>
> >>> Open items:
> >>> * Formalize the release procedure
> >>>It should be documented in the Wiki so that each committer should be
> >>> able to perform a release.
> >>>
> >>> For all committers, please go through these documents to familiarize
> with
> >>> the process, and provide feedback on the Pulsar release instruction
> wiki:
> >>>
> >>>   - https://incubator.apache.org/guides/releasemanagement.html
> >>>   - http://www.apache.org/legal/release-policy.html
> >>>   - http://www.apache.org/dev/release-distribution.html
> >>>   - http://www.apache.org/dev/release-publishing.html
> >>>   - http://www.apache.org/dev/publishing-maven-artifacts.html
> >>>
> >>>
> >>> Matteo
> >>>
> >>> --
> >>> Matteo Merli
> >>> 
> >>>
> >
>

Re: [DISCUSS] Pulsar release

[Matteo Merli]

Yes, as far as I understand, first the release needs to be voted in the
Pulsar dev ML. The Pulsar PPMC votes would be binding there (basically all
committers at this point), though anyone else is welcome to verify the
release and vote as well.

Once the Pulsar PPMC approves, then we need to vote again for the Incubator
PMC to approve it.

I have update the wiki with the clarification:

https://github.com/apache/incubator-pulsar/wiki/Release-process/_compare/7da648d5d4a91b56c1807acf49720f2b21a95a76...cf001e26c4062f7104b62ff166dbd8d02cf8c8e4



--
Matteo Merli


On Tue, Jul 25, 2017 at 8:02 AM, Masakazu Kitajo  wrote:

> Just to confirm, do we have to run two separated votes on step 7 and 8?
> Should we run them step by step?
>
> Masakazu
>
> On Tue, Jul 25, 2017 at 8:45 AM, Matteo Merli  wrote:
>
> > Ok, my question was also on the line: "is anyone ok in doing a release
> now"
> > ? :)
> >
> > > Instruction seems to be detailed. Though I will only be able to add
> > > meaningful comment when I attempt these(volunteer for a release
> > > manager) :)
> >
> > Sure, the wiki is a living document. Anyone should feel free to update
> and
> > correct!
> >
> > > Slightly off topic question: Are we not using Jenkins for the build?
> >
> > Ideally we should try to move to Jenkins. The Java unit tests execution
> is
> > much more reliable compared to Travis.
> >
> > There are few minor things to resolve for Jenkins, but the only major
> > problem is how to compile C++.
> >
> > Now, we don't include the C++ build in our releases, but we should make
> > sure everything compiles. I haven't found a way to install
> > all the required dependency on Jenkins so that we can compile the C++
> > client.
> >
> > WIP Jenkins build: https://builds.apache.org/job/pulsar-master/
> >
> > If anyone has time to look into it.. welcome ;)
> >
> >
> > So my take is, for now use Travis, since it's already in place and
> working
> > and move to Jenkins once the build is setup there.
> >
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
> > On Mon, Jul 24, 2017 at 3:41 PM, Sahaya Andrews 
> > wrote:
> >
> > > Instruction seems to be detailed. Though I will only be able to add
> > > meaningful comment when I attempt these(volunteer for a release
> > > manager) :)
> > >
> > > Slightly off topic question: Are we not using Jenkins for the build?
> > >
> > > Andrews.
> > >
> > > On Mon, Jul 24, 2017 at 3:20 PM, Dave Fisher 
> > > wrote:
> > > >
> > > >> On Jul 24, 2017, at 3:09 PM, Matteo Merli 
> wrote:
> > > >>
> > > >> I have created a wiki page with the WIP instructions for the release
> > > >>
> > > >> https://github.com/apache/incubator-pulsar/wiki/Release-process
> > > >>
> > > >> I also have created an INFRA ticket to create the Nexus project so
> > that
> > > we
> > > >> can publish maven artifacts.
> > > >> https://issues.apache.org/jira/browse/INFRA-14694
> > > >>
> > > >>
> > > >> Any comments on the questions asked?
> > > >
> > > > In item #3 of the release process two additional checks are needed to
> > > help with voting.
> > > >
> > > > (a) The output for the RAT report needs to be examined to make sure
> > that
> > > all of the source has the proper headers. That any excluded files are
> > > properly explained.
> > > > (b) The two release artifacts need to have hashes generated so that
> > > users can be sure that they downloaded the genuine package. I believe
> > that
> > > MD5 and SHA256 are now typical ...
> > > >
> > > >>
> > > >> If there are no concerns/objections I could volunteer to be the
> > release
> > > >> manager for this release and then we could start a rotation across
> the
> > > >> committers for the subsequent releases.
> > > >
> > > > Rotation and a well documented build release process is excellent
> > > practice.
> > > >
> > > > Regards,
> > > > Dave
> > > >
> > > >>
> > > >> Matteo
> > > >>
> > > >> --
> > > >> Matteo Merli
> > > >> 
> > > >>
> > > >> On Fri, Jul 21, 2017 at 4:23 PM, Matteo Merli 
> > > wrote:

Re: [DISCUSS] Pulsar release

[Matteo Merli]

On Mon, Jul 24, 2017 at 3:20 PM, Dave Fisher  wrote:

> In item #3 of the release process two additional checks are needed to help
> with voting.
>
> (a) The output for the RAT report needs to be examined to make sure that
> all of the source has the proper headers. That any excluded files are
> properly explained.
>

Sure, right now the RAT check is performed in every CI build (to make sure
that Pull Requests are validated before getting merged). In any case, the
-src.tar.gz package content would be slightly different from the git
repository workdir, so it's a good idea to check it at the release.

I have updated the step #3 to add the "mvn apache-rat:check" verification.
As for the excluded files, the current list is in:
https://github.com/apache/incubator-pulsar/blob/master/pom.xml#L635


> (b) The two release artifacts need to have hashes generated so that users
> can be sure that they downloaded the genuine package. I believe that MD5
> and SHA256 are now typical ...
>

Yes, I have included that in the following step: "4. Sign and stage the
artifacts"
There is a small shell script that creates checksum and sign all the tar.gz
files.

https://github.com/apache/incubator-pulsar/blob/master/src/sign-release.sh

Re: New Slack app installed

[Matteo Merli]

I'm trying to re-install the digest app, since I was receiving the digest
myself instead of that getting sent here to the mailing list.

--
Matteo Merli


On Tue, Jul 25, 2017 at 9:39 AM, Slack  wrote:

> Hi Apache Pulsar Admin,
>
> You just installed Digest.AI for the pulsar Slack team.
>
> If you didn't make this change (or if you've changed your mind), you can
> remove this app from your team by visiting the Digest.AI configuration
> page:
>
> https://apache-pulsar.slack.com/applications/A1QP2Q8DB
>
> If you have any questions, we're happy to help. Please reach out to us
> at feedb...@slack.com.
>
> Cheers,
>
> - The team at Slack
>

Re: New Slack app installed

[Matteo Merli]

I got the complete text log from yesterday, though on my email only. By
default it sends a "summary", but there's the option to get the full
transcript.

The issue I got is that the "pulsar-admin" account was created on my email
and then I changed the email address to dev@pulsar.incubator.apache.org,
but the digest still goes there.

I just applied to create a new account (it should show up in the moderator
queue). That should hopefully fix the mail issue.



--
Matteo Merli


On Tue, Jul 25, 2017 at 10:17 AM, Dave Fisher  wrote:

> Hi Matteo,
>
> I got the digest today. It seems to truncate a long conversation with a
> link back to Slack. If that cannot be overcome then the podling will need
> to make sure to copy relevant decisions from Slack to the Dev list.
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Jul 25, 2017, at 9:43 AM, Matteo Merli  wrote:
> >
> > I'm trying to re-install the digest app, since I was receiving the digest
> > myself instead of that getting sent here to the mailing list.
> >
> > --
> > Matteo Merli
> > 
> >
> >> On Tue, Jul 25, 2017 at 9:39 AM, Slack  wrote:
> >>
> >> Hi Apache Pulsar Admin,
> >>
> >> You just installed Digest.AI for the pulsar Slack team.
> >>
> >> If you didn't make this change (or if you've changed your mind), you can
> >> remove this app from your team by visiting the Digest.AI configuration
> >> page:
> >>
> >> https://apache-pulsar.slack.com/applications/A1QP2Q8DB
> >>
> >> If you have any questions, we're happy to help. Please reach out to us
> >> at feedb...@slack.com.
> >>
> >> Cheers,
> >>
> >> - The team at Slack
> >>
>
>

Fwd: [jira] [Resolved] (INFRA-14694) Enable Nexus Access For Pulsar

[Matteo Merli]

The Nexus project is already active and we're already pushing snapshots
artifacts in the repo.
--
Matteo Merli


-- Forwarded message --
From: Chris Lambertus (JIRA) 
Date: Fri, Jul 28, 2017 at 5:37 PM
Subject: [jira] [Resolved] (INFRA-14694) Enable Nexus Access For Pulsar
To: mme...@apache.org


Chris Lambertus
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cml> *resolved*
as *Fixed*

-- Nexus repository was prepared successfully. --

Configuration has been prepared, now you can:
* Deploy snapshot artifacts into repository https://repository.apache.org/
content/repositories/snapshots
* Deploy release artifacts into the staging repository
https://repository.apache.org/service/local/staging/deploy/maven2
* Promote staged artifacts into repository 'Releases'
* Download snapshot and release artifacts from group
https://repository.apache.org/content/groups/public
* Download snapshot, release and staged artifacts from staging group
https://repository.apache.org/content/groups/staging
*
Infrastructure <https://issues.apache.org/jira/browse/INFRA> / [image: Task]
<https://issues.apache.org/jira/browse/INFRA-14694> INFRA-14694
<https://issues.apache.org/jira/browse/INFRA-14694>
Enable Nexus Access For Pulsar
<https://issues.apache.org/jira/browse/INFRA-14694>
Change By: Chris Lambertus
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cml>
Resolution: Fixed
Status: Waiting for Infra Closed
[image: Add Comment]
<https://issues.apache.org/jira/browse/INFRA-14694#add-comment> Add Comment
<https://issues.apache.org/jira/browse/INFRA-14694#add-comment>

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
[image: Atlassian logo]

Re: [DISCUSS] Pulsar release

[Matteo Merli]

Since there have been no further updates on this thread, I think we should
be good to cut the 1.19.0-incubating release.

The last big change that got merged was about the non-persistent topics.

I have one more PR that I'd like to include:
https://github.com/apache/incubator-pulsar/pull/514
(to have a flag that enable static linking, which is needed to create the
self-contained python client dist file)


After that, if there no objections, I'll start preparing the first
candidate package.

Matteo



--
Matteo Merli


On Tue, Jul 25, 2017 at 9:41 AM, Matteo Merli  wrote:

> On Mon, Jul 24, 2017 at 3:20 PM, Dave Fisher 
> wrote:
>
>> In item #3 of the release process two additional checks are needed to
>> help with voting.
>>
>> (a) The output for the RAT report needs to be examined to make sure that
>> all of the source has the proper headers. That any excluded files are
>> properly explained.
>>
>
> Sure, right now the RAT check is performed in every CI build (to make sure
> that Pull Requests are validated before getting merged). In any case, the
> -src.tar.gz package content would be slightly different from the git
> repository workdir, so it's a good idea to check it at the release.
>
> I have updated the step #3 to add the "mvn apache-rat:check" verification.
> As for the excluded files, the current list is in:
> https://github.com/apache/incubator-pulsar/blob/master/pom.xml#L635
>
>
>> (b) The two release artifacts need to have hashes generated so that users
>> can be sure that they downloaded the genuine package. I believe that MD5
>> and SHA256 are now typical ...
>>
>
> Yes, I have included that in the following step: "4. Sign and stage the
> artifacts"
> There is a small shell script that creates checksum and sign all the
> tar.gz files.
>
> https://github.com/apache/incubator-pulsar/blob/master/src/sign-release.sh
>
>

[VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

This is the first release candidate for Apache Pulsar, version
1.19.0-incubating.

Major changes included in this release are:
 * Added stateless Pulsar proxy
 * Support for non-persistent topics
 * Upgraded RocksDB to comply with ASF policy
 * Instrumentation of ZooKeeper client to expose metrics
 * Fixes for TLS auth in WebSocket proxy


Complete list of changes can be found at:

https://github.com/apache/incubator-pulsar/milestone/8?closed=1

*** Please download, test and vote by August 3th 2017, 15:00 PDT.

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/incubator/pulsar/pulsar-1.19.0-incubating-candidate-0/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1000/

The tag to be voted upon:
v1.19.0-incubating-candidate-0 (5125279b50f7a3da8c211b698580e1d2f4dd65e2)

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS

Please download the the source package, and follow the README to build
and run the Pulsar standalone service.



--
Matteo Merli

Re: Aug Podling report - Draft

[Matteo Merli]

Draft looks good to me.

Just one note: the release in the working is "1.19.0-incubating"

--
Matteo Merli


On Mon, Jul 31, 2017 at 3:02 PM, Joe F  wrote:

> All,
>
> It's that time of the month again, the report is due by Aug 2.
> Let me know your comments.
>
> Cheers,
> Joe
>
>
> Pulsar
> ===
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> Pulsar has been incubating since 2017-06-01.
>
> Three most important issues to address in the move towards graduation:
>
>   1.Make an Apache Release
>   2.Set up a test cluster to be able to run system tests
>   3.Grow the community with new Committers/PPMC members.
>
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
>
> None.
>
> How has the community developed since the last report?
>
> The community added one committer and one contributor. There is a healthy
> discuss on issues related to development, tools and processes among the
> community members.
>
> How has the project developed since the last report?
>
> 6 authors have pushed 21 commits to master and 31 commits to all branches.
> The first release from the incubator is in process. A Release Candidate is
> up for vote,  scheduled to close on Aug 3.
> The project has completed setting up its website and project documentation.
> The community has added a slack channel.
>
> How would you assess the podling's maturity?
>
> The podling is progressing well on its way to it's first Apache release (v
> 1.18). There is good participation in governance and in contributions.
>
> Please feel free to add your own commentary.
>
>   [ ] Initial setup
>   [x] Working towards first release
>   [x] Community building
>   [ ] Nearing graduation
>   [ ] Other:
>
> Date of last release:
>
>   2017-06-17, v 1.18
>
> When were the last committers or PPMC members elected?
>
> 2017-07-11
>
> Signed-off-by:
>
>   [ ](pulsar) Dave Fisher
>  Comments:
>   [ ](pulsar) Jim Jagielski
>  Comments:
>   [ ](pulsar) P. Taylor Goetz
>  Comments:
>   [ ](pulsar) Francis Liu
>  Comments:
>
> IPMC/Shepherd notes:
>

Re: [DISCUSS][VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

On Mon, Jul 31, 2017 at 4:53 PM, Dave Fisher  wrote:

> Hi Pulsar Team,
>
> I’ve been doing my checks so far so good. I do have two concerns:
>
> The license for Aspect J is the EPL 1.0 which is Category B -
> https://www.apache.org/legal/resolved.html#category-b
>
> Is this optional?
>

We use AspectJ to instrument ZooKeeper server and client, in order to
collect and expose metrics that are not available in ZooKeeper itself.
At this point is not optional and it's not possible to turn it off.

I've been reading the Cat-B section and, in my understanding it should be
fine to include it in form of binary dependencies.
Quoting from that page:

>> Software under the following licenses may be included in binary form
within an Apache product if the inclusion is appropriately labeled (see
below)

and

>> By including only the object/binary form, there is less exposed surface
area of the third-party work from which a work might be derived; this
addresses the second guiding principle of this policy.


Also - is everything in the binary’s notice file necessary?
>
> I have a feeling that a DEPENDENCIES file might make sense.


I have created 2 different NOTICE files, one for src and one for the bin
dist package.
I have been following the guide at http://apache.org/legal/
src-headers.html#notice plus I've peaked at what other TLPs are currently
including in NOTICE files.

The bulk of the file comes from :

>> The remainder of the NOTICE file is to be used for required third-party
notices.
>> The NOTICE file may also include copyright notices moved from source
files submitted to the ASF.

Basically, I've interpreted this as to include the NOTICEs from all the
dependencies. In most cases it was just the copyright line. In other cases,
like Netty, the NOTICE file was much bigger.

I'd be glad to slim it down :)




--
Matteo Merli

CI builds on Jenkins

[Matteo Merli]

As we discussed some time back, I have added CI builds in Jenkins.

Jenkins seems to be much more stable compare to Travis and it's bit more
flexible in terms of what we can do. For example, for C++ build we can use
custom Docker image with all the dependencies we require to build the
Pulsar client lib.

I have asked INFRA to disable the mandatory check on Travis (to merge the
PRs) and instead use Jenkins for the same purpose.

The build page links:

https://builds.apache.org/job/pulsar-master/

https://builds.apache.org/job/pulsar-pull-request/

The only minor issue that I can see is that the "Rebuild" button the on the
PR builder is not updating back the status on the Github pull request.
The workaround is to "repush" the branch on github. If there are no
changes, you can do a "git commit --amend" and it will just update the
timestamp of the commit and force a new sha hash. Then push --force to
update the PR.

If anyone knows or wants to take a look for a better solution.. :)




--
Matteo Merli

Re: [DISCUSS][VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

On Wed, Aug 2, 2017 at 8:30 AM, Dave Fisher  wrote:

> > We use AspectJ to instrument ZooKeeper server and client, in order to
> > collect and expose metrics that are not available in ZooKeeper itself.
> > At this point is not optional and it's not possible to turn it off.
>
> If someone took the Source release and built the product then AspectJ
> would be included. This means that they would need to take an EPL 1.0
> license. It is my understanding that this must be avoided.
>
> I think that we can probably include this in the first Incubating release.
> These issues are why we incubate.


I've been checking about Cat-B dependencies. So, we also have Jersey jars
which is under CDDL (again Cat-B). All the Servlet and Rest services API
from Java are all licensed with CDDL.

Just doing a quick check, I'm seeing many Apache TLPs are bundling these
dependencies (EPL and CDDL) in the binary releases (and depending on them
in src releases). For example:

Spark : https://github.com/apache/spark/blob/master/NOTICE
Kafka: https://github.com/apache/kafka/blob/trunk/NOTICE
Hadoop: https://github.com/apache/hadoop/blob/trunk/NOTICE.txt
Samza: https://github.com/apache/samza/blob/master/LICENSE



> I would ask if there is another AOP that could be used to expose ZK
> metrics? For example, I wonder what Apache Solr or BookKeeper do.
>

BookKeeper doesn't "bundle" ZooKeeper as well, it just instruct you to have
ZK cluster running. So BK doesn't have this problem because it doesn't deal
it.

In Pulsar, to facilitate the deployment we provide scripts to also start
BookKeeper and ZooKeeper.
And when we run ZooKeeper we instrument with AOP and expose metrics in the
same way as Pulsar broker and BookKeeper.

Re: CI builds on Jenkins

[Matteo Merli]

As most of you have seen, we have a bunch of intermittent test failures.

This, added to the difficulties in rebuilding a pull-request and updating
the status lead to a bit of a cumbersome process to get a PR merged.

I have created issues for the tests I've seen failing and marked them with
label "component/test". The full list is at
https://github.com/apache/incubator-pulsar/issues?q=is%3Aissue+is%3Aopen+label%3Acomponent%2Ftest

Please anyone that has time to investigate these, take one, assign to
yourself and try to reproduce. Fortunately, in new builds we now have
individual INFO logs for each of the test, so that it might help in the
debugging.



--
Matteo Merli


On Tue, Aug 1, 2017 at 9:40 AM, Matteo Merli  wrote:

> As we discussed some time back, I have added CI builds in Jenkins.
>
> Jenkins seems to be much more stable compare to Travis and it's bit more
> flexible in terms of what we can do. For example, for C++ build we can use
> custom Docker image with all the dependencies we require to build the
> Pulsar client lib.
>
> I have asked INFRA to disable the mandatory check on Travis (to merge the
> PRs) and instead use Jenkins for the same purpose.
>
> The build page links:
>
> https://builds.apache.org/job/pulsar-master/
>
> https://builds.apache.org/job/pulsar-pull-request/
>
> The only minor issue that I can see is that the "Rebuild" button the on
> the PR builder is not updating back the status on the Github pull request.
> The workaround is to "repush" the branch on github. If there are no
> changes, you can do a "git commit --amend" and it will just update the
> timestamp of the commit and force a new sha hash. Then push --force to
> update the PR.
>
> If anyone knows or wants to take a look for a better solution.. :)
>
>
>
>
> --
> Matteo Merli
> 
>

Re: CI builds on Jenkins

[Matteo Merli]

Reading on the Jenkins Github plugin documentation at
https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin

You can trigger the build of a PR by adding a comment "retest this please"
on the PR itself. This will kick a new build that will properly update the
PR status.

This will work for all committers.



--
Matteo Merli


On Wed, Aug 2, 2017 at 10:24 AM, Matteo Merli  wrote:

> As most of you have seen, we have a bunch of intermittent test failures.
>
> This, added to the difficulties in rebuilding a pull-request and updating
> the status lead to a bit of a cumbersome process to get a PR merged.
>
> I have created issues for the tests I've seen failing and marked them with
> label "component/test". The full list is at
> https://github.com/apache/incubator-pulsar/issues?q=is%
> 3Aissue+is%3Aopen+label%3Acomponent%2Ftest
>
> Please anyone that has time to investigate these, take one, assign to
> yourself and try to reproduce. Fortunately, in new builds we now have
> individual INFO logs for each of the test, so that it might help in the
> debugging.
>
>
>
> --
> Matteo Merli
> 
>
> On Tue, Aug 1, 2017 at 9:40 AM, Matteo Merli  wrote:
>
>> As we discussed some time back, I have added CI builds in Jenkins.
>>
>> Jenkins seems to be much more stable compare to Travis and it's bit more
>> flexible in terms of what we can do. For example, for C++ build we can use
>> custom Docker image with all the dependencies we require to build the
>> Pulsar client lib.
>>
>> I have asked INFRA to disable the mandatory check on Travis (to merge the
>> PRs) and instead use Jenkins for the same purpose.
>>
>> The build page links:
>>
>> https://builds.apache.org/job/pulsar-master/
>>
>> https://builds.apache.org/job/pulsar-pull-request/
>>
>> The only minor issue that I can see is that the "Rebuild" button the on
>> the PR builder is not updating back the status on the Github pull request.
>> The workaround is to "repush" the branch on github. If there are no
>> changes, you can do a "git commit --amend" and it will just update the
>> timestamp of the commit and force a new sha hash. Then push --force to
>> update the PR.
>>
>> If anyone knows or wants to take a look for a better solution.. :)
>>
>>
>>
>>
>> --
>> Matteo Merli
>> 
>>
>
>

Re: [VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

Remainder for all committers:

please check the release artifacts and vote on this thread, before tomorrow
3pm PDT.

Vote as in :

[ ] +1 Release this package
[ ] 0 I don't feel strongly about it, but don't object
[ ] -1 Do not release this package because...


The documentation on the release process and what needs to be included can
be found at:
   - https://incubator.apache.org/guides/releasemanagement.html
   - http://www.apache.org/legal/release-policy.html
   - http://www.apache.org/dev/release-distribution.html
   - http://www.apache.org/dev/release-publishing.html
   - http://www.apache.org/dev/publishing-maven-artifacts.html

--
Matteo Merli


On Mon, Jul 31, 2017 at 2:39 PM, Matteo Merli  wrote:

> This is the first release candidate for Apache Pulsar, version
> 1.19.0-incubating.
>
> Major changes included in this release are:
>  * Added stateless Pulsar proxy
>  * Support for non-persistent topics
>  * Upgraded RocksDB to comply with ASF policy
>  * Instrumentation of ZooKeeper client to expose metrics
>  * Fixes for TLS auth in WebSocket proxy
>
>
> Complete list of changes can be found at:
>
> https://github.com/apache/incubator-pulsar/milestone/8?closed=1
>
> *** Please download, test and vote by August 3th 2017, 15:00 PDT.
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> pulsar-1.19.0-incubating-candidate-0/
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1000/
>
> The tag to be voted upon:
> v1.19.0-incubating-candidate-0 (5125279b50f7a3da8c211b698580e1d2f4dd65e2)
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
>
> Please download the the source package, and follow the README to build
> and run the Pulsar standalone service.
>
>
>
> --
> Matteo Merli
> 
>

Re: CI builds on Jenkins

[Matteo Merli]

Another option would be to make the Jenkins build status non-mandatory for
merging the PRs.

In this case, if we see any known flaky tests, we could proceed with
merging. For any new failing test, we
should re-run the build and create a new issue to track it.

Pros:
   * We don't have to spend big amount of time rebuilding the same PRs to
merge

Cons:
 * We'll get acquainted with the failing tests and possibly never fix them.

Any thoughts or preferences?



--
Matteo Merli


On Wed, Aug 2, 2017 at 12:22 PM, Matteo Merli  wrote:

> Reading on the Jenkins Github plugin documentation at
> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
>
>
> You can trigger the build of a PR by adding a comment "retest this please"
> on the PR itself. This will kick a new build that will properly update the
> PR status.
>
> This will work for all committers.
>
>
>
> --
> Matteo Merli
> 
>
> On Wed, Aug 2, 2017 at 10:24 AM, Matteo Merli  wrote:
>
>> As most of you have seen, we have a bunch of intermittent test failures.
>>
>> This, added to the difficulties in rebuilding a pull-request and updating
>> the status lead to a bit of a cumbersome process to get a PR merged.
>>
>> I have created issues for the tests I've seen failing and marked them
>> with label "component/test". The full list is at
>> https://github.com/apache/incubator-pulsar/issues?q=is%3Aiss
>> ue+is%3Aopen+label%3Acomponent%2Ftest
>>
>> Please anyone that has time to investigate these, take one, assign to
>> yourself and try to reproduce. Fortunately, in new builds we now have
>> individual INFO logs for each of the test, so that it might help in the
>> debugging.
>>
>>
>>
>> --
>> Matteo Merli
>> 
>>
>> On Tue, Aug 1, 2017 at 9:40 AM, Matteo Merli  wrote:
>>
>>> As we discussed some time back, I have added CI builds in Jenkins.
>>>
>>> Jenkins seems to be much more stable compare to Travis and it's bit more
>>> flexible in terms of what we can do. For example, for C++ build we can use
>>> custom Docker image with all the dependencies we require to build the
>>> Pulsar client lib.
>>>
>>> I have asked INFRA to disable the mandatory check on Travis (to merge
>>> the PRs) and instead use Jenkins for the same purpose.
>>>
>>> The build page links:
>>>
>>> https://builds.apache.org/job/pulsar-master/
>>>
>>> https://builds.apache.org/job/pulsar-pull-request/
>>>
>>> The only minor issue that I can see is that the "Rebuild" button the on
>>> the PR builder is not updating back the status on the Github pull request.
>>> The workaround is to "repush" the branch on github. If there are no
>>> changes, you can do a "git commit --amend" and it will just update the
>>> timestamp of the commit and force a new sha hash. Then push --force to
>>> update the PR.
>>>
>>> If anyone knows or wants to take a look for a better solution.. :)
>>>
>>>
>>>
>>>
>>> --
>>> Matteo Merli
>>> 
>>>
>>
>>
>

Re: Pulsar End to End Encryption - Design review

[Matteo Merli]

(Did receive it the first time too ;) )

Thanks Andrews, the proposal seems very good and well detailed. I'll take a
thorough look at it later today.

On Thu, Aug 3, 2017 at 8:14 AM Sahaya Andrews  wrote:

> Resending since my yesterday's email did not seems to have delivered.
>
>
> https://github.com/apache/incubator-pulsar/wiki/PIP-4:-Pulsar-End-to-End-Encryption
>
> Please go over and provide your Feedback/comments.
>
> Thanks,
> Andrews.
>

Feedback on release

[Matteo Merli]

[ Starting a new thread to detach from vote ]


> Masakazu Kitajo to dev 5:53 AM
> +0
>
> Ran standalone server on macOS 10.12 / Java 8, with Java client.
>

Thanks Masakazu for testing thoroughly!

> I saw lots of warnings and a few errors during building from the source
> package, I was able to run standalone server and successfully transferred
a
> message with pulsar-client though. It seems they are not critical. I'll
> create issues for each warnings and errors later.

Yes, there are many warnings during the build and they've been there for
quite
a while. I suspect at least few of them will cause some problem in the
future
when moving to Java 9.
Please go ahead and create issues for all of them (I think there are also
some
javadoc related ones). We should try to get them fixed (at least most of
them)
for the next release.

> Also, I followed Getting started on our documentation, however, the steps
> to build are only on README file. It's not user friendly. We should
improve
> it before announcing the release.

You're right, in the page we have the links to download the "src" and "bin"
release,
but all the remainder is relative to the "bin" release.

What do people would prefer there? Dropping the link to src release in the
getting
started page or add instructions to compile?

Matteo
-- 
Matteo Merli

Re: Feedback on release

[Matteo Merli]

> Sure. Build, installation and run instructions are critical. I know you
have bits of these in the GitHub wiki.

Yes, in the top level README.md file there are the instruction to compile
and start the standalone server. Though I've avoided to include more info
than that and just linked back to website.

In the getting started first page (
https://pulsar.incubator.apache.org/docs/latest/getting-started/LocalCluster/
),
there are instructions to download both src and bin packages.

The instrutions, though, don't mention to compile it:

# Source release
$ tar xvf pulsar-1.18-src.tar.gz
$ cd pulsar-1.18
-

We should mention the command to compile. Also, probably we should just
have 2 subsctions there: src release and bin release.




On Thu, Aug 3, 2017 at 4:59 PM Dave Fisher  wrote:

>
> > On Aug 3, 2017, at 4:57 PM, Sahaya Andrews  wrote:
> >
> >>> What do people would prefer there? Dropping the link to src release in
> the
> >>> getting
> >>> started page or add instructions to compile?
> >
> > Can we add the link and then below that add a link to the instructions
> > in the README?
>
> Sure. Build, installation and run instructions are critical. I know you
> have bits of these in the GitHub wiki.
>
> Regards,
> Dave
>
> >
> >
> >
> >
> > On Thu, Aug 3, 2017 at 4:49 PM, Dave Fisher 
> wrote:
> >> Hi -
> >>
> >> It is important to keep in mind that the official release of an Apache
> project is a source release and that the binary packages are convenience.
> >>
> >> I would say that the instructions need to be available for both types
> of downloads. The source needs to be as easy or easier to find than the
> binary.
> >>
> >> I’m willing to test any instructions. I’m on a Mac Sierra 10.12.6.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Aug 3, 2017, at 4:22 PM, Matteo Merli  wrote:
> >>>
> >>> [ Starting a new thread to detach from vote ]
> >>>
> >>>
> >>>> Masakazu Kitajo to dev 5:53 AM
> >>>> +0
> >>>>
> >>>> Ran standalone server on macOS 10.12 / Java 8, with Java client.
> >>>>
> >>>
> >>> Thanks Masakazu for testing thoroughly!
> >>>
> >>>> I saw lots of warnings and a few errors during building from the
> source
> >>>> package, I was able to run standalone server and successfully
> transferred
> >>> a
> >>>> message with pulsar-client though. It seems they are not critical.
> I'll
> >>>> create issues for each warnings and errors later.
> >>>
> >>> Yes, there are many warnings during the build and they've been there
> for
> >>> quite
> >>> a while. I suspect at least few of them will cause some problem in the
> >>> future
> >>> when moving to Java 9.
> >>> Please go ahead and create issues for all of them (I think there are
> also
> >>> some
> >>> javadoc related ones). We should try to get them fixed (at least most
> of
> >>> them)
> >>> for the next release.
> >>>
> >>>> Also, I followed Getting started on our documentation, however, the
> steps
> >>>> to build are only on README file. It's not user friendly. We should
> >>> improve
> >>>> it before announcing the release.
> >>>
> >>> You're right, in the page we have the links to download the "src" and
> "bin"
> >>> release,
> >>> but all the remainder is relative to the "bin" release.
> >>>
> >>> What do people would prefer there? Dropping the link to src release in
> the
> >>> getting
> >>> started page or add instructions to compile?
> >>>
> >>> Matteo
> >>> --
> >>> Matteo Merli
> >>> 
> >>
>
> --
Matteo Merli

Re: [VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

Closing the vote.

Link to this thread:
https://lists.apache.org/thread.html/396ffc9c8fe171bc272691414a39307e0929d40639d9ec631ea3aa15@%3Cdev.pulsar.apache.org%3E

Binding votes:

+1s :
  Matteo Merli
  Dave Fisher
  Rajan Dhabalia
  Nozomi Kurihara
  Masahiro Sakamoto
  Yuki Shiga
  Hiroyuki Sakai
  Sahaya Andrews
  Jai Asher

+0s :
  Masakazu Kitajo

Non-Binding votes:

+1s:
  Sijie Guo
  Jia Zhai


Thanks all for testing and validating the release.
I'll start the vote thread on the general@ mailing list.

Matteo


On Thu, Aug 3, 2017 at 4:26 PM Jai Asher  wrote:

> +1 Release!
>
> - Checked build success
> - Ran the binaries on my local machine
>
> Jai
>
> On Thu, Aug 3, 2017 at 4:19 PM, Sahaya Andrews  wrote:
>
> > +1
> >
> > Environment: RHEL 6.4
> > - Verified checksum
> > - Ran local build against the src distribution
> > - Ran standalone server from the binary distribution
> > - Ran perf producer and consumer test against locally built binary and
> > binary distribution
> >
> > I did not see any errors in producer/consumer/pulsar logs.
> >
> > Thanks,
> > Andrews.
> >
> >
> > On Thu, Aug 3, 2017 at 5:53 AM, Masakazu Kitajo 
> wrote:
> > > +0
> > >
> > > Ran standalone server on macOS 10.12 / Java 8, with Java client.
> > >
> > > I saw lots of warnings and a few errors during building from the source
> > > package, I was able to run standalone server and successfully
> > transferred a
> > > message with pulsar-client though. It seems they are not critical. I'll
> > > create issues for each warnings and errors later.
> > >
> > > Also, I followed Getting started on our documentation, however, the
> steps
> > > to build are only on README file. It's not user friendly. We should
> > improve
> > > it before announcing the release.
> > >
> > > Committers, it would be great if you could write down about your
> > > environment (e.g. OS, Java version, compiler, etc) when you cast a
> vote.
> > > Since we support Linux and macOS (and a few languages for clients), we
> > > should confirm that the new version works on the environments we
> support.
> > >
> > > Thanks,
> > > Masakazu
> > >
> > >
> > >
> > > On Thu, Aug 3, 2017 at 4:47 AM, Matteo Merli 
> wrote:
> > >
> > >> Remainder for all committers:
> > >>
> > >> please check the release artifacts and vote on this thread, before
> > tomorrow
> > >> 3pm PDT.
> > >>
> > >> Vote as in :
> > >>
> > >> [ ] +1 Release this package
> > >> [ ] 0 I don't feel strongly about it, but don't object
> > >> [ ] -1 Do not release this package because...
> > >>
> > >>
> > >> The documentation on the release process and what needs to be included
> > can
> > >> be found at:
> > >>- https://incubator.apache.org/guides/releasemanagement.html
> > >>- http://www.apache.org/legal/release-policy.html
> > >>- http://www.apache.org/dev/release-distribution.html
> > >>- http://www.apache.org/dev/release-publishing.html
> > >>- http://www.apache.org/dev/publishing-maven-artifacts.html
> > >>
> > >> --
> > >> Matteo Merli
> > >> 
> > >>
> > >> On Mon, Jul 31, 2017 at 2:39 PM, Matteo Merli 
> > wrote:
> > >>
> > >> > This is the first release candidate for Apache Pulsar, version
> > >> > 1.19.0-incubating.
> > >> >
> > >> > Major changes included in this release are:
> > >> >  * Added stateless Pulsar proxy
> > >> >  * Support for non-persistent topics
> > >> >  * Upgraded RocksDB to comply with ASF policy
> > >> >  * Instrumentation of ZooKeeper client to expose metrics
> > >> >  * Fixes for TLS auth in WebSocket proxy
> > >> >
> > >> >
> > >> > Complete list of changes can be found at:
> > >> >
> > >> > https://github.com/apache/incubator-pulsar/milestone/8?closed=1
> > >> >
> > >> > *** Please download, test and vote by August 3th 2017, 15:00 PDT.
> > >> >
> > >> > Note that we are voting upon the source (tag), binaries are provided
> > for
> > >> > convenience.
> > >> >
> > >> > Source and binary files:
> > >> > https://dist.apache.org/repos/dist/dev/incubator/pulsar/
> > >> > pulsar-1.19.0-incubating-candidate-0/
> > >> >
> > >> > Maven staging repo:
> > >> > https://repository.apache.org/content/repositories/
> > orgapachepulsar-1000/
> > >> >
> > >> > The tag to be voted upon:
> > >> > v1.19.0-incubating-candidate-0 (5125279b50f7a3da8c211b698580e1
> > >> d2f4dd65e2)
> > >> >
> > >> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > >> > https://dist.apache.org/repos/dist/release/incubator/pulsar/KEYS
> > >> >
> > >> > Please download the the source package, and follow the README to
> build
> > >> > and run the Pulsar standalone service.
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Matteo Merli
> > >> > 
> > >> >
> > >>
> >
>
-- 
Matteo Merli

Re: [VOTE] Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

Thanks Francis,

On Thu, Aug 3, 2017 at 5:39 PM Francis Liu  wrote:

> Ran mvn clean install on my macbook pro (sierra) twice and it failed
> during unit testing. Are these just flakey? (jdk 1.8.0_45-b14)
> DiscoveryServiceTest.testClientServerConnectionTls:149 expected [true] but
> found [false]
>

This test was flakey, but in theory should have been fixed long ago in
https://github.com/apache/incubator-pulsar/issues/116
I haven't seen it failing in a long time.

org.apache.pulsar.broker.admin.AdminApiTest.testIncrementPartitionsOfTopic(org.apache.pulsar.broker.admin.AdminApiTest)
> Run 1: AdminApiTest.

...

on AdminApiTest there is testIncrementPartitionsOfTopic which a flaky test
for which we have an issue open at:
https://github.com/apache/incubator-pulsar/issues/635

This test causes the subsequent tests on the same class to fail as well.


> Also ran mvn rat:check. Was wondering if this file needs a header:
> dashboard/conf/uwsgi_params?
>

Shouldn't that be "apache-rat:check" the official tool? We run it in CI
builds and I've run it multiple times on src package as well and it should
work.
For "dashboard/conf/uwsgi_params" I wasn't sure if the uwsgi tools were
handling comments correctly. I will test it out and and add headers if it
works.
Would it be ok to fix in next release?



Matteo
-- 
Matteo Merli

Re: Feedback on release

[Matteo Merli]

Created issue https://github.com/apache/incubator-pulsar/issues/639

Additionally, we will have to fix the instructions once the 1.19 release
would be available, to point to the Apache dist and mirrors.

Once the release is (eventually) in place, we'll update the website prior
to release announcement.

Matteo

On Fri, Aug 4, 2017 at 6:06 AM Masakazu Kitajo  wrote:

>
> --
Matteo Merli

Fwd: [RESULT] [VOTE] Apache Pulsar 1.19.0-incubating Release Candidate 0

[Matteo Merli]

I will proceed now in publishing the artifacts.

Before announcing the release, I'll wait for the artifacts to be replicated
in the mirrors and I'll send out PRs to update the download page and to
"snapshot" the 1.19 documentation pages.


Thank you,
Matteo


-- Forwarded message -----
From: Matteo Merli 
Date: Tue, Aug 8, 2017 at 4:24 PM
Subject: [RESULT] [VOTE] Apache Pulsar 1.19.0-incubating Release Candidate 0
To: gene...@incubator.apache.org 


The vote for releasing Apache Pulsar 1.19.0-incubating is now closed.

With a total of +5 binding votes and no -1 votes, the vote passes.

+1s (binding):
 Dave Fisher
 Josh Elser
 Francis Liu
 P. Taylor Goetz
 Justin Mclean

Thank you to all the reviewers for taking the time to validate this release
and provide very good feedback. We will make sure to incorporate all
the suggested changes in the next release.

Matteo

-- 
Matteo Merli

-- 
Matteo Merli

[ANNOUNCE] Apache Pulsar 1.19.0-incubating released

[Matteo Merli]

The Apache Pulsar team is proud to announce Apache Pulsar version
1.19.0-incubating.

This is the first Pulsar release after entering the Apache Incubator.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:
https://pulsar.incubator.apache.org/download

Release Notes are at:
https://github.com/apache/incubator-pulsar/releases/tag/v1.19.0-incubating

We would like to thank the contributors that made the release possible.


Regards,
The Pulsar Team

Re: Committer info in commits

[Matteo Merli]

Hi Masakazu,

> Look at the second commit. The commit is made by  GitHub <
nore...@github.com>

I think that this happens when the author of the commit is pushing the
"merge" button on its own commit (after having received +1s from other
committers)..

In this case:

> commit 2b63573eea220a8710a93d87f451188e0f253b22
> Author: Matteo Merli 
> Commit: GitHub 

I submitted the PR and then merged it. Though if I merge someone else's PR,
my name/email will appear in the "Commit" field. eg:

> commit de24aaf11ce61f8af28bb1dd70c099a8e43d0f08
> Refs:
> Author: hrsakai 
> AuthorDate: Wed Aug 9 15:28:04 2017 +0900
> Commit: Matteo Merli 
> CommitDate: Tue Aug 8 23:28:04 2017 -0700
>
> support tls in performance tool (#662)

I think that should be fine, given that is clear who authored the change
and who made the commit.

Having said that, I would suggest for all committers to use the proper
"first-name last-name" and apache
email in the commit signatures. Also, please add that in Github as well, so
that it will be used when
you merge PRs from the github interface.

Matteo




On Tue, Aug 8, 2017 at 5:26 AM Masakazu Kitajo  wrote:

> Committers,
>
> It seems some commits doesn't have a valid committer name. It's not cool.
>
> I couldn't find any official rules but some (most?) projects require to use
> your email address as a committer when you push commits. We should probably
> do this. IIRC, you can set the address at GitHub Settings - Email - Primary
> email address, and it need to be public.
>
> Look at the second commit. The commit is made by  GitHub <
> nore...@github.com
> >.
>
> $ git log --pretty=full
>
> commit d2ab2b599210dff5d2551a3cab875a8d7ef4a3a0 (HEAD -> master,
> apache/master)
> Author: Luc Perkins 
> Commit: Masakazu Kitajo 
>
> Add link checker and fix broken links (#647)
>
> * add htmltest link checker setup steps and fix a few links
>
> * add license header to new script
>
> * add license header to htmltest YAML config file
>
> * fix Makefile and some broken links
>
> commit 2b63573eea220a8710a93d87f451188e0f253b22
> Author: Matteo Merli 
> Commit: GitHub 
>
> Build website using the current user rather than root (#652)
>
-- 
Matteo Merli

Re: Rebase and merge

[Matteo Merli]

I am ok in having rebased pull-requests and to allow the "Rebase & Merge"
option in github.
That can be helpful in cases in which there are multiple logical (though
related) commits for
which you want to keep separation.
For most cases (eg: PRs comments and incremental fixes on the PR) I think
the squash option
is still better, because the history of the PR is anyway available, with
all the comments and commits.

My only concern is that once you chose that option and "rebase & merge" one
PR , it becomes your default, until you change it back.
So one has to really pay attention to not forget to go back to the squash
option.

One other option would be to have a customized script to merge PRs. This
can help in enforce a uniform style in the
commit logs (as well as adding the name of the reviewers). For example,
this is what we use in BookKeeper:
https://github.com/apache/bookkeeper/blob/master/dev/bk-merge-pr.py

It's a bit more involved than just hitting "Squash & merge" on github, but
nothing really major.

What do other people think about this?


Matteo

On Tue, Aug 8, 2017 at 4:50 AM Masakazu Kitajo  wrote:

> Hi,
>
> I'd like to propose enabling "Rebase and merge"[1] as an option to merge
> pull requests.
>
> Currently, we have only one option, "Squash and merge". However, I'm not a
> big fun of this option because it makes commit history too clean. Even if
> you made several meaningful commits on your branch, they would be squashed
> when we merge the pull-request.
>
> What if we want to revert a part of the changes?
> What if we want to cherry-pick a part of changes?
>
> These sometimes happen, and in general, it's a good practice to keep
> commits small.
>
> As long as commits in a pull request seem meaningful, I think we should
> keep the commits separated.
>
> What do you think?
>
> [1] https://github.com/blog/2243-rebase-and-merge-pull-requests
>
> Masakazu
>
-- 
Matteo Merli

[DISCUSS] Planning for next release 1.20.0-incubating

[Matteo Merli]

Since the first release is out, I'd like to get started on the discussion
for the next one :)

I propose to try to keep a monthly cadence for releases so that we can
quickly iterate over new feature and bug fixes, and avoid having to
create to many patch releases in between, which can be time consuming.

Also, I'd like to get some planning upfront, for determine the date and
then each contributor can comment on which major changes he wants to
target for inclusion in next release.

That should give a good level of transparency and clarity on the
expectations.

Finally, we should start a rotation between committers for the role of
"release manager", the one who will help coordinate the whole process
of the release.

Since I've already done the first release, who wants to do next?


Matteo
-- 
Matteo Merli

[PROPOSAL] Move github notifications to a separate mailing list

[Matteo Merli]

As you have surely noticed already, all the github notifications are being
sent to
this mailing list (dev@pulsar.incubator.apache.org).
This results in a big amount of mails sent to the list (one for each
comment made on issues or PRs and also when commits are pushed to the PRs).

I would propose to use a different mailing list for these notification, in
order to
split the "real" discussions on the dev list from the noise of the
notiifcations.

Most people are already receiving notifications from github anyway, and you
can
always subscribe to both lists and receive the same messages as now.

The question would be which mailing list should we be using for that?
 * comm...@pulsar.incubator.apache.org ? This is already existing and
mostly unused (just SVN dist commits notifications are sent here)
 * or create a new list, like iss...@pulsar.incubator.apache.org or
git...@pulsar.incubator.apache.org ?

I think that, from the ASF perspective, this shouldn't pose any problem,
since the primary object is to archive these messages in an ASF list, not
necessarily dev@.
Mentors, please comment on this aspect.


Thanks,
Matteo



-- 
Matteo Merli