Re: civetweb upstream/downstream divergence

[Ken Dreyer]

On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil  wrote:
> On Tue, 3 Nov 2015, Nathan Cutler wrote:
>> IMHO the first step should be to get rid of the evil submodule. Arguably
>> the most direct path leading to this goal is to simply package up the
>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
>> the supported distros. The resulting package would be Ceph-specific,
>> obviously, so it could be called "civetweb-ceph".
>>
>> Like Ken says, the upstreaming effort can continue in parallel.
>
> I'm not sure I agree.  As long as everything is not upstream and we are
> running a fork, what is the value of having it in a separate package?
> That just means all of the effort of managing the package dependency and
> making sure it is in all of the appropriate distros (and similar pain for
> those building manually) without any of the benefits (upstream bug fixes,
> etc.).

I think there's value in getting the packaging bits ready ahead of
time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
continue to merge Ceph's civetweb changes to Civetweb upstream.

Now that Civetweb with RGW is mainstream, I'm looking forward to
eventually using a pre-built civetweb package that can shave time off
our Ceph Gitbuilder/Jenkins runs :)

- Ken
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Martin Millnert]

On Wed, 2015-11-04 at 16:43 -0700, Ken Dreyer wrote:
> When I was talking about a "parallel effort", what I meant is that
> we'd get vanilla civetweb upstream into the distros, and we'd also
> continue to bundle civetweb in Ceph, until we can reliably use the
> upstream Civetweb package.

That's what sounds logical to me too.
So that aside, [Sage's] concern to this could probably be reduced to:
 1) getting new code out to clients via distro packages is slow,
 2) getting new code into upstream projects is tricky

2) as I understand it, has shown tricky for, for example for Sage, with
some FS projects, at least. And there's probably a risk to fall back
into the urge to fork again in the future if.
1) is probably a minor concern, at least simply considering how we're
deploying packages currently (as ceph operator). If you want the latest
and greatest, you solve your dependencies... If you run distro packages,
well, you're on distro packages already and they are packaged for
compatibility as-is. Maintainers of Ceph distro packages would need to
assure they work with civetweb packages.

Ultimately it's a question of whether or not the added labor of keeping
a fork is worth the benefit it brings in reduction of external
dependencies when deploying. The downside is, as in this case, risk of
lagging behind if fork isn't kept up-to-date.

Not obvious which is better.
/M

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Ken Dreyer]

On Wed, Nov 4, 2015 at 1:25 PM, Ken Dreyer  wrote:
> On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil  wrote:
>> On Tue, 3 Nov 2015, Nathan Cutler wrote:
>>> IMHO the first step should be to get rid of the evil submodule. Arguably
>>> the most direct path leading to this goal is to simply package up the
>>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
>>> the supported distros. The resulting package would be Ceph-specific,
>>> obviously, so it could be called "civetweb-ceph".
>>>
>>> Like Ken says, the upstreaming effort can continue in parallel.
>>
>> I'm not sure I agree.  As long as everything is not upstream and we are
>> running a fork, what is the value of having it in a separate package?
>> That just means all of the effort of managing the package dependency and
>> making sure it is in all of the appropriate distros (and similar pain for
>> those building manually) without any of the benefits (upstream bug fixes,
>> etc.).
>
> I think there's value in getting the packaging bits ready ahead of
> time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
> continue to merge Ceph's civetweb changes to Civetweb upstream.
>
> Now that Civetweb with RGW is mainstream, I'm looking forward to
> eventually using a pre-built civetweb package that can shave time off
> our Ceph Gitbuilder/Jenkins runs :)

Oh, I just re-read this, and Nathan's proposing to package up
"civetweb-ceph" as a fork... I'm not sure that's worth it (at least,
speaking for packaging in Fedora).

When I was talking about a "parallel effort", what I meant is that
we'd get vanilla civetweb upstream into the distros, and we'd also
continue to bundle civetweb in Ceph, until we can reliably use the
upstream Civetweb package.

- Ken
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Sage Weil]

On Wed, 4 Nov 2015, Ken Dreyer wrote:
> On Wed, Nov 4, 2015 at 1:25 PM, Ken Dreyer  wrote:
> > On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil  wrote:
> >> On Tue, 3 Nov 2015, Nathan Cutler wrote:
> >>> IMHO the first step should be to get rid of the evil submodule. Arguably
> >>> the most direct path leading to this goal is to simply package up the
> >>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
> >>> the supported distros. The resulting package would be Ceph-specific,
> >>> obviously, so it could be called "civetweb-ceph".
> >>>
> >>> Like Ken says, the upstreaming effort can continue in parallel.
> >>
> >> I'm not sure I agree.  As long as everything is not upstream and we are
> >> running a fork, what is the value of having it in a separate package?
> >> That just means all of the effort of managing the package dependency and
> >> making sure it is in all of the appropriate distros (and similar pain for
> >> those building manually) without any of the benefits (upstream bug fixes,
> >> etc.).
> >
> > I think there's value in getting the packaging bits ready ahead of
> > time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
> > continue to merge Ceph's civetweb changes to Civetweb upstream.
> >
> > Now that Civetweb with RGW is mainstream, I'm looking forward to
> > eventually using a pre-built civetweb package that can shave time off
> > our Ceph Gitbuilder/Jenkins runs :)
> 
> Oh, I just re-read this, and Nathan's proposing to package up
> "civetweb-ceph" as a fork... I'm not sure that's worth it (at least,
> speaking for packaging in Fedora).
> 
> When I was talking about a "parallel effort", what I meant is that
> we'd get vanilla civetweb upstream into the distros, and we'd also
> continue to bundle civetweb in Ceph, until we can reliably use the
> upstream Civetweb package.

Ah, this sounds better to me.  There may be some work to build civetweb as 
a shared library (currently it's just a statically linked module) but 
probably not too bad.

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Nathan Cutler]

IMHO the first step should be to get rid of the evil submodule. Arguably
the most direct path leading to this goal is to simply package up the
downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
the supported distros. The resulting package would be Ceph-specific,
obviously, so it could be called "civetweb-ceph".

Like Ken says, the upstreaming effort can continue in parallel.

After we get Ceph/RGW working fine with civetweb-ceph 1.6, we can rebase
the package to upstream civetweb 1.7.

I am not volunteering to do all the work, but we at SUSE are certainly
prepared to shoulder our share of it.

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Sage Weil]

On Tue, 3 Nov 2015, Nathan Cutler wrote:
> IMHO the first step should be to get rid of the evil submodule. Arguably
> the most direct path leading to this goal is to simply package up the
> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
> the supported distros. The resulting package would be Ceph-specific,
> obviously, so it could be called "civetweb-ceph".
> 
> Like Ken says, the upstreaming effort can continue in parallel.

I'm not sure I agree.  As long as everything is not upstream and we are 
running a fork, what is the value of having it in a separate package?  
That just means all of the effort of managing the package dependency and 
making sure it is in all of the appropriate distros (and similar pain for 
those building manually) without any of the benefits (upstream bug fixes, 
etc.).

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Loic Dachary]

Hi Pete,

On 30/10/2015 13:57, Pete Zaitcev wrote:
> On Thu, 29 Oct 2015 10:58:07 -0700
> Yehuda Sadeh-Weinraub  wrote:
> 
>> We should definitely do it. We're based off civetweb 1.6, and there
>> was no official civetweb version for quite a while, but 1.7 was tagged
>> a few months ago. I made some effort and got most of our material
>> changes upstream, however, there are some changes that might need some
>> more work before we can get them merged, or might not make complete
>> sense at all.
> 
> I take it Nathan is volunteering to parse the delta into logical pieces
> and identify what upstream is willing to accept, right?

I've discussed with Nathan about this general problem a few times. The issue is 
much less about volunteering and much more about how to track the progress of 
the delta over time.

> Dunno about SuSE, but as a Fedora packager I would prefer if we (Ceph)
> talked upstream into making regular releases and then for us to stop
> carrying it entirely. One less git submodule if nothing else.

Right now we have no method. For the jerasure / gf-complete sub-modules, I'm 
watching the delta and do the right thing but it's mostly an unwritten process: 
someone else would do it completely differently. For other Ceph sub-modules I 
suppose each developer has his own way of dealing with the delta.I remember 
Sage recently proposed patches upstream for rocksdb but I'm unaware of where or 
how. I would not be able to help him in any way. And I don't think anyone could 
figure out exactly how to deal with the jerasure / gf-complete sub-modules 
either.

Do you happen to know a project that is using submodules (or copies of projects 
instead of dependencies) and that is also well organized to track the delta ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature

civetweb upstream/downstream divergence

[Nathan Cutler]

Hi Ceph:

The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
which is a fork of https://github.com/civetweb/civetweb. The last commit
to our fork took place on March 18.

Upstream civetweb development has progressed ("This branch is 19 commits
ahead, 972 commits behind civetweb:master.")

Are there plans to rebase to a newer upstream version or should we think
more in terms of backporting (to ceph/civetweb.git) from upstream
(civetweb/civetweb.git) when we need to fix bugs or add features?

Thanks and regards

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb upstream/downstream divergence

[Wido den Hollander]

On 29-10-15 10:19, Nathan Cutler wrote:
> Hi Ceph:
> 
> The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
> which is a fork of https://github.com/civetweb/civetweb. The last commit
> to our fork took place on March 18.
> 
> Upstream civetweb development has progressed ("This branch is 19 commits
> ahead, 972 commits behind civetweb:master.")
> 
> Are there plans to rebase to a newer upstream version or should we think
> more in terms of backporting (to ceph/civetweb.git) from upstream
> (civetweb/civetweb.git) when we need to fix bugs or add features?
> 

I think it would be smart to keep tracking civetweb from upstream
otherwise we forked Civetweb.

We might run into some issues with Civetweb which we need to fix
upstream, that's a lot easier if we are close to where upstream is.

Wido

> Thanks and regards
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html