Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-12-10 06:54 +1030, Ron wrote:

> So I've now orphaned this package, and he has my blessing and sympathy
> for being responsible for whatever happens with it from here.  I haven't
> filed a WNPP bug for that as we don't need to offer it to someone random.

OK. Cheers. Warm potato accepted :-)
 
> Wookey: if you want the complete git history, right back to the very first
> package in 1999, you can grab it from the Vcs-Git URL in the sid package.
> I'm not going to go Full Bruce and rage delete it, but eventually I should
> decruft alioth and remove it from there, so if you want it you should
> probably clone it somewhere that works for you.

OK. I don't use git unless I have to, so I'll let Punit worry about that.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Sam Hartman]

> "Colin" == Colin Watson  writes:


Colin> As a maintainer who has sometimes had cause to do similar
Colin> things, I'm concerned at the standard being applied here.
Colin> Could you perhaps review the history around groff 1.18.1.1 ->
Colin> 1.20 for comparison?  This is a case that's all over and done
Colin> with seven years ago, so should be free of emotive
Colin> associations by now, and the history can be read through
Colin> reasonably quickly.

Colin> Here are some references: https://bugs.debian.org/196762
Colin> https://bugs.debian.org/374569
Colin> https://lists.gnu.org/archive/html/groff/2007-11/msg00011.html

Reading this first reference was enough for me to identify several
differences I believe are key.

First, upgrading to new upstream is presumed good, not always good.
My concern with Ron's position is mostly that  he wants the people
requesting a new upstream to justify that rather than wanting the htags
users to justify not breaking htags.

I think it was fairly obvious to everyone involved in the groff
discussion that the multibyte patch was required for Japanese man pages
among other things.
That is, I think there is evidence of the importance of the groff
multibyte patch in a way there's not evidence of heavy htags CGI use
here.
Put another way, the record presents evidence sufficient to overcome the
presumption that upgrading is good.

Second, you were responsive and pro-active.  You reached out to the
multibyte patch author.
You tried to forward port it yourself.

Third, there was an eventual way forward.  It was clear that groff would
eventually get to UTF-8 support good enough that we could use it.

So for these reasons, I think the groff situation is different in ways
that matter at least to my analysis.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Fri, Dec 09, 2016 at 05:13:48PM +0100, Didier 'OdyX' Raboud wrote:
> 
> What I see as fundamental difference here was your use of #196762 as a single 
> point of contact for the problem you were facing with groff 1.19, in which 
> you 
> explained, commented and followed up on what the problem was, what were your 
> thoughts, asking for help, and updating this bug regularly. You also used 
> this 
> single point of contact in later bugs:
 
  ...

> That's really the thing I miss most in 'global's history: it shouldn't need a 
> TC bug to get the 'htags' problem described properly, on a public, standalone 
> bug.

So you're saying "if only #574947 existed which did that"?
Oh wait.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Didier 'OdyX' Raboud]

Le vendredi, 9 décembre 2016, 15.03:14 h CET Colin Watson a écrit :
> On Fri, Dec 09, 2016 at 08:58:10AM -0500, Sam Hartman wrote:
> > However, the time at which Debian has last synced with upstream does
> > matter.
> > Six years is a long time.
> > Moreover, I believe that the standard you've used to evaluate whether
> > failing to sync for six years was acceptable is inconsistent with best
> > practices of the project.
> 
> As a maintainer who has sometimes had cause to do similar things, I'm
> concerned at the standard being applied here.  Could you perhaps review
> the history around groff 1.18.1.1 -> 1.20 for comparison?  This is a
> case that's all over and done with seven years ago, so should be free of
> emotive associations by now, and the history can be read through
> reasonably quickly.

Thank you for the example case, including its references.

> Now, my perception of this case is that there was a complex blocking
> issue with the new upstream release that affected a minority of users,
> and therefore I chose to hold back the new upstream until it could be
> handled in a way that did not cause serious regressions.
> 
> (…)
> 
> If the TC thinks my actions in the past were reasonable, then I would
> like any ruling here to be a bit more nuanced about cases of delayed
> syncing with upstream, reflecting whatever important differences you see
> between the two cases.

By a quick glance, your past actions _were_ indeed reasonable. And commenting 
on what differentiates this example with 'global' is a worthwile exercise.

What I see as fundamental difference here was your use of #196762 as a single 
point of contact for the problem you were facing with groff 1.19, in which you 
explained, commented and followed up on what the problem was, what were your 
thoughts, asking for help, and updating this bug regularly. You also used this 
single point of contact in later bugs:
> Unfortunately, for technical reasons (see bug #196762), it is extremely
> difficult to upgrade to the new upstream release.

All-in-all, although there are appeareances of equivalence between the two 
packages' histories, I certainly see fundamental differences in how the "new 
upstream version has problems that are hard to fix" question was addressed.

There can be valid, strong and solid technical reasons to hold back on new 
upstream versions, but what matters for us as a distribution, is how we 
collaborate and communicate about these blockers. Our SC's "We will not hide 
problems" is not to be understood strictly as "our bugtracker is public" only; 
it is a much larger concern that we should share, so as to make our 
collaboration as frictionless as possible, as well as make the technical 
problems not only reside in the package maintainer's heads, but to share them 
with the project.

That's really the thing I miss most in 'global's history: it shouldn't need a 
TC bug to get the 'htags' problem described properly, on a public, standalone 
bug.

Things like
> (we) are currently discussing changes to the CGI mechanism which may alter
> several things about how this is currently arranged.
in #590053, or the discussion in #574947 where Ron claims to have reached a 
private agreement that is not confirmed by upstream authors are un-
comprehensible for outsiders, which are left without a clue upon what needs to 
be fixed, or done.

The problem I have with how 'global' is maintained is not at all technical. 
Holding back on an old version because of 'htags' breakage _was_ a good 
technical decision back then. But the way this hurdle was (not) documented is 
just not how I want to see maintainers collaborate and "not hide problems" for 
packages in Debian.

The fact that the hurdle is _still not_ properly and publicly documented 
doesn't open room for helping the maintainer, doesn't bring clarity to 
newcomers and therefore puts the maintainer in sole responsiblity. Wei Lui 
expressed that problem clearly in #816924 (March 2016):
> I'm aware of the disagreement between Ron and Shigio, but it's so
> frustrating that no resolution has been found for so many years. There
> were talks about possible designs proposed by Ron and / or Shigio but
> it is just impossible for us users to do archaeology dating back to
> 1999 with no useful references whatsoever.

Another difference I see is this urge you felt. Two months after groff 1.19 
was released, you filed this bug documenting your concerns with it and no less 
than three months later, you wrote:
> I'm beginning to get a bit itchy about falling too far behind upstream
> here.

As a matter of fact, I do _expect_ maintainers to get itchy when they fall too 
far behind upstream. If Ron had documented the problem from the start (or at 
any later point in time, really), users would have found it, and debated ways 
out, patches, etc. This bug would have included upstream developers for a 
discussion _in public_. That bug would have helped joining forces, helped the 
maintainer determine the 

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Fri, Dec 09, 2016 at 11:58:02AM +0100, Didier 'OdyX' Raboud wrote:
> Le vendredi, 9 décembre 2016, 04.55:20 h CET Ron a écrit :
> > > If you haven't yet, I urge you to use our standard interface to report
> > > such
> > > bugs; please make sure issues like this one are public on our bugtracker,
> > > with correct found/notfound version markers.
> > 
> > Do you really want entries in the tracker for buggy code that was never
> > in Debian, because I nacked Punit uploading things he didn't understand
> > with a vague promise to maybe look at them later?
> 
> That code is now in Debian (experimental), so yes, I do expect you to act in 
> good faith and report bugs you see. You are obviously quite versed in how 
> 'global' works, and that's undoubtedly valuable to produce the best possible 
> 'global' package.

No, the code in experimental has that 'fixed', by commenting it out and
inviting the user to uncomment it themselves.

The context for this, was that was the code which was proposed to be
uploaded, and which was last discussed, at the time this was brought
to the ctte.  It never was uploaded to any suite, just published in
collab-maint, and I think Punit provided packages somewhere else.

The code in experimental does have some eye raising things in it, but
nothing that I've yet traced through as being definitely exploitable.
But I also haven't given it a serious audit yet, just eyeballed it
quickly for obvious things.

> > Now we're talking about what to do among a wider group of people, given
> > that it still looks like nothing material will change.  The system works?
> 
> It doesn't: it shouldn't take 3 stable releases to get a new upstream release 
> for a leaf package.

There's a difference between blindly uploading a new upstream and
actually having a solution to the problem which is the reason that
it wasn't.

I made that reason very clear, and invited proposed solutions in
the original 'new upstream' bug, #574947.  Nobody else, except
Taisuke and I ever made any effort to deal with that.

Taisuke and I both considered the secure use of htags to be an
important use case.  But given the time that has gone by, and the
fact that the upstream code in what is currently in experimental
has completely eliminated any possible use from a secure system
location now, and how doxygen's seach facility has improved in
the last couple of years - my opinion has likewise changed in line
with that changed circumstance.

But it's taken all of "3 stable releases" for that to actually
change ...  this wasn't all the case at the time of the freeze
for Jessie, or before.


I still think it would be rude to burn the remaining users on
such short notice - but I don't think we should delay doing that
any longer than the end of the Stretch freeze.  And if there is
sufficient consensus to say "burn them immediately", I've already
said I'm ok with that too.  But I would want there to be a consensus
of people who'd have my back about doing that.  Else we just have
the same situation as we do now, where people abuse me for not doing
exactly what _they_ would have preferred.


> > That report led to both me and the reporter having a (very) long
> > discussion with upstream about how to resolve the real problem that
> > you keep assuming we never tried to do anything about.
> 
> By "(very) long discussion", do you mean these 8 mails ?
> 
>   http://lists.gnu.org/archive/html/bug-global/2010-08/threads.html#6

No.  That was one thread of many.  But aside from what's also in
the BTS, and on -devel (or was it -project?), the vast majority
were private emails, and span many years of trying to move this
forward one way or another.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-12-08 17:03 +0100, Didier 'OdyX' Raboud wrote:

Sorry missed this in all yesterday's mail.

> Wookey, Vincent, Punit: would any of you be willing to receive the 'global' 
> package maintainer hat? (which would of course come with the possibility to 
> share and change the maintainer status afterwards.)

Punit is happy to be maintainer. I am happy to co-maintain with him
(now that I'e gone the trouble of finding out a reasonable amount
about the package).

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-12-08 17:33 +, Wookey wrote:
> On 2016-12-08 23:32 +1030, Ron wrote:

> > But it also outputs a .htaccess enabling execution in the directory
> > where the output is generated, whether you want to use it from there
> > or not (and adds a second CGI, and a bunch of jquery doing AJAX too).
> 
> The world is absolutely full of jquery doing AJAX. That's not bad in
> itself (much as some of us prefered the 'old web'. We should make it
> use system jquery. 

OK. I tried making it use system jquery and some of the icons in htags
seem to go missing. So the internal jquery-treeview and
jquery-suggests may be incompatible. Needs a bit more research, so
we'll ship the internal one in 6.5.5-0.3 for now.
 
> > the icons and javascript and css for htags 

> Good point. 
> That's an easy fix.

Included in 6.5.5-0.3 which I'm about to upload. So that fixes
#587489, (which has only been pending with a patch since May 2011).

(currently shipped in /usr/share/gtags rather than
/use/share/global/gtags because upstream needs patching to sort that
properly. One thing at a time.)

And now htags browsing has nice little icons for moving about. shiny :-)

We checked that 6.5.5 also fixes #847303. It does.

I have a query pending with Ron for if it's OK with him to move to
0-day uploads to experimental as I don't think the conventional NMU
delay is actually helping anyone in this case. It just makes things
slow. I'll just replace the pending 0.2 upload with this 0.3 one for
now.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Sam Hartman]

> "Didier" == Didier 'OdyX' Raboud  writes:


Didier> That code is now in Debian (experimental), so yes, I do
Didier> expect you to act in good faith and report bugs you see. You
Didier> are obviously quite versed in how 'global' works, and that's
Didier> undoubtedly valuable to produce the best possible 'global'
Didier> package.

Ron,  I would prefer that Didier use a different tone.
However, it's my opinion as someone who will be voting on this that
Didier is essentially right that your view of this situation and of the
responsibilities of a Debian maintainer are inconsistent with the
project as a whole.
You have continued to try and frame the discussion in the terms you
would prefer.
I have considered those terms, and I do not find framing the discussion
in those terms compelling.

In the language similar to the  IETF, used only because we're both familiar with
it, the technical issues surrounding global-6 and the question of
evidence regarding htags have been considered.  My judgment of the
discussion is that the rough consensus here is that those issues are not
significant compared to failing to upgrade global in six years.  That
is, we in this discussion have reach an informed opinion that those
issues are not significant enough to block.  It is my opinion you are in
the rough.

I think the question before the TC is (and is properly) how should
global-6 be maintained and whether you are the person to do that.

You have tried to frame it arguing that the version number doesn't
matter.
I absolutely agree.
However, the time at which Debian has last synced with upstream does
matter.
Six years is a long time.
Moreover, I believe that the standard you've used to evaluate whether
failing to sync for six years was acceptable is inconsistent with best
practices of the project.

--Sam

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Shigio YAMAGUCHI]

Hi all,
Please forgive me one more comment.

Mon, 24 Oct 2016 19:31:05 +1030, Ron wrote:
> ... and it may well be that this has actually happened now with
> upstream's decision to drop all support for providing a secure
> system CGI of any form that people can use for this.  The upstream
> code is basically now back to what it was in the 90's, with the only
> way to use this being to allow execution of a generated CGI in the
> same tree as the html content.  Which was already well known to be a
> dangerous and ill advised idiom even back then ...

Apache + system CGI is somewhat overdone to use htags.
GLOBAL is just a source code tagging tool for developers;
it is not a system to publish something to the world.

My answer is htags-server(1), a private http server for htags.
You should invoke this command for each project like this:

$ gtags
$ htags --suggest2
$ htags-server
Please access at http://127.0.0.1:8000
Python2 http/cgi server
Serving HTTP on 127.0.0.1 port 8000 ...

You can see the output of htags through 'http://127.0.0.1:8000'.
It is easy to use, and is safer because it runs with user's
privilege without publishing to the network by default.
This command was added to GLOBAL-6.3 in 2014.

IMO, it is useless to continue supporting system CGI.
It is difficult to set up, and never makes something safer.

Regards,
Shigio
-- 
Shigio YAMAGUCHI 
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3
I don't like FUD.
   -- Anonymous

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Shigio YAMAGUCHI]

Hi,
2016-12-09 18:26 GMT+09:00 Philip Hands :
> > open(PIPE, "-|") || exec '@globalpath@', '--result=ctags-xid',
> > $flags, $pattern;
>
> Is it not the case that this last line forks and execs global, passing
> $pattern as a parameter to global's -e option, and that $pattern is
> untrusted input?

Yes. $patern is untrusted input.

> Looking at global.c it seems that before it is passed on to popen, it is
> run through quote_shell() which quotes any single-quotes in the string.
>
> That seems to deal with Ron's assertion that it's exploitable, although
> I have a slight feeling of impending doom about relying upon just this.

I agree.
I uses popen() in global.c to call idutils(1). I would like to rewrite it
in near future.

> Would it not be wise to make the network-facing perl code runnable with
> strict and taint turned on, if only to stop people reacting with horror
> at first glance?
>
> I presume patches would be welcome?

Of course! Thank you.

Regargs,
Shigio

-- 
Shigio YAMAGUCHI 
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Didier 'OdyX' Raboud]

Le vendredi, 9 décembre 2016, 04.55:20 h CET Ron a écrit :
> > If you haven't yet, I urge you to use our standard interface to report
> > such
> > bugs; please make sure issues like this one are public on our bugtracker,
> > with correct found/notfound version markers.
> 
> Do you really want entries in the tracker for buggy code that was never
> in Debian, because I nacked Punit uploading things he didn't understand
> with a vague promise to maybe look at them later?

That code is now in Debian (experimental), so yes, I do expect you to act in 
good faith and report bugs you see. You are obviously quite versed in how 
'global' works, and that's undoubtedly valuable to produce the best possible 
'global' package.

> Now we're talking about what to do among a wider group of people, given
> that it still looks like nothing material will change.  The system works?

It doesn't: it shouldn't take 3 stable releases to get a new upstream release 
for a leaf package.

> That report led to both me and the reporter having a (very) long
> discussion with upstream about how to resolve the real problem that
> you keep assuming we never tried to do anything about.

By "(very) long discussion", do you mean these 8 mails ?

http://lists.gnu.org/archive/html/bug-global/2010-08/threads.html#6

As far as I could see, this is the only thread in all of bug-global's (public) 
history in which you contributed, hence your only public input to upstream 
about how >> 5.7.1 versions have problems in your opinion.

Is there another public bug tracker somewhere that I missed?

Did you have other conversations with upstream? If so, where can we find them?

> > If all the problems come from "the htags from v6", what is blocking you
> > from at least providing the latest 5.x versions?
> 
> We are providing the latest 5.x version that didn't break the interfaces
> we need.  I'm talking about v6 here, because v6 is what we are talking
> about moving to next.

Fair enough. Sorry for assuming the breakage was in from 6.0 on.

I'm purposedly not answering to the rest of your email, as I think the TC now 
has enough information to issue a decision.

-- 
Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-12-09 11:58 +0900, Shigio YAMAGUCHI wrote:
> Hello all,
> 2016 19:05:55 +, Wookey wrote:
> > The .cgi scripts are generated from .in files which are correctly
> > parameterised with @PERLPATH@ and @GLOBALPATH@ etc. Upstream
> > unhelpfully ships pre-generated versions with the above arbitrary
> > local paths, but we kicked the build to force regeneration of these so
> > that all the scripts come out with correct debian paths. That was in
> > 6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
> > too). Please file a bug if we missed any.
> 
> It's my mistake. I will fix it soon.
> 
> It is helpful if these bug reports are sent to bug-glo...@gnu.org.
> Thank you in advance.

Yes. Now that we have the package in some sort of reasonable shape in
debian we will forward the relevant bits upstream, in order to
minimise debian patches. There are quite a few things we want to
discuss :-)

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Philip Hands]

Hi Shigio,

Thanks for getting involved.

Shigio YAMAGUCHI  writes:

> Hello all,
>
> 2016 23:32:44 +1030, Ron wrote:
>> open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");
>>
>> Which for those who don't speak it, is perl for "anyone can execute
>> arbitrary shell commands by typing them into a web browser", since
>> $pattern is an unsanitised, untrusted, input from the query string.
>
> This code is for Windows; it is not used in UNIX.
> Ron's quotation seems to be part of the following code:
>
> --
> [global.cgi.tmpl.in] (global-6.5.2)
> --
> if ($^O eq 'MSWin32') {
> open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern
> |");
> } else {
> open(PIPE, "-|") || exec '@globalpath@', '--result=ctags-xid',
> $flags, $pattern;

Is it not the case that this last line forks and execs global, passing
$pattern as a parameter to global's -e option, and that $pattern is
untrusted input?

Looking at global.c it seems that before it is passed on to popen, it is
run through quote_shell() which quotes any single-quotes in the string.

That seems to deal with Ron's assertion that it's exploitable, although
I have a slight feeling of impending doom about relying upon just this.

Would it not be wise to make the network-facing perl code runnable with
strict and taint turned on, if only to stop people reacting with horror
at first glance?

I presume patches would be welcome?

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Vincent Bernat]

 ❦  9 décembre 2016 02:35 +1030, Ron  :

>> > How much am I supposed to hound you when you give a non-answer?
>> 
>> Maybe assume good faith and tell me that the answer doesn't fit you?
>
> You said you weren't interested in debugging it further, and so did
> Punit - how is it not assuming good faith to take what you said at
> face value?  You're the only two users of ggtags that I know of.

I said I weren't interested in debugging it further two months ago (and
I am still not interested). I didn't say such a thing during my initial
participation more than two years ago.

There is no amount of distraction that you can do _now_ that would hide
the inactivity you had during all those years. Even people reporting
easily fixable packaging-related problems with patches were ignored:

 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587489
-- 
The lunatic, the lover, and the poet,
Are of imagination all compact...
-- Wm. Shakespeare, "A Midsummer Night's Dream"


signature.asc
Description: PGP signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Shigio YAMAGUCHI]

Hello all,

2016 23:32:44 +1030, Ron wrote:
> open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");
>
> Which for those who don't speak it, is perl for "anyone can execute
> arbitrary shell commands by typing them into a web browser", since
> $pattern is an unsanitised, untrusted, input from the query string.

This code is for Windows; it is not used in UNIX.
Ron's quotation seems to be part of the following code:

--
[global.cgi.tmpl.in] (global-6.5.2)
--
if ($^O eq 'MSWin32') {
open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern
|");
} else {
open(PIPE, "-|") || exec '@globalpath@', '--result=ctags-xid',
$flags, $pattern;
if ($?) {
error_and_exit("Cannot execute global.");
}
}
--

Though I don't recognize it is a security hole on Windows, I don't know
whether
it is true in the future. So, it is commented out in the latest release now.

--
[global.cgi.in] (global-6.5.5)
--
if ($^O eq 'MSWin32') {
#
# This code was commented out, because it may have a security hole
in the
# future.  To use this code, please uncomment in your own
responsibility.
#
#open(PIPE, "$global_command" . " --result=ctags-xid $flags
\"$pattern\" |");
error_and_exit("Feature not implemented.");
} else {
open(PIPE, "-|") || exec "$global_command", '--result=ctags-xid',
$flags, $pattern;
if ($?) {
error_and_exit("Cannot execute global.");
}
}
--

Please see the following thread, for the details.

[A CGI security hole on Windows?]
http://lists.gnu.org/archive/html/bug-global/2016-03/msg2.html


2016 19:05:55 +, Wookey wrote:
> The .cgi scripts are generated from .in files which are correctly
> parameterised with @PERLPATH@ and @GLOBALPATH@ etc. Upstream
> unhelpfully ships pre-generated versions with the above arbitrary
> local paths, but we kicked the build to force regeneration of these so
> that all the scripts come out with correct debian paths. That was in
> 6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
> too). Please file a bug if we missed any.

It's my mistake. I will fix it soon.

It is helpful if these bug reports are sent to bug-glo...@gnu.org.
Thank you in advance.

Regards,
Shigio

-- 
Shigio YAMAGUCHI 
PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3
A long mail is hell.
   -- An anonymous philosopher in ancient Greece

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Thu, Dec 08, 2016 at 06:24:32PM +0100, Didier 'OdyX' Raboud wrote:
> Le jeudi, 8 décembre 2016, 18.14:12 h CET Tollef Fog Heen a écrit :
> > Using open like in the code snippet above is pretty much inexcusable in
> > this day and age.
> 
> Fair enough, thanks for the explanation.
> 
> Ron: could you point us to your report of this problem to the upstream 
> bugtracker or list? What was the answer you got?

I didn't audit that code exhaustively when Punit proposed uploading it,
there were already enough things obviously wrong with what he was
suggesting to go through all of it with a fine toothed comb to find more
before he'd shown any interest in addressing the first lot.

But it stood out like a sore thumb when I was fact checking the answer
to Phil's question about the CGI being a hopeless case, to be sure that
my answer was as accurate as possible over the range of changes that
have happened to it.

It certainly seems like something that anyone professing that they
should be trusted to maintain this probably should have been looking
at when the red flags went up about upstream's idea of what is
adequately secure ...

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Thu, Dec 08, 2016 at 05:03:40PM +0100, Didier 'OdyX' Raboud wrote:
> Just commenting to some specific points as, despite an explicit request, you 
> have failed (and spectacularly so) to provide brief answers. That's not 
> helping a quick and clear path towards resolution.

Now I feel like goldilocks, first I'm bad because I didn't respond enough,
now I'm bad because I respond too much.

But apparently, I should have actually said just a little more in this
one too, to explain to you how perl works :)  So I'll do that now!

 
> Le jeudi, 8 décembre 2016, 23.32:44 h CET Ron a écrit :
> > On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> > > Perhaps you'd be kind enough to either confirm or correct my perceptions
> > > of the current situation:
> > >   Version 6 includes a CGI script that one is expected to install in a
> > >   manner so hopelessly insecure that we'd not accept it in Debian.
> > 
> > For the version (…) that I nacked, which is where this appeal to the ctte
> > started from, that's absolutely true.  Not only did it have the 'chmod 777'
> > interface to enable it, it had little gems in it like this too:
> > 
> >  open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");
> > 
> > Which for those who don't speak it, is perl for "anyone can execute
> > arbitrary shell commands by typing them into a web browser", since
> > $pattern is an unsanitised, untrusted, input from the query string.
> 
> If you haven't yet, I urge you to use our standard interface to report such 
> bugs; please make sure issues like this one are public on our bugtracker, 
> with 
> correct found/notfound version markers.

Do you really want entries in the tracker for buggy code that was never
in Debian, because I nacked Punit uploading things he didn't understand
with a vague promise to maybe look at them later?


> This also applies to group who has uploaded the experimental version: please 
> version-close bugs that this version fixes.
> 
> For that specific Perl problem, I'd love to be enlightened in how the version 
> in 6.5.5 is significantly worse than the code in 5.7.1-3's global.cgi.tmpl:
> 
> http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?hl=152#L152

Ok, you don't speak perl ...

http://perldoc.perl.org/functions/open.html
http://perldoc.perl.org/perlsec.html

But in this case, it's also not that different to shell best practice.

 "You would want to use the list form of the pipe so you can pass
  literal arguments to the command without risk of the shell
  interpreting any shell metacharacters in them."


> > It also made changes that guarantee everyone will need to fork it
> > for distro use, because it now hardcodes a fun mix of paths, like
> > /opt/local/bin for perl and /usr/local for global et al.
> 
> That's a _very_ typical distro maintainer's job, really.

But still a regression in this code where that previously wasn't needed.


> > There's little things like it still having .la files in it, and
> > static libs for things that are supposedly 'plugins'.  Which aren't
> > a big deal on their own to fix, but again suggests that if 'obvious'
> > things like that were missed, there's a good chance there will be
> > more issues than what I've already seen in a cursory review.
> 
> I don't really appreciate how you're sniping at the maintainers and uploaders 
> of the version currently in experimental, while doing the job to keep up with 
> recent versions of global was _your_ duty all along. What about actually 
> _helping_ making this global version as good as you think it should be?

How is pointing out real issues with it "sniping at them"?

We have people here saying what a wonderful and perfect job they are
doing, while slagging off at me.  Including yourself.

I didn't see you telling them that sort of thing is not appreciated.

Last I heard it was considered bad form to change the package format
in an NMU too if you want to talk about "duty".


> > What I'd _like_ to see a good consensus on though, is a little more
> > than that - because I don't think "should we ship v6 in Stretch" is
> > the right question, or rather a _sufficient_ question.  Because if
> > the answer to that is "yes" - then we still have the question of
> > "what should we do with htags in v6", and that's actually the one
> > where things go all pear-shaped if we get it wrong.
> > 
> > And even if the answer to it is "no", then that question _still_
> > remains as "what should we do with htags in v6 for stretch+1" ...
> 
> The question was supposed to be asked, and answered in september 2011, when 
> global 6.0 was released. This question should have been answered for squeeze, 
> eventually wheezy, really.

The question *was* asked (by me), repeatedly.  Nothing material changed
to alter the problem and hence the answer.

Now we're talking about what to do among a wider group of people, given
that it still looks like nothing material will change.  The system works?


> > And ignoring 

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-12-08 23:32 +1030, Ron wrote:
> On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> > Ron  writes:
> > 

Thanks for comprehensive reply, Ron. 

I wonder if it would make more sense to have this discussion in
#816924 which is the actual bug for 'should we upgrade global to v6,
and if not, why not?' You have to be in the know to find this on the
ctte mailing list.
 
> >   Version 6 includes a CGI script that one is expected to install in a
> >   manner so hopelessly insecure that we'd not accept it in Debian.
 
> That one had removed the ability to run it from a secure system CGI
> location at all, and changed the way that it's generated yet again.
> It also made changes that guarantee everyone will need to fork it
> for distro use, because it now hardcodes a fun mix of paths, like
> /opt/local/bin for perl and /usr/local for global et al.

The .cgi scripts are generated from .in files which are correctly
parameterised with @PERLPATH@ and @GLOBALPATH@ etc. Upstream
unhelpfully ships pre-generated versions with the above arbitrary
local paths, but we kicked the build to force regeneration of these so
that all the scripts come out with correct debian paths. That was in
6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
too). Please file a bug if we missed any.

So this is not an outstanding issue, and no fork needed,but it would
be nice to improve upstrema's build and reduce debian patching here.

> It does comment out that line from above with a note:
> "To use this code, please uncomment in your own responsibility."

OK, so as shipped, that's not actually an active problem.

> But it also outputs a .htaccess enabling execution in the directory
> where the output is generated, whether you want to use it from there
> or not (and adds a second CGI, and a bunch of jquery doing AJAX too).

The world is absolutely full of jquery doing AJAX. That's not bad in
itself (much as some of us prefered the 'old web'. We should make it
use system jquery. 

> It has a bunch of immediately obvious problems:
> 
>  - it still passes the unsanitised $pattern to global, which then
>passes it to popen (which again lets it be interpreted by the
>shell).

So $pattern used to be sanitised, because that's what CVE-2000-0952 is
about.  Ah and in fact it still is on line 148. Hmm, but that's after
using it in a pipe, which probably isn't much use... That is pretty
shoddy.

>  - it won't run with perl's taint mode enabled, because that
>simply kills it warning of unsafe operations on untrusted data.
> 
>  - perl's warnings aren't enabled, and it complains of other
>rookie code problems if you do.
> 
>  - Enabling 'use strict' makes it scream with pain and completely
>fail to compile and run.
> 
> 
> It would need more thorough auditing to confirm that there is(n't) an
> exploitable path through that, but given that it ignores even the most
> basic advice which perl has strongly recommended since perl4 was new
> and shiny, then if there isn't, it would be because of luck more than
> obvious care.

I don't claim any web security expertise, so will ask a
potentially-dumb question.  How much of a real world problem is the
shoddiness of this script if it is only used locally, using
htags-server (which is the only functionality now provided by the
package)? It exploses the script only to localhost unless you
explicitly configure it to do more, but not 'the net'. Nothing is done
as root, but it is run as 'you' so potentially gives access to 'you's
data, rather than all of www-data's data. 

The only report of an actual vulnerability I can find online ('global'
is a very unhelpful name to search for vulnerbilities!) is:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2000-0952 (from
2000 for global prior to v4.0.1

Which claimed to be fixing this very issue, but apparently it didn't stay fixed?

And this same code exists in global v5.7.1 (indeed both usages are
there, including the now-commented-out one), so how exactly was that
safer?

> It might not be utterly unfixable, but you'd have to convince upstream
> that security is important, or completely fork it for debian, which
> brings us back to exactly where we are - unless we just remove it all.
> But that would need Time, whoever does it.

I have not grokked why the shoddy code in 5.7.1 is safe but the same
shoddy code in v6 cannot be let out. Did htmake+htconfig stop people
entering $pattern in the form?

> It is what we now have enabled in the package that Wookey uploaded to
> experimental.

Indeed. Bugs (and even better patches) are welcome.

> >   Also, for people that want personal access to htags there is a
> >   htags-server command that brings up a dedicated server to run the CGI
> >   as the invoking user, by default bound to a localhost port.
> 
> htags-server is a shell script that generates python or ruby code
> to use the HTTP server functionality they can provide.

I'd describe it as a script that runs the 

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Didier 'OdyX' Raboud]

Le jeudi, 8 décembre 2016, 18.14:12 h CET Tollef Fog Heen a écrit :
> Using open like in the code snippet above is pretty much inexcusable in
> this day and age.

Fair enough, thanks for the explanation.

Ron: could you point us to your report of this problem to the upstream 
bugtracker or list? What was the answer you got?

-- 
Cheers,
OdyX

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Tollef Fog Heen]

]] Didier 'OdyX' Raboud 

> Le jeudi, 8 décembre 2016, 23.32:44 h CET Ron a écrit :
> > On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> > > Perhaps you'd be kind enough to either confirm or correct my perceptions
> > > of the current situation:
> > >   Version 6 includes a CGI script that one is expected to install in a
> > >   manner so hopelessly insecure that we'd not accept it in Debian.
> > 
> > For the version (…) that I nacked, which is where this appeal to the ctte
> > started from, that's absolutely true.  Not only did it have the 'chmod 777'
> > interface to enable it, it had little gems in it like this too:
> > 
> >  open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");
> > 
> > Which for those who don't speak it, is perl for "anyone can execute
> > arbitrary shell commands by typing them into a web browser", since
> > $pattern is an unsanitised, untrusted, input from the query string.
> 
> If you haven't yet, I urge you to use our standard interface to report such 
> bugs; please make sure issues like this one are public on our bugtracker, 
> with 
> correct found/notfound version markers.
> 
> This also applies to group who has uploaded the experimental version: please 
> version-close bugs that this version fixes.
> 
> For that specific Perl problem, I'd love to be enlightened in how the version 
> in 6.5.5 is significantly worse than the code in 5.7.1-3's global.cgi.tmpl:
> 
> http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?
> hl=152#L152

It's completely different.  It's basically system(3) on a concatenated
string with partial user-defined content vs execve(2) on a list of
arguments (some of which are user-provided).

perldoc -f exec and perldoc -f open might be useful.

Using open like in the code snippet above is pretty much inexcusable in
this day and age.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Thu, Dec 08, 2016 at 03:41:14PM +0100, Vincent Bernat wrote:
>  ❦  9 décembre 2016 00:32 +1030, Ron  :
> 
> > How much am I supposed to hound you when you give a non-answer?
> 
> Maybe assume good faith and tell me that the answer doesn't fit you?

You said you weren't interested in debugging it further, and so did
Punit - how is it not assuming good faith to take what you said at
face value?  You're the only two users of ggtags that I know of.

But I can repeat it again one more time.  Pretty please with sugar
on top can we have a real bug report, with real information to let us
assess whether your problem can be fixed with a trivial patch or not?

I'm assuming in good faith that you do know what a proper bug report
should look like, but I can give you some pointers in good faith
if you don't.  Looking at the bugs that Wookey filed should be a
good guide.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Didier 'OdyX' Raboud]

Just commenting to some specific points as, despite an explicit request, you 
have failed (and spectacularly so) to provide brief answers. That's not 
helping a quick and clear path towards resolution.

Le jeudi, 8 décembre 2016, 23.32:44 h CET Ron a écrit :
> On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> > Perhaps you'd be kind enough to either confirm or correct my perceptions
> > of the current situation:
> >   Version 6 includes a CGI script that one is expected to install in a
> >   manner so hopelessly insecure that we'd not accept it in Debian.
> 
> For the version (…) that I nacked, which is where this appeal to the ctte
> started from, that's absolutely true.  Not only did it have the 'chmod 777'
> interface to enable it, it had little gems in it like this too:
> 
>  open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");
> 
> Which for those who don't speak it, is perl for "anyone can execute
> arbitrary shell commands by typing them into a web browser", since
> $pattern is an unsanitised, untrusted, input from the query string.

If you haven't yet, I urge you to use our standard interface to report such 
bugs; please make sure issues like this one are public on our bugtracker, with 
correct found/notfound version markers.

This also applies to group who has uploaded the experimental version: please 
version-close bugs that this version fixes.

For that specific Perl problem, I'd love to be enlightened in how the version 
in 6.5.5 is significantly worse than the code in 5.7.1-3's global.cgi.tmpl:

http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?
hl=152#L152

> It also made changes that guarantee everyone will need to fork it
> for distro use, because it now hardcodes a fun mix of paths, like
> /opt/local/bin for perl and /usr/local for global et al.

That's a _very_ typical distro maintainer's job, really.

> It is what we now have enabled in the package that Wookey uploaded to
> experimental.

At least now we have a version in Debian we can compare with. Noone has 
claimed it would be perfect, but uploading this new version (to whichever 
suite, really) was your duty as maintainer and you failed to do so in 6+ 
years.

> > Are there any other regressions that you are aware of in v6?
> 
> In terms of the upstream code, the issues with htags are the main
> 'showstopper' I'm currently aware of.  But I haven't tested the
> rest of it anywhere near exhaustively yet either.
> 
> In terms of the package currently in experimental, there's a bunch
> of issues with it that would need to be fixed if we were going to
> want it in Stretch. 

Please file these as bugreports, with appropriate severities. This is the only 
way we will all be able to have a clear picture of what makes each version the 
most suitable to release in Stretch.

> There's little things like it still having .la files in it, and
> static libs for things that are supposedly 'plugins'.  Which aren't
> a big deal on their own to fix, but again suggests that if 'obvious'
> things like that were missed, there's a good chance there will be
> more issues than what I've already seen in a cursory review.

I don't really appreciate how you're sniping at the maintainers and uploaders 
of the version currently in experimental, while doing the job to keep up with 
recent versions of global was _your_ duty all along. What about actually 
_helping_ making this global version as good as you think it should be?

> What I'd _like_ to see a good consensus on though, is a little more
> than that - because I don't think "should we ship v6 in Stretch" is
> the right question, or rather a _sufficient_ question.  Because if
> the answer to that is "yes" - then we still have the question of
> "what should we do with htags in v6", and that's actually the one
> where things go all pear-shaped if we get it wrong.
> 
> And even if the answer to it is "no", then that question _still_
> remains as "what should we do with htags in v6 for stretch+1" ...

The question was supposed to be asked, and answered in september 2011, when 
global 6.0 was released. This question should have been answered for squeeze, 
eventually wheezy, really.

> Because people saying "it's irrelevant, uploading 'something newer'
> is overwhelmingly more important" completely misses the point that
> uploading something newer unavoidably involves someone answering
> that question.

Uploading newer versions of upstream software is constantly about compromises, 
functionality losses, and new functionalities. That's just how software 
development works, and Debian is constantly following up on the new versions 
of all sorts of software. With your approach, we'd have Gnome 2, KDE 3, 
PostgreSQL 8.4 and GCC 4.4 in stretch, just with some compatibility patches, 
because we'd have been too conservative. That's not the sort of Debian 
releases I want to see.

> And ignoring the issues around it totally leads to fun paradoxes
> like OdyX saying that one 

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Thu, Dec 08, 2016 at 02:39:44PM +0100, Vincent Bernat wrote:
>  ❦  8 décembre 2016 23:32 +1030, Ron  :
> 
> > One is whatever it is that the third-party ggtags wrapper needs, which
> > aiui is what Vincent and Punit are most annoyed about.  But I don't
> > use emacs, and ggtags isn't even in Debian - and they haven't even
> > told me what error they see, let alone what operation(s) trigger it.
> >
> > Vincent just gave me the output of global's short --help, and said
> > "look, there's new options" - but we don't even know if it's actually
> > a 'missing' command line option that it fails on (or which one that
> > might be), or something else entirely.  My hunch is that one would
> > probably be pretty trivial to fix - either in ggtags or global, or
> > both - if someone who uses it engages with what I asked originally,
> > to file a separate bug from the 'new upstream' one, detailing the
> > actual problem, and is willing to test any proposed fixes even if
> > they aren't up to actually submitting a patch for that themselves.
> > I don't mind writing a patch if we know what actually needs patching.
> 
> And for this particular case, you didn't asked for anything more. You
> just said nothing.

This was the very simple question I asked, and your non-response to it:

  >> I am using gg-tags in Emacs and the current version of global in
  >> Debian just doesn't work with this mode.
  >
  > What changed incompatibly to make it not work?  And what would need
  > patching to fix that?
  >
  > I'd really much rather see problems get fixed than layered under even
  > more problems.  If someone familiar with Emacs has some details of
  > what doesn't work, and what needs to be done so that it will, that
  > sounds like a separate bug to be addressed to me.

  Some arguments seem to not exist in previous versions. I did not
  investigate more


How much am I supposed to hound you when you give a non-answer?
I've asked you again several times here, and each time you put in a
bunch of work to trot out some new accusations, but do nothing to
actually answer the question.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Vincent Bernat]

 ❦  8 décembre 2016 23:32 +1030, Ron  :

> One is whatever it is that the third-party ggtags wrapper needs, which
> aiui is what Vincent and Punit are most annoyed about.  But I don't
> use emacs, and ggtags isn't even in Debian - and they haven't even
> told me what error they see, let alone what operation(s) trigger it.
>
> Vincent just gave me the output of global's short --help, and said
> "look, there's new options" - but we don't even know if it's actually
> a 'missing' command line option that it fails on (or which one that
> might be), or something else entirely.  My hunch is that one would
> probably be pretty trivial to fix - either in ggtags or global, or
> both - if someone who uses it engages with what I asked originally,
> to file a separate bug from the 'new upstream' one, detailing the
> actual problem, and is willing to test any proposed fixes even if
> they aren't up to actually submitting a patch for that themselves.
> I don't mind writing a patch if we know what actually needs patching.

Or, more likely, just like for #587489, #613011, #182188, #212040,
#218111, #669617, #756367 and #130091 (more than half of the open bugs
for global), you will just ignore the bug report. I have not included
#715980 and #716006 which are automated. I don't understand how you can
try to paint yourself as someone who cares about your users while there
is this public track record of your unwilligness to discuss with
anybody.

And for this particular case, you didn't asked for anything more. You
just said nothing.
-- 
The surest protection against temptation is cowardice.
-- Mark Twain


signature.asc
Description: PGP signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> Ron  writes:
> 
> > I'm not insisting that's what we should do.  But it's certainly an
> > option, and it dodges the bullet of having to say "Sucks to be you"
> > without any notice at all.  And it doesn't take anything away from
> > the people who want "new upstream or bust" for Stretch, because it
> > can still be available to them in backports.
> 
> Perhaps you'd be kind enough to either confirm or correct my perceptions
> of the current situation:

I'll try to be concise, but I might fail at 'brief', since I think it's
important to fairly summarise all sides of this.  Leaving people to
somehow reconcile narrowly polarised, conflicting sound bites isn't very
helpful.  And since I don't have the option to abstain from an opinion,
and aren't starting from an immutable one, I want to be as sure as I can
that _I've_ got the whole picture clear in my head before settling on
what I think is really best too.

If I show how I got there, and you disagree with my rationale, then I
have something that _I_ can rethink to maybe agree with you.  If all
I do is state a conclusion, and handwave the details of how I got there,
then at best we can agree to disagree, and at worst we get more random
trolls throwing mindless insults at me again.


>   Version 6 includes a CGI script that one is expected to install in a
>   manner so hopelessly insecure that we'd not accept it in Debian.

For the version that Punit and Vincent wanted me to let them upload
and that people complained that I nacked, which is where this appeal
to the ctte started from, that's absolutely true.  Not only did it
have the 'chmod 777' interface to enable it, it had little gems in it
like this too:

 open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");

Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
The version Punit published and suggested people should use had all
of that enabled in it - and he did that after I'd warned him there
was trouble that shouldn't just be ignored.


That was the shared state and history when this ctte bug was opened.
In the interests of seeing what new options we had _today_, I then did
an initial review of the most recent upstream state, which nobody else
had yet looked at, to see if anything material had changed which might
improve the situation.  My second post here included details of that,
but the short answer is "it got more complicated".

That one had removed the ability to run it from a secure system CGI
location at all, and changed the way that it's generated yet again.
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.

It does comment out that line from above with a note:
"To use this code, please uncomment in your own responsibility."
But it also outputs a .htaccess enabling execution in the directory
where the output is generated, whether you want to use it from there
or not (and adds a second CGI, and a bunch of jquery doing AJAX too).

It has a bunch of immediately obvious problems:

 - it still passes the unsanitised $pattern to global, which then
   passes it to popen (which again lets it be interpreted by the
   shell).

 - it won't run with perl's taint mode enabled, because that
   simply kills it warning of unsafe operations on untrusted data.

 - perl's warnings aren't enabled, and it complains of other
   rookie code problems if you do.

 - Enabling 'use strict' makes it scream with pain and completely
   fail to compile and run.


It would need more thorough auditing to confirm that there is(n't) an
exploitable path through that, but given that it ignores even the most
basic advice which perl has strongly recommended since perl4 was new
and shiny, then if there isn't, it would be because of luck more than
obvious care.

I'd not think it was a Good Plan to push this out as a "rush to beat the
freeze" upload of this source without a much more careful look at it,
and some effort spent actually fixing the already obvious deficiencies,
but the size of the truck you might drive through its holes is a bit
different to where we started from.

It might not be utterly unfixable, but you'd have to convince upstream
that security is important, or completely fork it for debian, which
brings us back to exactly where we are - unless we just remove it all.
But that would need Time, whoever does it.

It is what we now have enabled in the package that Wookey uploaded to
experimental.


>   However, it is possible to generate static content ...

Yes.

>   ... that achieves the
>   same aims, but at the cost of approximately doubling the disk usage,
>   and presumably also requiring more time to generate.

No.  This isn't just a 

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Philip Hands]

Ron  writes:

...
> I'm not insisting that's what we should do.  But it's certainly an
> option, and it dodges the bullet of having to say "Sucks to be you"
> without any notice at all.  And it doesn't take anything away from
> the people who want "new upstream or bust" for Stretch, because it
> can still be available to them in backports.

Perhaps you'd be kind enough to either confirm or correct my perceptions
of the current situation:

  Version 6 includes a CGI script that one is expected to install in a
  manner so hopelessly insecure that we'd not accept it in Debian.

  However, it is possible to generate static content that achieves the
  same aims, but at the cost of approximately doubling the disk usage,
  and presumably also requiring more time to generate.

  Also, for people that want personal access to htags there is a
  htags-server command that brings up a dedicated server to run the CGI
  as the invoking user, by default bound to a localhost port.

  Version 6 fixes some bugs that are still present in your version 5
  package and/or provides features that are missing, but bug reports of
  sufficient quality to allow you to fix/backport to v5 are lacking.

Is that about right?

Are there any other regressions that you are aware of in v6?

Your suggestion, as I understand it, is that v6 should hit unstable
after stretch's release, and that people who are currently complaining
about bugs/missing-features in v5 (or are overly focused on numbers) can
then grab v6 out of stretch-backports.

Could you consider the relative merits of instead putting v6 into
stretch now, and dealing with the people that are currently clinging to
v5 by pointing out that:

  0) there are now other alternatives to htags that might suit them better.

  1) htags-server lets them run the CGI for local access.

  2) htags can generate static content, and thus safely provide general
 access while avoiding the need for a CGI

  3) If there is anyone that cannot do either for some reason, they can
 install global v5 from jessie, pin it to avoid upgrades, and report
 the reasons why they had to do this to the BTS.

Have I missed some vital aspect of this?

Is there a compelling reason to favour the theoretical users that might
want to stick with v5 over the actual users that we've been hearing ask
for v6?

If the TC were to achieve consensus that v6 should be in stretch, will
it be sufficient for us to inform you of that in order to make it so?

I gather from what you just wrote that it would be sufficient.

If however you are likely to insist on resolutions to override the
maintainer, or transfer the maintainership, I think you ought to be
up-front about that in order to avoid any accusations of intentional
time-wasting later on.

If you can keep your answers brief, you'll earn my gratitude.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Nikolaus Rath]

On Dec 05 2016, Ron  wrote:
> Hi Sam,
>
> On Wed, Nov 30, 2016 at 11:39:08AM -0500, Sam Hartman wrote:
>> > "Ron" == Ron   writes:
>> 
>> Ron> Hi OdyX,
>> 
>> Ron> On Wed, Nov 16, 2016 at 03:23:47PM +0100, Didier 'OdyX' Raboud 
>> wrote:
>> >> Hi there,
>> >> 
>> >> I've been mostly VAC, and only now found enough time to properly
>> >> read through this bug log. In the interest of transparency and to
>> >> help the TC reach a consensus (also through understanding what
>> >> the other members' understanding, ideas and positions are), here
>> >> comes my current understanding of the problem at hand.
>> >> 
>> >> As preamble, I will upfront state that I have _not_ looked in the
>> >> precise details of the so-called 'htags' functionality, as, the
>> >> rest will show, my current viewpoint on the problem is that it
>> >> doesn't matter.
>> 
>> Ron> If we run with your proposal, what are you actually suggesting
>> Ron> we tell the people who'd be upset by the loss of htags without
>> Ron> notice in Stretch?  Because I don't really see how you've
>> Ron> addressed that here.
>> 
>> Ron, thanks for working with the process.
>
> Thanks for engaging with the question :)
>
> I was a little disappointed that all OdyX had to say about it was:
>
>   I've sent a long mail with my opinion, and Ron's answer hasn't
> altered my conviction yet.
>
> Because "lalala, I'm not listening" isn't really an answer to a simple
> direct question.  And definitely not something that I can wrap with
> "Sorry, but a group of Debian Developers considered this all in careful
> detail, and this is what we have decided", when breaking the news to
> affected people.

...whose existence has only been postulated so far though.

In any case, it seems there are plenty of people who'd be willing to
relieve you of that burden and take over maintenance of global -
including dealing with any fallout from updating to an up-to-date
version.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

Hi Sam,

On Wed, Nov 30, 2016 at 11:39:08AM -0500, Sam Hartman wrote:
> > "Ron" == Ron   writes:
> 
> Ron> Hi OdyX,
> 
> Ron> On Wed, Nov 16, 2016 at 03:23:47PM +0100, Didier 'OdyX' Raboud wrote:
> >> Hi there,
> >> 
> >> I've been mostly VAC, and only now found enough time to properly
> >> read through this bug log. In the interest of transparency and to
> >> help the TC reach a consensus (also through understanding what
> >> the other members' understanding, ideas and positions are), here
> >> comes my current understanding of the problem at hand.
> >> 
> >> As preamble, I will upfront state that I have _not_ looked in the
> >> precise details of the so-called 'htags' functionality, as, the
> >> rest will show, my current viewpoint on the problem is that it
> >> doesn't matter.
> 
> Ron> If we run with your proposal, what are you actually suggesting
> Ron> we tell the people who'd be upset by the loss of htags without
> Ron> notice in Stretch?  Because I don't really see how you've
> Ron> addressed that here.
> 
> Ron, thanks for working with the process.

Thanks for engaging with the question :)

I was a little disappointed that all OdyX had to say about it was:

  I've sent a long mail with my opinion, and Ron's answer hasn't
altered my conviction yet.

Because "lalala, I'm not listening" isn't really an answer to a simple
direct question.  And definitely not something that I can wrap with
"Sorry, but a group of Debian Developers considered this all in careful
detail, and this is what we have decided", when breaking the news to
affected people.


> I think that one thing you sometimes find when you try and build
> consensus is that your conception of the problem and even the values by
> which a solution should be judged is not shared.

I think you and I shared some opinions about the merits of the IETF
model for consensus on #-devel back when people were last exasperated
with Ian's antics and looking to get some new blood and new ways of
thinking into the ctte.  So yes, I don't have a problem with my view
of things ending up being what's in the rough - the important part is
that the questions raised have been answered, and that there's rough
consensus those answers are sufficient to actually address the problem
in some manner.

Or more concisely:

 "Rough consensus is achieved when all issues are addressed, but not
  necessarily accommodated"

https://tools.ietf.org/html/rfc7282, if people aren't already familiar
with that process.


> As I understand it, you believe we should be evaluating the technical
> options and that preserving things that work is of high value.
> 
> I think a lot of us believe that get newest code is a presumptive value
> especially over the multiple releases time frame.

It's probably a bit more subtle than that, though my context is certainly
going to be coloured by knowing the history of this particular package.

This is a package which required significant patching to bring it up to
what was considered acceptable practice for distro software from its very
first release with debian back in 1999.  Some of the issues were just
general problems and bugs, and upstream took patches for those.  And some
of them were things he just didn't care about (or understand) at all,
but he welcomed them being maintained separately as distro patches.

So I don't see it as an unusual thing for us to patch the problems that
are effecting distro users.

And I've looked at what the newer versions have added, and it's not like
there's a lot there which could really be described as earth shattering
must-haves.

And I don't think the only answer to fixing what Vincent is complaining
about, or the issues that Wookey looked into more recently is "new
upstream or bust".  I'm pretty sure that we could probably fix those
relatively simply with patches to what we have now.

I even asked for enough information to be able to assess and/or do that.
But there was no interest in providing that from Vincent or Punit, and
no other user of ggtags has surfaced to put their hand up, so like most
things with insufficient information and interest, that went no further
so far.


So as a one sentence answer, "my belief" is probably more like "we can
have the best of both worlds for Stretch".  If the people expending
energy arguing here, spent just a tiny portion of that on looking into
their *problem* instead of fighting over their line-of-least-work
solution.


I'm not insisting that's what we should do.  But it's certainly an
option, and it dodges the bullet of having to say "Sucks to be you"
without any notice at all.  And it doesn't take anything away from
the people who want "new upstream or bust" for Stretch, because it
can still be available to them in backports.  Without going scorched
earth on the other users, who would have no other option but to
package their own local versions for the duration of Stretch.

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Sam Hartman]

> "Wookey" == Wookey   writes:

>> That is, I disagree with you that I need to address the question
>> of htags and figure out whether htags users are being impacted.
>> That might have been true for one release.  But over a longer
>> time frame, the really strong presumption is that we prefer a
>> version that is being actively developed to one that isn't.  That
>> if people won't step forward and maintain htags, it goes awy.

Wookey> Just to be clear, you are confusing 'htags' and
Wookey> 'htmake/htconfig' (I was too when this started). htags is
Wookey> still here, works fine, and is not going away, as Punit
Wookey> explained. The thing that is likely to go away is
Wookey> 'centralised htags browsing using system webserver' which is
Wookey> what Ron's htmake and htconfig provided.

No, I'm not:-)
Ron claimed or at least I read him as claiming that htags was not very
useful without system web browsing.
I've said that I'm not commenting on the technical issues, so since I
was talking to Ron, I took that claim as a postulate.
You've disagreed with that claim elsewhere in the thread.
While I do have an opinion, I was not intending to inject it.

Re: Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-11-30 11:39 -0500, Sam Hartman wrote:

> That is, I disagree with you that I need to address the question of
> htags and figure out whether htags users are being impacted.
> That might have been true for one release.
> But over a longer time frame, the really strong presumption is that we
> prefer a version that is being actively developed to one that isn't.
> That if people won't step forward and maintain htags, it goes awy.

Just to be clear, you are confusing 'htags' and 'htmake/htconfig' (I
was too when this started). htags is still here, works fine, and is
not going away, as Punit explained. The thing that is likely to go
away is 'centralised htags browsing using system webserver' which is
what Ron's htmake and htconfig provided.

> But I think what you're getting here from a lot of people is a belief
> that our community norm strongly favors active development and new
> software.  And sometimes when features are effectively not maintained in
> a manner that we can package them, they go away.

Right. We are a distro and by default we package what upstream
provides. A maintainer can choose to go further that that and
add/maintain extra functionality if they like, but it's an ongoing
workload that has to be carried. It is no doubt possible to modify
htmake/htconfig or write something equivalent to provide a centralised
index of 'globalified' source trees, but as upstream no longer
attempts to do such a thing there is a good argument that the debian
packaging shouldn't either. 

I certainly don't think that the fact that this was done once is a
good reason to stop packaging new releases.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Sam Hartman]

> "Ron" == Ron   writes:

Ron> Hi OdyX,

Ron> On Wed, Nov 16, 2016 at 03:23:47PM +0100, Didier 'OdyX' Raboud wrote:
>> Hi there,
>> 
>> I've been mostly VAC, and only now found enough time to properly
>> read through this bug log. In the interest of transparency and to
>> help the TC reach a consensus (also through understanding what
>> the other members' understanding, ideas and positions are), here
>> comes my current understanding of the problem at hand.
>> 
>> As preamble, I will upfront state that I have _not_ looked in the
>> precise details of the so-called 'htags' functionality, as, the
>> rest will show, my current viewpoint on the problem is that it
>> doesn't matter.

Ron> If we run with your proposal, what are you actually suggesting
Ron> we tell the people who'd be upset by the loss of htags without
Ron> notice in Stretch?  Because I don't really see how you've
Ron> addressed that here.

Ron, thanks for working with the process.

I think that one thing you sometimes find when you try and build
consensus is that your conception of the problem and even the values by
which a solution should be judged is not shared.

As I understand it, you believe we should be evaluating the technical
options and that preserving things that work is of high value.

I think a lot of us believe that get newest code  is a presumptive value
especially over the multiple releases time frame.

That is, I disagree with you that I need to address the question of
htags and figure out whether htags users are being impacted.
That might have been true for one release.
But over a longer time frame, the really strong presumption is that we
prefer a version that is being actively developed to one that isn't.
That if people won't step forward and maintain htags, it goes awy.

That's a presumption.  If someone gets evidence that htags is heavily
used, we can consider that.
We might even go so far given sufficient evidence to decide that forking
global and never taking a new upstream was the right answer for our
users.  That would take a lot of evidence.

I disagree with your approach that the people wanting to remove htags
need to show that it is being used.  I disagree with your view that over
the timescales involved, people wanting a new upstream need to justify
that.

Yes, removing htags creates a regression.

Yes, I'm effectively saying "If you're using htags, sucks to be you.  If
you're willing to spend effort maintaining a version of htags that is
secure, then we might be able to bring it back.

Yes, it's possible that we'll learn we broke things and need to revert.

But I think what you're getting here from a lot of people is a belief
that our community norm strongly favors active development and new
software.  And sometimes when features are effectively not maintained in
a manner that we can package them, they go away.

I don't think it's reasonable to defer this to upstream and wait until
upstream removes htags.

I have tried to understand the technical issues.  I'm not sure I'm 100%
in agreement in your analysis there, but I'm fairly close.  However, I
haven't found the technical issues tend to affect my analysis of what
Debian should do.


I'd strongly urge you to work on a global6 package.  I don't care
whether it's called global or global6.  I don't think it should include
insecure cgi scripts that are enabled by default.  I'd be fine if it
didn't include htags at all.
I'm not saying upload that package now; I'm not saying where to upload
it.
(Although I wouldn't object to an upload to experimental or unstable)
I think having that package ready and keeping options open as long as
possible would help preserve our ability to work through this process.
I hope that would go a long way to addressing people's feelings of
frustration.

Thanks for your consideration,

--Sam

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Punit Agrawal]

On Wed, Nov 16, 2016 at 2:23 PM, Didier 'OdyX' Raboud  wrote:

[...]

>
> I see the following good ways out of this problem:
>
> A) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
>version to unstable;
> B) (With or without an explicit decision by the TC), src:global is handed over
>to new maintainers and they'd upload a reasonably recent 6.x version to
>unstable;
>
> (
> In both A) and B) cases, whoever is maintainer of src:global would be in
> charge of handling the subsequent (so far hypothetical) bugs in time for
> the release. As usual, everyone is welcome to help with finding,
> forwarding, and fixing bugs.
>)

One suggestion to keep extra debian (non-upstream) functionality in
global is to move it to another package package. The package would
build on upstream htags and allows users to share their html
cross-indexing via cgi-bin.

As I understand it, the extra feature pertains to enabling access via
web-server to html index of sources generated by htags.

>
> I see this as a suboptimal outcome, because it rewards inertia and stop-
> energy:
>
> C) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
> version to experimental, with the explicit goal of making that version later
> available through stretch-backports.
>
> --
> Cheers,
> OdyX
>
> P.S. Sorry for the wall of text, including too many repetitions :/
>
> [0] I know, it's all obvious.
> [1] Yes, 6+ years, even at the Debian rhythm, is indefinite.

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron]

Hi OdyX,

On Wed, Nov 16, 2016 at 03:23:47PM +0100, Didier 'OdyX' Raboud wrote:
> Hi there,
> 
> I've been mostly VAC, and only now found enough time to properly read 
> through this bug log. In the interest of transparency and to help the TC 
> reach a consensus (also through understanding what the other members' 
> understanding, ideas and positions are), here comes my current 
> understanding of the problem at hand.
> 
> As preamble, I will upfront state that I have _not_ looked in the precise 
> details of the so-called 'htags' functionality, as, the rest will show, my 
> current viewpoint on the problem is that it doesn't matter.

If we run with your proposal, what are you actually suggesting we tell
the people who'd be upset by the loss of htags without notice in Stretch?
Because I don't really see how you've addressed that here.

AFAICS, there's just either an implicit "Sucks to be you", or an
implication that this is a simple "regression" which might be fixed
by sending patches upstream.

But since people have actually tried the latter, and it's not a "simple
regression", but rather upstream rejecting such discussion and then
actively removing features needed to implement it sanely in a distro
package context - it seems that only leaves the former ...


FWIW, I actually agree with a lot of the general rules of thumb that
you outlined here, about how things should work in the normal case.
But this isn't really a "normal" case, if it was we wouldn't be talking
about this here at all.  What to do would be obvious to everyone.

If anything, it probably has more in common with the cdrtools situation
where upstream actively opposed making it suitable for distro use, while
insisting we accept those changes ...  and we're split between some users
which want what has been crippled, and some which need what appears to be
just a few small changes which have been made upstream since.

And all of those people are supposedly software developers (as there's
not a lot of use for this package for people who aren't).


The group complaining loudly now have basically squandered the entire
release cycle by not reporting actionable bugs about what they need,
and haven't sent any patches to remedy that.  And they are proposing
to solve their problem by penalising the other group, who so far wasn't
complaining, without any notice or real opportunity to contribute to
improving that for themselves if they end up on the losing side of this.

Which is why I suggested what I proposed in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841294#155
better summarised by Tollef in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841294#245

That gives the people who care about htags time to (re)act, and as soon
as the backports suite for Stretch opens, it means we get the essential
essence of what Phil had suggested for free too.

For Stretch people would be able to pick what they want in the same way
as he suggested, without triggering the concerns I expressed about
forking this in a single release and juggling about with which one owns
'global' as an unqualified name, and having to decide when to 'unfork'
it again, and face the fallout that might bring on.


How is that worse than telling the people who want htags, who do have
a history of sending thought out patches, that "your only option is to
create your own packages locally if you want to use this in Stretch"?

Maybe I missed something in what you're suggesting, but I don't see
how you suggest we defuse that from blowing up in our faces through
some reasonable and fair and justifiable action.  Because their
complaint wouldn't really be substantively any different from what
the people who don't use htags are currently making.

"You're on the wrong side of a version number" isn't a very compelling
or satisfying or technically astute rationale, if that's all it boils
down to.  I'd like to have something a bit more substantial and fair
than that to offer the people who'd get burned without notice by this.


  Cheers,
  Ron

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Wookey]

On 2016-11-16 15:23 +0100, Didier 'OdyX' Raboud wrote:

[offlist]

> * I'm not happy with delaying the landing of a 6.x version of global for
>   another release, and despite the somehwat short time before the stretch full
>   freeze (2017-Feb-05), we're talking about a single binary package without
>   reverse dependencies. I'm really afraid that a side-effect of the TC
>   discussion will be yet-another release without an up-to-date src:global.

Thank you for some sanity on this matter. 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Didier 'OdyX' Raboud]

Hi there,

I've been mostly VAC, and only now found enough time to properly read 
through this bug log. In the interest of transparency and to help the TC 
reach a consensus (also through understanding what the other members' 
understanding, ideas and positions are), here comes my current 
understanding of the problem at hand.

As preamble, I will upfront state that I have _not_ looked in the precise 
details of the so-called 'htags' functionality, as, the rest will show, my 
current viewpoint on the problem is that it doesn't matter.

* src:global hasn't had a Debian upload since October 2010, for wheezy , 
  and no new upstream-release Debian upload since September 2007, for 
  lenny (although there was on upload yesterday)
* there was no src:global upload to experimental
* its maintainer, Ron Lee, failed to provide timely and comprehensive answers
  to various requests through bugreports over the years. On #574947
  specifically, a bug filed in March 2010, Ron's first answer in April 2013
  boiled down to "we're in freeze now, no".
* Ron claims to have had various private conversations about src:global with 
  various parties about the problems that a new upstream upgrade would pose.
* How the 'htags' functionality is implemented in newer upstream releases
  seems to be at the heart of what Ron sees as making these "unsuitable for
  release with the distro".
* There is arguably frustration on all sides of the discussion about upgrading 
  src:global to newer upstream releases, both sides arguing that they are
  doing the right thing for Debian.

Now. I'd like to write down some considerations about the Debian namespace.

Debian, as a Free Software distribution, ships various software made by 
various upstream authors, thereby filling the Debian source packages, binary 
packages, and binaries namespaces. Upstream's global_*.tar.gz is shipped as 
src:global source package, producing the 'global' binary package, shipping the 
/usr/bin/global binary [0].

The TC has had various discussions about uses and abuses of these various 
namespaces (especially the 'binary packages' and 'binaries' namespaces) in the 
past, usually in cases of conflict. Various developers also often safeguard 
the namespaces by launching discussions upon ITPs proposing too generic names. 
How we handle these namespaces of ours is certainly an important topic for the 
project. (But don't mistake me, the 'global' name isn't the problem here).

All these namespaces are also interfaces of what we as a project create, with 
the community of our users. The most important of these three is arguably the 
binary packages namespace, which is the usual interface with the software we 
ship. It has been customary to have a one-to-one mapping between upstream 
projects, and binary package names, and in all cases where we decided 
otherwise (iceweasel, cupsys, nodejs, …), it has imposed a lot of 
documentation and convincing for our users to get used to these.

All in all, I think there is a reasonable expectation from our users to have 
our namespaces match the rest of the ecosystem; there is an expectation that a 
given 'foobar' upstream, iff packaged for Debian, gets to be shipped as the 
'foobar' binary package. And I think that extends to versions as well; it is 
expected that our releases ship with 'reasonably recent' versions, as well.

(
 I know, Debian stable releases aren't known for the freshness of their
 packages. There's still an expectation from our users
)

Outside of special times with regards to releases (that is, outside frozen 
times), it is quite customary for packages to 'catch back' on new upstream 
releases. This has multiple effects: it prepares the new _functionalities_, 
and bug fixes for the next stable release; it also often comes with 
regressions and new bugs; annoyances and headaches for users and maintainers 
alike. Uploading new upstream releases and letting them be consumed by a 
portion of our userbase (unstable & testing users), is a way to share the 
burden, and _discover_ what those regressions and bugs are; identify, then try 
to fix or document these.

As you have understood by now, I think it is a reasonable expectation from 
package users to get new upstream releases between stable releases. The 
upstream software evolves independently of Debian, and if we're not forking 
the upstream software, we owe updates to our users as part of our "package
maintainers" duty. It's also our role to ship the newer works of our upstreams 
to our users. (To clarify: I'm not saying there can't be exceptions.)

Now, coming back to src:global, there was a first request in March 2010 
(#574947, 5.8.1) to get a new upstream release, before the squeeze freeze, 
only answered late during the wheezy freeze (at that time, 6.2.8 was 
released). The other request (#816924, 6.5.2) landed in March 2016, 9 months 
before the stretch release. For me, it looks like the squeeze, jessie _and_ 
stretch release cycles have