Re: Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Wookey]

On 2016-12-01 22:56 +0100, Philip Hands wrote:
> Wookey  writes:
> > On 2016-11-30 16:56 +, Ian Jackson wrote:
> >
> >> > And this last bit (integration with system web server) is the
> >> > functionality that had security concerns raised by Ron [etc.]
> >> 
> >> So, to be clear, it is this functionality which is dropped in the
> >> package that you and Wookey uploaded to experimental/delayed ?
> >
> > Said package is available as of today:
> > https://buildd.debian.org/status/package.php?p=global=experimental
> >  
> > And that functionality was left in as it was a self-contained patch,
> > pending determining whether in fact it remained useful/compatible or
> > not. So you still get htmake and htconfig commands.
> >
> > However I have now done some checking, and no it doesn't work any more
> > because it uses the btreeop command, which has been removed in current
> > upstream, and the --action option to htags which is also gone.
> >>  - Run the htags server on a high-numbered port somehow.  (Is there
> >>some kind of simple scriptery provided, to do this?)
> >
> > There is an example in the NEWS file, which should go in the man page. It's 
> > trivial:
> >
> > cd HTML; python -m CGIHTTPServer
> > or
> > cd HTML; python3 -m http.server --cgi
> >
> > Then browse to http://localhost:8000/
> 
> This functionality is also provided by htags-server(1), which is
> included in your package:
> 
>   https://www.mankier.com/1/htags-server
> 
> [BTW htags-server/htags-server.1 should be added to debian/global.manpages]

Well spotted. cheers for feedback.

OK. Punit (and I) prepared an updated package with the above fixes,
and htmake/htconfig dropped, so it's now pretty much a vanilla
upstream package.

Git repo is here:
https://github.com/punitagrawal/global

I've done a 5-day delayed NMU to experimental of 6.5.5-0.2. I'll put
some more details in #816924

This is essentially a candidate for unstable/stretch, should 'putting
v6 in unstable or stretch' be agreed-upon. So anyone who cares should
check it out and file bugs. 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Re: Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Philip Hands]

Wookey  writes:

> On 2016-11-30 16:56 +, Ian Jackson wrote:
>
>> On to some questions it raises for me:
>> 
>> > Optionally, "htags" can generate a dynamic index (which reduces disk
>> > space usage) but requires an http server setup to be able to serve the
>> > pages. In this scenario, you will also need to be able to execute CGI
>> > scripts as the symbol lookup is done when serving the http request (as
>> > opposed to pre-processed when using static pages).
>> > 
>> > And this last bit (integration with system web server) is the
>> > functionality that had security concerns raised by Ron [etc.]
>> 
>> So, to be clear, it is this functionality which is dropped in the
>> package that you and Wookey uploaded to experimental/delayed ?
>
> Said package is available as of today:
> https://buildd.debian.org/status/package.php?p=global=experimental
>  
> And that functionality was left in as it was a self-contained patch,
> pending determining whether in fact it remained useful/compatible or
> not. So you still get htmake and htconfig commands.
>
> However I have now done some checking, and no it doesn't work any more
> because it uses the btreeop command, which has been removed in current
> upstream, and the --action option to htags which is also gone.
>
> We are having a think about whether this code can be hacked about a
> bit to remain useful or if its approach is simply no longer relevant,
> given upstream changes. And I just said replying to Sam, there is a
> lot be said for not diverging significantly from upstream, because
> there is significant overhead in doing so. So this patch is currently
> likely to get dropped unless someone works out a way to make it
> work/be useful.
>
>> But AIUI this functionality remains:
>> 
>> > In addition to the above mechanisms, upstream also provides "htags"
>> > which can be used to generate an HTML version of the index. "htags"
>> > uses the index created by "gtags" and the source tree as input and
>> > generates html files which allow you to navigate the source code in
>> > the browser. By default, htags generates static pages which means you
>> > can browse the code in a local browser without requiring a web server.
>
>> So the impact for a user of the existing functionality the regression
>> is not that their entire workflow is disrupted.
>  
>> Rather (unless the software is improved) they have these choices:
>> 
>>  - Put up with pregenerated html indices.  Is the only downside
>>of this that they use additional disk space, and presumably
>>cpu time to generate them ?
>
> Correct. And they can easily be symlinked from say
> /var/www/html/global/ to make them a) accessible from system
> webserver and b) conveniently listed. The static HTML is 7G for a
> kernel source tree. The dynamic version is 3.3G. (or 4.8G with the
> suggested '--suggest2' options) So there is a x2 space overhead.
>
>>  - Run the htags server on a high-numbered port somehow.  (Is there
>>some kind of simple scriptery provided, to do this?)
>
> There is an example in the NEWS file, which should go in the man page. It's 
> trivial:
>
> cd HTML; python -m CGIHTTPServer
> or
> cd HTML; python3 -m http.server --cgi
>
> Then browse to http://localhost:8000/

This functionality is also provided by htags-server(1), which is
included in your package:

  https://www.mankier.com/1/htags-server

[BTW htags-server/htags-server.1 should be added to debian/global.manpages]

>>  - If the machine is not really multi-user, tear down the security
>>boundary defending the webserver from their user account, and give
>>their user account access to the webserver cgi directory (or plumb
>>it in with symlinks).  (Are there any instructions or scripts for
>>this?)  (Also this assumes that the source code is not super
>>secret.)
>
> I've just started playing with this to see what can be done (bearing
> in mind the question of how much effort we should put into this sort
> of upstream divergence anyway).
>
> Simply symlinking /var/www/global/sourcetree to  
> from 
> gives you a convenient list of globalised htags to look through if they are 
> static.
>
> The issue with dynamic htags (and the search functionality) is that by
> default your web server won't run .cgi scripts in arbitrary
> directories (for good reasons). I don't know if we can have a central
> .cgi that 'sources' the global.cgi or completion.cgi in the htags
> sourcetree/HTML dir and uses it, or if that's still not actually a
> good idea.
>
> Possibly something nifty with apache config scripts would work. Suggestions 
> welcome.
>
>> I don't know much about this, but several of these choices seem likely
>> to be plausible choices for many if not most current users of htags.
>
> Right. I think the functionality upstream provides is fine.

It strikes me that users are likely to fall into one of two groups:

  Local access by users that do not wish to offer remote access and are
  not offering 

Re: Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Wookey]

On 2016-11-30 16:56 +, Ian Jackson wrote:

> On to some questions it raises for me:
> 
> > Optionally, "htags" can generate a dynamic index (which reduces disk
> > space usage) but requires an http server setup to be able to serve the
> > pages. In this scenario, you will also need to be able to execute CGI
> > scripts as the symbol lookup is done when serving the http request (as
> > opposed to pre-processed when using static pages).
> > 
> > And this last bit (integration with system web server) is the
> > functionality that had security concerns raised by Ron [etc.]
> 
> So, to be clear, it is this functionality which is dropped in the
> package that you and Wookey uploaded to experimental/delayed ?

Said package is available as of today:
https://buildd.debian.org/status/package.php?p=global=experimental
 
And that functionality was left in as it was a self-contained patch,
pending determining whether in fact it remained useful/compatible or
not. So you still get htmake and htconfig commands.

However I have now done some checking, and no it doesn't work any more
because it uses the btreeop command, which has been removed in current
upstream, and the --action option to htags which is also gone.

We are having a think about whether this code can be hacked about a
bit to remain useful or if its approach is simply no longer relevant,
given upstream changes. And I just said replying to Sam, there is a
lot be said for not diverging significantly from upstream, because
there is significant overhead in doing so. So this patch is currently
likely to get dropped unless someone works out a way to make it
work/be useful.

> But AIUI this functionality remains:
> 
> > In addition to the above mechanisms, upstream also provides "htags"
> > which can be used to generate an HTML version of the index. "htags"
> > uses the index created by "gtags" and the source tree as input and
> > generates html files which allow you to navigate the source code in
> > the browser. By default, htags generates static pages which means you
> > can browse the code in a local browser without requiring a web server.

> So the impact for a user of the existing functionality the regression
> is not that their entire workflow is disrupted.
 
> Rather (unless the software is improved) they have these choices:
> 
>  - Put up with pregenerated html indices.  Is the only downside
>of this that they use additional disk space, and presumably
>cpu time to generate them ?

Correct. And they can easily be symlinked from say
/var/www/html/global/ to make them a) accessible from system
webserver and b) conveniently listed. The static HTML is 7G for a
kernel source tree. The dynamic version is 3.3G. (or 4.8G with the
suggested '--suggest2' options) So there is a x2 space overhead.

>  - Run the htags server on a high-numbered port somehow.  (Is there
>some kind of simple scriptery provided, to do this?)

There is an example in the NEWS file, which should go in the man page. It's 
trivial:

cd HTML; python -m CGIHTTPServer
or
cd HTML; python3 -m http.server --cgi

Then browse to http://localhost:8000/

>  - If the machine is not really multi-user, tear down the security
>boundary defending the webserver from their user account, and give
>their user account access to the webserver cgi directory (or plumb
>it in with symlinks).  (Are there any instructions or scripts for
>this?)  (Also this assumes that the source code is not super
>secret.)

I've just started playing with this to see what can be done (bearing
in mind the question of how much effort we should put into this sort
of upstream divergence anyway).

Simply symlinking /var/www/global/sourcetree to  
from 
gives you a convenient list of globalised htags to look through if they are 
static.

The issue with dynamic htags (and the search functionality) is that by
default your web server won't run .cgi scripts in arbitrary
directories (for good reasons). I don't know if we can have a central
.cgi that 'sources' the global.cgi or completion.cgi in the htags
sourcetree/HTML dir and uses it, or if that's still not actually a
good idea.

Possibly something nifty with apache config scripts would work. Suggestions 
welcome.

> I don't know much about this, but several of these choices seem likely
> to be plausible choices for many if not most current users of htags.

Right. I think the functionality upstream provides is fine.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature

Re: Bug#841294: Overrule maintainer of "global" to package a new upstream version

[Wookey]

On 2016-10-25 07:33 +0200, Tollef Fog Heen wrote:
> ]] Ian Jackson 
> 
> >  * Specifically, failed to give clear and constructive directions to
> >those willing to do the work;
> 
> I disagree with those characterisations. He's asked for clarifications
> on what is broken without anything resembling an adequate reply.

NOW, yes, when after 5 years people gave up filing bugs and brought it
to the technical committee. _Now_ Ron has ample time to write very
long responses. But the problem is that he didn't respond usefully in
the bugs, for years. Various people said 'the debian version doesn't
work, but the current upstream version does, can we upgrade
please?'. And were mostly ignored or given short shrift. (Vincent just
posted URLS).

Normal people, who are not of the Ian and Ron (and me) 'old-school
Debian, we quite like an argument' personality type, find this
_extremely_ offputting. I am only involved because I work with Punit
and he came to me asking for advice on what to do.

He's not posting here because the last thing he wants is an argument
(you know, the sort of quiet person who _really_ doesn't like
arguments). He thinks it's silly that we can't have a newer version of
global, but if that's what decided, then fine. He won't argue - he'll
just use upstream, debian will be stuck with the ancient version, and
we'll have lost a contributor. As he points out - there are many, many
other things he can do with his spare time that don't involve an
argument or dealing with a difficult maintainer.

So if this thing is only assessed in terms of who is the most
enthusiastic debater then Ron wins (he's very good at it). 

I realise this makes things harder for the TC to assess, because they
see a rather one-sided view. I would just ask them to bear
personalities and communication preferences in mind.

(As I said to Punit, 'everything used to be like this, but we improved
a lot in the last few years, and this is much rarer than it used to
be'). Debian used to have a (largely deserved) reputation as an
unpleasant project to work in. We've done a lot in the last few years
to improve that situation. I invite the TC to reflect on how this
would have played out if global had had a different maintainer. This
is (or should be) about attitude, responsiveness, and helpfulness, at
least as much as the technical (htags) debate.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: Digital signature