Re: [Savannah-hackers-public] bzr post-commit email hook

[Bob Proulx]

Glenn Morris wrote:
> Bob Proulx wrote:
> > I just made a big edit, import, cleanup to the authorized keys file.
> > Try it again now.  I it should allow you in.  You should be able to
> > log in from fencepost to mgt0 now.
> 
> Thanks. If security is a concern you should probably all disable my
> Savannah ssh access, since I don't foresee a need for me to login again.

If you don't mind I wish to keep you available for additional help at
potential times in the future.  You are active with Emacs and other
projects and your access there will continue to be beneficial.  One
never know what the future holds.

Bob

Re: [Savannah-hackers-public] bzr post-commit email hook

[Glenn Morris]

Assaf Gordon wrote:

> I assume it happens because over long period of time,
> various savannah admins changed various files
> (e.g. scripts on 'vcs'), but never checked them into the 'savane'
> repository.

I never knew anything about the repository. :)

> I'll have some time tomorrow, so I'll commit your updates
> to the repository as needed - otherwise we'll lose them
> again the next time savane in updated/installed.

Thanks. For the record, I also patched the installed bzr-email plugin
as shown at the end of this message.

>> I no longer have any writable bzr repos on Savannah to test with.
>
> I took the liberty of adding you (Glenn) to our dedicated test-project
>   https://savannah.nongnu.org/projects/test-project/

Thanks, but probably not needed.
I see there are only 8 bzr projects on Savannah that updated in the past
year (frankly, this is more than I expected). Only 3 Savannah projects
ever seem to have used the bzr-email plugin, and gsrc is the only one
still active. So I expect never to have to go through any of this again. :)

*** /usr/share/pyshared/bzrlib/plugins/email/emailer.py.ORIG2012-07-13 
08:57:32.0 -0400
--- /usr/share/pyshared/bzrlib/plugins/email/emailer.py 2017-02-10 
22:07:52.808398220 -0500
***
*** 58,63 
--- 58,64 
  fields = {
  'committer': self.revision.committer,
  'message': self.revision.get_summary(),
+ 'nick': self.branch._get_nick(local=True),
  'revision': '%d' % self.revno,
  'url': self.url()
  }
***
*** 92,98 
 to_file=outf
 )
  
! if len(self.revision.parent_ids) <= 1:
  # This is not a merge, so we can special case the display of one
  # revision, and not have to encur the show_log overhead.
  lr = log.LogRevision(self.revision, self.revno, 0, None)
--- 93,101 
 to_file=outf
 )
  
! # rgm: hack to always get the effect of verbose=True.
! # https://bugs.launchpad.net/bzr-email/+bug/1189292
! if len(self.revision.parent_ids) <= 0:
  # This is not a merge, so we can special case the display of one
  # revision, and not have to encur the show_log overhead.
  lr = log.LogRevision(self.revision, self.revno, 0, None)
***
*** 188,194 
  return
  for line in headers:
  key, value = line.split(": ", 1)
! result[key] = value
  return result
  
  def send(self):
--- 191,197 
  return
  for line in headers:
  key, value = line.split(": ", 1)
! result[key] = self._format(value)
  return result
  
  def send(self):
***
*** 205,212 
  mailer = self.mailer()
  if mailer == 'smtplib':
  self._send_using_smtplib()
! else:
  self._send_using_process()
  finally:
  self.repository.unlock()
  self.branch.unlock()
--- 208,217 
  mailer = self.mailer()
  if mailer == 'smtplib':
  self._send_using_smtplib()
! elif mailer == '/usr/bin/mail':
  self._send_using_process()
+ else:
+ raise errors.BzrError("Bad value for post_commit_mailer")
  finally:
  self.repository.unlock()
  self.branch.unlock()
***
*** 247,253 
  msg = EmailMessage(from_addr, to_addrs, subject, body)
  
  if diff:
! msg.add_inline_attachment(diff, self.diff_filename())
  
  # Add revision_mail_headers to the headers
  if header != None:
--- 252,258 
  msg = EmailMessage(from_addr, to_addrs, subject, body)
  
  if diff:
! msg.add_inline_attachment(diff, self.diff_filename(), 'x-diff')
  
  # Add revision_mail_headers to the headers
  if header != None:
***
*** 291,300 
  opt_post_commit_log_format = Option('post_commit_log_format',
  default='long', help="Log format for option.")
  opt_post_commit_difflimit = Option('post_commit_difflimit',
! default=1000, from_unicode=int_from_store,
  help="Maximum number of lines in diffs.")
  opt_post_commit_push_pull = Option('post_commit_push_pull',
! from_unicode=bool_from_store,
  help="Whether to send emails on push and pull.")
  opt_post_commit_diffoptions = Option('post_commit_diffoptions',
  help="Diff options to use.")
--- 296,305 
  opt_post_commit_log_format = Option('post_commit_log_format',
  default='long', help="Log format for option.")
  opt_post_commit_difflimit = Option('post_commit_difflimit',
! default=1, from_unicode=int_from_store,
  help="Maximum 

Re: [Savannah-hackers-public] bzr post-commit email hook

[Glenn Morris]

Bob Proulx wrote:

> I just made a big edit, import, cleanup to the authorized keys file.
> Try it again now.  I it should allow you in.  You should be able to
> log in from fencepost to mgt0 now.

Thanks. If security is a concern you should probably all disable my
Savannah ssh access, since I don't foresee a need for me to login again.

Re: [Savannah-hackers-public] bzr post-commit email hook

[Assaf Gordon]

Hi,

I've incorporated your changes here:
https://git.savannah.gnu.org/cgit/administration/savane.git/commit/?id=d23caef7bd0ec7bbddf06

Thanks for your help in fixing bzr!

-assaf

Re: [Savannah-hackers-public] bzr post-commit email hook

[Bob Proulx]

Glenn Morris wrote:
> Bob Proulx wrote:
> > Yes.  Looks like bzr is logging to /var/lib/bzr/bzr-anon.log.  Which I
> > think should be /var/log/bzr/bzr-anon.log instead.  I am going to make
> > that change.  Will copy the logrotate over for the new location.
> 
> It's as you prefer. But you will need to edit
> /opt/savannah/bin/sv_membersh as well, for the non-anonymous log.

I moved the log file for both of those to /var/log.  I modified
/etc/logrotate.d/bzr for the new location.  I shorted the logging save
time because there is a big push via rms to avoid logging data.

> >> Other relevant stuff I did from the old ChangeLog on mgt:
> >> 
> >>   * [vcs]: `apt-get install locales'.  Generate only en_US.utf8,
> >>   and en_US.ISO8859-1, leave system default as none (ie leave
> >>   /etc/default/locale empty).
> >
> > Why en_US.ISO8859-1 and not en_US.UTF-8 as en_US.UTF-8 has become the
> > standard these days?  Of course any admin using a different locale is
> > free to set up one that is useful to them.  But by default only
> > en_US.UTF-8 was set up on the new servers.
> 
> en_US.ISO8859-1 is what bzr needed, because Savannah gecos information
> was stored in latin-1. See my comment in sv_membersh and the cited thread
> https://lists.ubuntu.com/archives/bazaar/2013q3/075681.html .

Ouch!  Thanks for the information on that.

Seems that en_US.ISO-8859-1 is available.  Someone else got to it first.

root@vcs0:~# dpkg-reconfigure locales
Generating locales...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.ISO-8859-1... up-to-date
  en_US.UTF-8... up-to-date
  en_ZA.UTF-8... done
  en_ZM.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.

> If Savannah gecos has since changed to utf-8, then the bzr LANG line in
> sv_membersh could be changed or removed.

I don't think there has been any change.  But that seems like a good
improvement.  It would need to be made in the PHP that stores the
information into the database.  If/When that is changed then it would
flow through as UTF-8.  We could use iconv on the existing data and
migrate it in place.  A good task for some time in the future.

Bob

Re: [Savannah-hackers-public] bzr post-commit email hook

[Bob Proulx]

Glenn Morris wrote:
> ChangeLog tells me to update the log on mgt0 instead, but I cannot
> log in to mgt0.

I just made a big edit, import, cleanup to the authorized keys file.
Try it again now.  I it should allow you in.  You should be able to
log in from fencepost to mgt0 now.

In other discussion there was implications that we should layer up
such that logins from the old less secure servers to the new should
not be allowed.  In case there is a breach on the old then they can't
be used to lever into the new.  So can login from mgt0 to the old but
not from the old to the new.  (However that was not the problem you
experience.  That was definitely an almost empty authorized keys file.)

Bob

Re: [Savannah-hackers-public] bzr post-commit email hook

[Assaf Gordon]

Hello Bob and Glenn,

I'll add one more thing:

> On Feb 10, 2017, at 19:38, Bob Proulx  wrote:
> [...]
> 
> /opt/savannah/savane/backend/accounts/sv_membersh is a checked out
> copy of the git source for Savannah's savane source code.  That is the
> source code copy of the file.  That isn't the live file.
> 
>  vcs:~# getent passwd rwp
>  rwp:x:65821:1003:Bob Proulx:/srv:/usr/local/bin/sv_membersh
> 
>  root@vcs0:~# getent passwd rwp
>  rwp:x:65821:1003:Bob Proulx:/srv:/opt/savannah/bin/sv_membersh
> 
> The live file is /srv:/opt/savannah/bin/sv_membersh on vcs0 and
> /usr/local/bin/sv_membersh on vcs.

The 'live' file /opt/savannah/bin/sv_membersh was not copied from any
other server. It is the result of installing the checked-in
source of the 'savane' repository by running:

cd /opt/savannah/savane
./configure --prefix=/opt/savannah/
make
make install

which is why:

>> vcs0:sv_membersh is not synced with vcs:sv_membersh .
>> In particular, it is missing all the bzr changes I made years ago.
>> I'm not sure how that could happen

I assume it happens because over long period of time,
various savannah admins changed various files
(e.g. scripts on 'vcs'), but never checked them into the 'savane'
repository.
Quite understandable, as it is a big mess, and many scripts
aren't checked-in at all (e.g. all those in "vcs:/usr/src/infra").


On Feb 10, 2017, at 22:30, Glenn Morris  wrote:

>I think I've copied all the changes across now, so it might be working.

I'll have some time tomorrow, so I'll commit your updates
to the repository as needed - otherwise we'll lose them
again the next time savane in updated/installed.


> I no longer have any writable bzr repos on Savannah to test with.

I took the liberty of adding you (Glenn) to our dedicated test-project
  https://savannah.nongnu.org/projects/test-project/

It has repositories of all types including bzr:
  http://bzr.savannah.gnu.org/lh/test-project/trunk/view/head:/README

You should be able to commit changes with:
  bzr branch bzr+ssh://g...@bzr.savannah.nongnu.org/test-project/branch

If you could help us test it, it would be very much appreciated.
If not - I'll try to test it a bit later.

Many thanks!!
 - assaf

Re: [Savannah-hackers-public] bzr post-commit email hook

[Glenn Morris]

I think I've copied all the changes across now, so it might be working.
I no longer have any writable bzr repos on Savannah to test with.

Re: [Savannah-hackers-public] bzr post-commit email hook

[Glenn Morris]

Bob Proulx wrote:

> Yes.  Looks like bzr is logging to /var/lib/bzr/bzr-anon.log.  Which I
> think should be /var/log/bzr/bzr-anon.log instead.  I am going to make
> that change.  Will copy the logrotate over for the new location.

It's as you prefer. But you will need to edit
/opt/savannah/bin/sv_membersh as well, for the non-anonymous log.

>> Other relevant stuff I did from the old ChangeLog on mgt:
>> 
>>   * [vcs]: `apt-get install locales'.  Generate only en_US.utf8,
>>   and en_US.ISO8859-1, leave system default as none (ie leave
>>   /etc/default/locale empty).
>
> Why en_US.ISO8859-1 and not en_US.UTF-8 as en_US.UTF-8 has become the
> standard these days?  Of course any admin using a different locale is
> free to set up one that is useful to them.  But by default only
> en_US.UTF-8 was set up on the new servers.

en_US.ISO8859-1 is what bzr needed, because Savannah gecos information
was stored in latin-1. See my comment in sv_membersh and the cited thread
https://lists.ubuntu.com/archives/bazaar/2013q3/075681.html .

If Savannah gecos has since changed to utf-8, then the bzr LANG line in
sv_membersh could be changed or removed.

Re: [Savannah-hackers-public] bzr post-commit email hook

[Bob Proulx]

Glenn Morris wrote:
> PS I would also copy /etc/logrotate.d/bzr across.

Yes.  Looks like bzr is logging to /var/lib/bzr/bzr-anon.log.  Which I
think should be /var/log/bzr/bzr-anon.log instead.  I am going to make
that change.  Will copy the logrotate over for the new location.

> Other relevant stuff I did from the old ChangeLog on mgt:
> 
>   * [vcs]: `apt-get install locales'.  Generate only en_US.utf8,
>   and en_US.ISO8859-1, leave system default as none (ie leave
>   /etc/default/locale empty).

Why en_US.ISO8859-1 and not en_US.UTF-8 as en_US.UTF-8 has become the
standard these days?  Of course any admin using a different locale is
free to set up one that is useful to them.  But by default only
en_US.UTF-8 was set up on the new servers.

Bob

Re: [Savannah-hackers-public] bzr post-commit email hook

[Glenn Morris]

Here are the bzr notes from a few years ago, which may or not still be useful:

https://lists.gnu.org/archive/html/savannah-hackers-public/2013-06/msg00042.html

It should basically be the bzr-email plugin, yes.

Re: [Savannah-hackers-public] bzr post-commit email hook (was: gsrc-commit auto messages stopped working)

[carl hansen]

On Wed, Feb 8, 2017 at 9:39 PM, Bob Proulx  wrote:

> Hello Carl,
>
> carl hansen wrote:
> > still still not working
>
> Sorry there has been delays.  We have been overwhelmed trying to
> juggle too many things.
>
> ​I know the feeling. Luckily it doesn't matter.
​


> I would love to help fix your hook problem.  But I enter this knowing
> nothing about bzr.  I've never used bzr before.  I poke into your bzr
> repository and explore.
>
>   vcs:/srv/bzr/gsrc# grep -rl gsrc-commit .
> ./trunk/.bzr/branch/branch.conf
>
>   vcs:/srv/bzr/gsrc# cat ./trunk/.bzr/branch/branch.conf
> last_revision_mailed = bran...@invergo.net-20130610174455-
> orqjagls9tgp1noq
> post_commit_to = gsrc-com...@gnu.org
> post_commit_body = ""
> post_commit_subject = $nick r$revision: $message
>
> Okay.  That seems to be the configuration for it.  But how does that
> even do anything?  I was expecting to see a hooks directory or
> something.  Searching the web for documentation on setting up bzr
> hooks did not yield anything useful to me.
>
> Do you know how hook scripts with bzr work?  Perhaps you can help me
> help you.
>
> Bob
>
​Ok I am poking around.
​$  bzr help hooks
spits out some text including:
"post_commit
~~~
Introduced in: 2.0
Called after a commit is performed on a tree. The hook is called with
a bzrlib.mutabletree.PostCommitHookParams object. The mutable tree the
commit was performed on is available via the mutable_tree attribute of
that object."

Re: [Savannah-hackers-public] bzr post-commit email hook (was: gsrc-commit auto messages stopped working)

[Assaf Gordon]

Hi Bob,

> On Feb 9, 2017, at 00:39, Bob Proulx  wrote:
> 
> Do you know how hook scripts with bzr work?  Perhaps you can help me
> help you.

Not sure if that's helpful, but last month I installed a missing 'email'
plugin:
  
http://lists.gnu.org/archive/html/savannah-hackers-public/2017-01/msg00046.html
clearly just installing it was not enough to fix the problem.

Perhaps one of these files will provide a hint ?

question for BZR experts: is this plugin even relevant for the email hooks?

===
vcs0:~$ dpkg -L bzr-email | grep 'py$'
/usr/share/pyshared/bzrlib/plugins/email/tests/__init__.py
/usr/share/pyshared/bzrlib/plugins/email/tests/testemail.py
/usr/share/pyshared/bzrlib/plugins/email/__init__.py
/usr/share/pyshared/bzrlib/plugins/email/emailer.py
/usr/lib/python2.7/dist-packages/bzrlib/plugins/email/tests/__init__.py
/usr/lib/python2.7/dist-packages/bzrlib/plugins/email/tests/testemail.py
/usr/lib/python2.7/dist-packages/bzrlib/plugins/email/__init__.py
/usr/lib/python2.7/dist-packages/bzrlib/plugins/email/emailer.py
===

regards,
 - assaf

Re: [Savannah-hackers-public] bzr post-commit email hook (was: gsrc-commit auto messages stopped working)

[Bob Proulx]

Hello Carl,

carl hansen wrote:
> still still not working

Sorry there has been delays.  We have been overwhelmed trying to
juggle too many things.

I would love to help fix your hook problem.  But I enter this knowing
nothing about bzr.  I've never used bzr before.  I poke into your bzr
repository and explore.

  vcs:/srv/bzr/gsrc# grep -rl gsrc-commit .
./trunk/.bzr/branch/branch.conf

  vcs:/srv/bzr/gsrc# cat ./trunk/.bzr/branch/branch.conf
last_revision_mailed = bran...@invergo.net-20130610174455-orqjagls9tgp1noq
post_commit_to = gsrc-com...@gnu.org
post_commit_body = ""
post_commit_subject = $nick r$revision: $message

Okay.  That seems to be the configuration for it.  But how does that
even do anything?  I was expecting to see a hooks directory or
something.  Searching the web for documentation on setting up bzr
hooks did not yield anything useful to me.

Do you know how hook scripts with bzr work?  Perhaps you can help me
help you.

Bob

Re: [Savannah-hackers-public] bzr post-commit email hook (was: gsrc-commit auto messages stopped working)

[carl hansen]

still still not working

On Tue, Jan 24, 2017 at 2:08 PM, carl hansen 
wrote:

> still not working
>
> On Sun, Jan 22, 2017 at 1:23 PM, Assaf Gordon 
> wrote:
>
>> Hello Carl,
>>
>> > On 01/20/2017 03:42 AM, carl hansen wrote:
>> >> When I commit to the bzr repository it used
>> >> to send a message automatically to "gsrc-com...@gnu.org". Since Jan
>> 5, that
>> >> has not happened. Probably an untied shoelace somewhere.
>>
>>
>> I've installed the missing bazaar plugin.
>> Please let us know if the commit emails are ok now.
>>
>> regards,
>>  - assaf
>>
>>
>>
>>
>> For savannah admins / future reference:
>>
>> Bazaar plugins guide:
>>  http://doc.bazaar.canonical.com/plugins/en/
>>
>>
>> Luckily the plugin was available as a package:
>>  http://packages.trisquel.info/belenos/bzr-email
>>
>>
>> Current bzr plugins on the new 'vcs0':
>> 
>> # bzr plugins
>> bash_completion 2.7.0dev1
>>   Generate a shell function for bash command line completion.
>> changelog_merge 2.7.0dev1
>>   Merge hook for GNU-format ChangeLog files
>> email
>>   Sending emails for commits and branch changes.
>> etckeeper
>>   Runs etckeeper pre-commit when necessary.
>> grep 2.7.0dev1
>>   Print lines matching PATTERN for specified files and revisions.
>> launchpad 2.7.0dev1
>>   Launchpad.net integration plugin for Bazaar.
>> loggerhead 1.18.1
>>   Loggerhead web viewer for Bazaar branches.
>> netrc_credential_store 2.7.0dev1
>>   Use ~/.netrc as a credential store for authentication.conf.
>> news_merge 2.7.0dev1
>>   Merge hook for bzr's NEWS file.
>> po_merge 2.7.0dev1
>>   Merge hook for ``.po`` files.
>> weave_fmt 2.7.0dev1
>>   Weave formats.
>> 
>>
>>
>

Re: [Savannah-hackers-public] bzr post-commit email hook (was: gsrc-commit auto messages stopped working)

[carl hansen]

still not working

On Sun, Jan 22, 2017 at 1:23 PM, Assaf Gordon  wrote:

> Hello Carl,
>
> > On 01/20/2017 03:42 AM, carl hansen wrote:
> >> When I commit to the bzr repository it used
> >> to send a message automatically to "gsrc-com...@gnu.org". Since Jan 5,
> that
> >> has not happened. Probably an untied shoelace somewhere.
>
>
> I've installed the missing bazaar plugin.
> Please let us know if the commit emails are ok now.
>
> regards,
>  - assaf
>
>
>
>
> For savannah admins / future reference:
>
> Bazaar plugins guide:
>  http://doc.bazaar.canonical.com/plugins/en/
>
>
> Luckily the plugin was available as a package:
>  http://packages.trisquel.info/belenos/bzr-email
>
>
> Current bzr plugins on the new 'vcs0':
> 
> # bzr plugins
> bash_completion 2.7.0dev1
>   Generate a shell function for bash command line completion.
> changelog_merge 2.7.0dev1
>   Merge hook for GNU-format ChangeLog files
> email
>   Sending emails for commits and branch changes.
> etckeeper
>   Runs etckeeper pre-commit when necessary.
> grep 2.7.0dev1
>   Print lines matching PATTERN for specified files and revisions.
> launchpad 2.7.0dev1
>   Launchpad.net integration plugin for Bazaar.
> loggerhead 1.18.1
>   Loggerhead web viewer for Bazaar branches.
> netrc_credential_store 2.7.0dev1
>   Use ~/.netrc as a credential store for authentication.conf.
> news_merge 2.7.0dev1
>   Merge hook for bzr's NEWS file.
> po_merge 2.7.0dev1
>   Merge hook for ``.po`` files.
> weave_fmt 2.7.0dev1
>   Weave formats.
> 
>
>