Re: Build failed in Jenkins: Build branch "master" » ubuntu-xenial-qt4-autotools-extended #794

[Christian Ridderström]

On Wed, 28 Feb 2018 at 20:27, Richard Heck  wrote:

> On 02/28/2018 01:25 PM, ci-...@inria.fr wrote:
> >
> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-xenial-qt4-autotools-extended/794/--
> > Started by an SCM change
> > Building remotely on lyx-linux1 (linux) in workspace <
> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-xenial-qt4-autotools-extended/ws/
> >
> > [WS-CLEANUP] Deleting project workspace...
> > java.io.IOException: Failed to mkdirs: <
> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-xenial-qt4-autotools-extended/ws/
> >
> >   at hudson.FilePath.mkdirs(FilePath.java:1191)
>
> Out of disk space?
>
​Yes. By a coincidence I'd actually just fixed it before reading this.
I've restarted some of the CI jobs manually to check they now work.

Cheers,
Christian

Re: CI job: Check author email in commit log

[Christian Ridderström]

Hi,

Just FYI, the CI job that checks authors' email has triggered.
https://ci.inria.fr/lyx/job/support/job/Check-author-email-in-commit-log/
Here's the difference:

Latest log:
cca365f26c 2018-02-15 11:28:34 -0800 name=Alexander Dunlap
email=alexander.dun...@gmail.com
c070c2f1f8 2018-01-10 02:38:31 +0100 name=Uwe Stöhr email=uwesto...@web.de
bb650bf8c2 2018-01-10 02:12:40 +0100 name=Uwe Stöhr email=uwesto...@web.de


So these commits by Alexander and Uwe were likely not sent to the list, but
I guess you did notice the "spam" on lyx-devel..?

I still have to update the data in the CI job itself so it stops triggering
in the future. For now I disabled the CI job to avoid further spam.

Rationale: Help us catch commits for which we wouldn't automatically be
notified via the the lyx-cvs-list.

Cheers,
Christian

PS
One of the CI nodes had also ran out of space, resulting in builds failing.
I've logged in an deleted old jobs freeing up 20-30 GB. So the CI nodes
should work again.


On 21 August 2017 at 03:31, Scott Kostyshak <skost...@lyx.org> wrote:

> On Sun, Aug 20, 2017 at 07:20:56PM +0200, Christian Ridderström wrote:
> > Hi,
> >
> > FYI, I've set up this CI job
> >
> > https://ci.inria.fr/lyx/job/support/job/Check-author-email-
> in-commit-log/
> >
> > that once a week checks the git log for commits where the author's
> > e-mail does not say it's from @lyx.org.
> >
> > Rationale: Help us catch commits for which we wouldn't automatically be
> > notified via the the lyx-cvs-list.
>
> Thanks for setting that up, Christian!
>
> Scott
>

Re: 404 on manuals on lyx.org

[Christian Ridderström]

Tommaso Cucinotta <tomm...@lyx.org> writes:

> On 20/08/2017 20:20, Christian Ridderström wrote:
>> - Perhaps write out English, Español etc in the headings
>
> I made an attempt to rotate the text, to keep the table compact, via
> CSS attributes, but the rotated one didn't show well, because it was
> rotating the cell borders as well :(. Perhaps a hack might be to keep
> the heading cell as a perfect square ;-P.

I tested making the header text smaller, have a look at the playground
page now. Looks ok to me, but maybe you think it's still to wide?

>> - The link texts could be LyX vs PDF instead of EN/FR etc.
>
> tried to upload a .pdf icon from LyX, I've put it here:

> Now, how can I get that image to show up in the wiki ? What is the URL
> it becomes visible as ? I tried to follow pmwiki's (upload:, attach:),
> but didn't work.

Seems like you've solved it now.

I added example of how you can use an image as the "link text".
Also how it can be resized. But it's better if the uploaded image has
the proper size from the beginning.

/Christian

Re: numbers/ids of posts on mail-archive.com vs. gmane.org?

[Christian Ridderström]

Scott Kostyshak  writes:

> On Mon, Aug 07, 2017 at 11:49:19AM +0200, Pavel Sanda wrote:
>> Scott Kostyshak wrote:
>> > When I'm reading email in mutt, all I do is press a shortcut (I use [ma for
>> > "mail archive"), and it copies the above link to my clipboard so I can 
>> > paste
>> > it in my email (or on trac). If anyone is using mutt, let met know if you
>> > want the script.
>> 
>> Nice hack, I'll be interested to see it. P
>
> Let me clean up the hack to make it a little less hackish. I've been meaning 
> to
> clean it up so this will be a good opportunity.

In case there's anyone else using Gnus, and if I need it in the future,
below is a function I just wrote to copy the message ID.

If I execute the function in a message buffer it will find the
message-ID and copy it into the "kill ring", so I can later e.g. "yank" it
into a message I'm composing.

(defun copy-message-ID ( arg)
  "Copy Message-ID field.
This command must be executed in a gnus message buffer."
  (interactive)
  (save-excursion
(beginning-of-buffer)
(gnus-summary-toggle-header 1)
(search-forward-regexp "^Message-ID: ")
(let ((beg-of-ID (point)))
  (end-of-line)
  (kill-ring-save beg-of-ID (point)))
(gnus-summary-toggle-header -1)
)
  )

/Christian

PS
Caveat: It's a long time since I wrote lisp so the function might not be
very robust etc.

Re: How to access the list efficiently? Re: Long threads? / list etiquette?

[Christian Ridderström]

Guenter Milde <mi...@users.sf.net> writes:

> On 2017-08-21, Pavel Sanda wrote:
>> Christian Ridderström wrote:
>>> * But then I stumbled upon a still running news service for the lists.
>>>   Possibly this is related to the old 'gmane' service. E.g.
>
>>> nntp+news.gwene.org:gmane.editors.lyx.devel
>
>>> The 'news' interface is working well so far. And I can go back and see
>>> posts from something like 2003 or so.
>
>> I added the link above to our email lists page.
>
> Also the "traditional" GMANE news interface works fine with lyx. I use it
> with the slrn news reader:
>
>   slrn -n -h news.gmane.org

Thanks Günter. I've now switched my Gnus to use 'news.gmane.org' instead
of 'news.gwene.org' and it seems to work fine.

I've therefore also updated the wiki page to refer to 'news.gmane.org'.

/Christian

The updated note:

-> '''Note:''' As of 14 September 2016, the GMANE web interface is being
   rebuilt following a transfer of ownership. Unfortunately access to
   the LyX mailing lists through the GMANE web interface is still (Aug
   2017) not working.  However, the 'news' service is still operational
   and you can use a news reader to e.g. access posts in the developer's
   list through

--> @@nntp+news.gmane.org:gmane.editors.lyx.devel@@.

Re: numbers/ids of posts on mail-archive.com vs. gmane.org?

[Christian Ridderström]

Christian Ridderström <c...@lyx.org> writes:

> Hi,
>
> The wiki has "InterLinks", which is simply a convenient way to refer
> to certain web pages.  For instance, this markup
>
> bug:10481
>
> is automatically converted into this link
> http://www.lyx.org/trac/ticket/10481.
>
> Similarly, there's prefixes like LyxDevelPost and LyxUsersPost which
> were used to refer to gmane.org. Unfortunately gmane.org doesn't seem
> to contain the LyX lists anymore :-(
>
> I just now tested repointing the prefix "LyxDevelPost" from gmane.org
> to mail-archive.com, so e.g.this currently works: LyxDevelPost:200333
>
> There's a problem though... it seems the IDs of posts, i.e. the
> number, is different for gmane.org versus mail-archive.com. My
> repointing would therefore mean that a pre-existing link into
> gmane.org "evolved" from initially working fine, to not working at
> all, to suddenly pointing to the wrong post in
> mail-archive.com. *sigh*

An update in case someone stumbles on this thread in the future.

As a news service, 'gmane' is still around. It's the web interface
that's not working. In another thread Günter wrote:

… the "traditional" GMANE news interface works fine with lyx. I
use it with the slrn news reader:

  slrn -n -h news.gmane.org

Separately I accidentally found that the news interface also works
through 'news.gwene.org', e.g. as:

nntp+news.gwene.org:gmane.editors.lyx.devel

/Christian

Re: Changes to git branches, regarding 2.3.0 development

[Christian Ridderström]

Scott Kostyshak <skost...@lyx.org> writes:

> On Sun, Aug 20, 2017 at 03:45:11PM +0200, Christian Ridderström wrote:
>
>> FYI, the CI jobs are testing branch 'master', so as it is we don't have
>> automatic testing of changes to branch '2.3.x'.
>
> OK thanks for the warning.

It should be possible to make a CI job operate on multiple branches, but
there's *ahem* apparently a risk of triggering a lot of jobs that way.

So if someone wants a CI job for 2.3.x, it might be easier to simply
duplicate one of the existing jobs.

/Christian

Re: How to access the list efficiently? Re: Long threads? / list etiquette?

[Christian Ridderström]

Scott Kostyshak <skost...@lyx.org> writes:

> On Fri, Aug 04, 2017 at 08:22:40AM +0200, Christian Ridderström wrote:
>> On 2 August 2017 at 10:03, Scott Kostyshak <skost...@lyx.org> wrote:
>> 
>> This worked well, and the drawback was mainly when people posted in HTML,
>> or included URLs which I manually had to copy and paste.
>
> HTML is still (probably even worse) a problem. Whenever I receive
> HTML-only emails, I respond and explain why I would prefer a
> multi-part MIME email. In the end, it's not really a problem: Mutt can
> automatically convert the HTML to text, using e.g. w3m. Or you can
> press the keys "vm" and it will view the email in your browser.

HTML seems to work fine in Gnus in Emacs. The text gets colours and
italics etc.

As I'm 'off' the gmail interface, it's suddenly (again) very noticeable
when people do things like bottom post, use very long lines or HTML with
some weird background colour.

Gnus also nags at me if I try to do things like include long lines or
top post.

/Christian

Re: How to access the list efficiently? Re: Long threads? / list etiquette?

[Christian Ridderström]

"Paul A. Rubin" <parubi...@gmail.com> writes:

> On 08/04/2017 10:02 PM, Scott Kostyshak wrote:
>> On Fri, Aug 04, 2017 at 08:22:40AM +0200, Christian Ridderström wrote:
>>> On 2 August 2017 at 10:03, Scott Kostyshak <skost...@lyx.org> wrote:
>>>
>>>>> 
>>>>> I'm using gmail's web interface these days. 
>>>>> 

>>> A decade or so ago I used 'pine' which worked really well for my
>>> 

> FWIW, I subscribe through GMail, using a filter to put the dev and
> user lists in separated virtual folders, but access my GMail account
> in Thunderbird (IMAP). The messages are individual. I don't know if
> there's a way to maintain threads in T-bird (haven't tried), but I can
> always use a "Quick Filter" to isolate a thread.

In case anyone is interested, here's where I'm currently at:

* I'm now using 'Gnus' in Emacs, accessing the list through a 'news'
  interface.

  Note: I'm a fast typer in Emacs, so I thought I should find a client
  that works directly in Emacs => kind of randomly picked Gnus.

* At first I used IMAP to Gmail from within Gnus to access the lists,
  using my @lyx.org-account. I actually automatically got groups such as
  'lyx-devel' since I already had filters that put 'labels' on lyx-devel
  etc. This way to interface the lists seems like it would work.

  Note: A drawback is that this Gmail account did not have all old
  posts, so I wouldn't be able to go arbitrarily back in time.

* But then I stumbled upon a still running news service for the lists.
  Possibly this is related to the old 'gmane' service. E.g.

nntp+news.gwene.org:gmane.editors.lyx.devel

The 'news' interface is working well so far. And I can go back and see
posts from something like 2003 or so.

It was some work to mark all the old posts. If anyone is curious, we're
talking about >150k posts on the devel lists. Marking most of them as
read while keeping the interesting ones "unread" took a little work,
especially as I intitially didn't know how to use 'Gnus' correctly.

As I scanned through old articles back to 2015 or so, I "ticked" the
ones I thought might be interesting to read/re-read.  I don't think I
could've done what I did in Gnus using e.g. Gmail, it simply wouldn't
have worked in practice with that amount of articles -- to slow.

Using 'Gnus' seems _way_ faster than using the gmail interface.

There's a learning threshold though... E.g. I only recently started
using M-g, for 'gnus-summary-rescan-group', as a quick way to refresh
and be notified about new posts.

One possible drawback with using 'news': There might be a delay before
I see posts as they have to go the list and then be forwarde to
news.gwene.org before I can see them.

If anyone is interested in details about my configuration etc I'd be
happy to share.

/Christian

Re: remove upload link from wiki?

[Christian Ridderström]

Scott Kostyshak  writes:

> It doesn't seem like it's going to be fixed soon. Is it possible just
> to remove the link?

The wiki page

   http://wiki.lyx.org/Site/PageActions

defines the meaning of the links at the top right side of a page.

I've repointed the link "Upload" to the page
   http://wiki.lyx.org/Site/AboutUploading

which tells the user to do FTP.

I suppose we could rename "Upload" to "FTP" and let it point to the ftp
service at the lyx server.
/Christian

PS.
I've got no idea if this post will reach the list.
Currently I'm trying to learn to use gnus from within Emacs to interface
the list.

Here goes...

And it didn't go, so this is actually the second time I'm trying to post
this.

Re: wiki - Category link pages

[Christian Ridderström]

Tommaso Cucinotta  writes:

> Hi,
>
> I took some time to clean-up the empty template that was annoyingly showing 
> up when clicking on the bottom category link of each wiki page, namely:
>
> " summary/description of the page and it's purpose. It could be something like: 
> How to insert equations with specified equation numbers.>" ...
>
> which I replaced with a simple
>
> !!! Pages in Category: {*$Name}
>
> So, now those links should look like as having some more meaningful and 
> usable contents:
>
>   http://wiki.lyx.org/Category/Development
>   http://wiki.lyx.org/Category/Documentation
>   http://wiki.lyx.org/Category/Glossary
>   http://wiki.lyx.org/Category/Keyboard
>   http://wiki.lyx.org/Category/Layouts
>   http://wiki.lyx.org/Category/Manuals
>
> [just in case u notice anything mistakenly broken on the wiki category 
> links...]

Hi Tommaso,

I'm not sure I understand what you did...

Did you actually edit the template page, or did you create a number of
pages...?

I looked at a page history, e.g.:
http://wiki.lyx.org/Category/Development?action=diff

It seems that in 2011 we were fighting some spam bots, which initially
created the page. Then a developer replaced the page content with the
content of the template page. I have no idea why.

IIRC, the wiki is configured to automatically list pages in a category
for categories where there's no category specific page. This behaviour
is controlled by some template page (I don't remember which right now).

The "correct" fix is therefore to delete the spammed category pages.

If you prefer the content of the category pages you created we could
find the template page for non-existing category pages and modify that
accordingly.

What's your preference?
/Christian

PS
How to delete a wiki page:
I google how to "delete" a wiki page. You simply replace it's entire
markup with the single word "delete". That's it. See e.g. this page that
I just "deleted":

http://wiki.lyx.org/Category/Layouts

Re: [LyX/master] oops, git is playing games with me.

[Christian Ridderström]

Pavel Sanda  writes:

> The way I can comfortably push this tree onto web server will make
> it pages of it's own so pmwiki format is of no help.
>
> If you want to make part of wiki the only reasonable way I see is
> to cooperate with Christian to create some sort of ftp/scp uploading
> tunnel. I have no problem with such solution, but it's in your hands
> then :)

There's an FTP on the LyX server... I sent out details earlier today.
So perhaps 'sshfs' is an option for uploading things.

I'm not really sure what you're trying to achieve, e.g.

- From where are the files uploaded? A developer's computer? A CI job?
- Are uploads to be triggered automatically?
- How often would we do new uploads? Once per release, a few times per release?
- To where should the files be uploaded?

As I wrote in some other post I also thinks it's good to have a separate
staging area for review, and a separate deployment area to which users
are referred.
/Christian

PS.
A long time ago I wrote some PmWiki markup that lists files in a folder.
It might be possible to use this markup to automatically generate lists/tables.

Re: 404 on manuals on lyx.org

[Christian Ridderström]

Tommaso Cucinotta  writes:

> On 15/08/2017 22:11, Uwe Stöhr wrote:
>> So fine with me if you change the order.
>
> done. Also, I gave a try to pmwiki tables, to get to a more compact form:
>
>   http://wiki.lyx.org/LyX/Manuals2

The table looks nice to me. Some minor thoughts:

- Perhaps write out English, Español etc in the headings
- The link texts could be LyX vs PDF instead of EN/FR etc.
- Perhaps state the LyX release before the table

>   (might be visually improved with some .lyx/.pdf icons, plus
>   country-code flags, but don't know how to attach images to the
>   lyx.org wiki)

Do you mean how to make an image appear on a wiki page
(http://www.pmwiki.org/wiki/PmWiki/Images) or how to upload an image
onto the server?

General tip: Test things in the group "/Test", e.g. see here
http://wiki.lyx.org/Test/Images

Or use a playground page, e.g. http://wiki.lyx.org/Playground/AngusWikiSandbox

/Christian

PS. It's a good idea to create a page like Manuals2 in the group
Playground/ for others to review. Then when people like it, move
improvements into LyX/Manuals. This is because we likely don't want to
have two pages. This way we won't have to figure out how to delete a
wiki page (which I've completely forgotten how to do).

PPS.

We can also consider maintaining an official page

http://www.lyx.org/LyX/Manuals

as the web pages are actually wiki pages, the task of creating and
updating the page is the same.

CI job generating PDFs? Was: 404 on manuals on lyx.org

[Christian Ridderström]

Guenter Milde <mi...@users.sf.net> writes:

> On 2017-08-11, Christian Ridderström wrote:
>> On 11 August 2017 at 13:03, Pavel Sanda <sa...@lyx.org> wrote:
>>> Christian Ridderström wrote:
>
>>> > [*] It might make sense to have a CI job that builds PDFs etc from the
>>> > manuals as a separate test.
>
> This is already done by Kornel's export test suite. All manuals are
> regularely exported and failures mailed to the list.
>
> There are known failures in additional tests for various "exotic" export
> routes and configurations but AFAIK all documentation exports fine using the
> set standard export format (mostly PDF(pdflatex)).

Some months ago I created a CI job that runs (some of?) the tests.
The CI job is currently located here

https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/

Due to the many tests failing it didn't seem so useful to deploy.
Here's the (presumably) relevant line from the log:

68% tests passed, 2073 tests failed out of 6522

That line is from job#310, i.e. here (might require logging in):

https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/310/console

I also logged in to the CI node and checked the workspace to look for
PDFs [2]. There's a whole bunch of PDFs in that list, so maybe
we already have a CI job that generates (some) of the manuals as PDFs?

The workspace of the most recent job can be accessed like this:

https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/ws/

>From there I then navigated to e.g.


https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/ws/build/autotests/out-home/sv/splash_defaultF.pdf3

I was able to download that file and view it, although my mac required
me (!!!???) to first rename the extension from .pdf3 to .pdf, even when
I tried to launch 'Preview' on the file from the command line.

/Christian


[1] This job is running regulary. Previously it checked if Git had
changed every eight hours, but as it's a big job I just tweaked this to
only check Git on Thursdays.

[2] I logged in to the CI node and checked the workspace to look for
PDFs. This is what I found

ci@ubuntu1604:~/workspace/testing-Jenkins/ctest$ find . -name "*.pdf"
./lib/examples/beamer-icsi-logo.pdf
./lib/doc/de/clipart/Zusammenfassung.pdf
./lib/doc/clipart/3D-structure-distort.pdf
./lib/doc/clipart/2D-intensity-plot.pdf
./lib/doc/clipart/Abstract.pdf
./lib/doc/clipart/endnotes.pdf
./lib/doc/clipart/without_fntright.pdf
./lib/doc/clipart/Star-structure.pdf
./lib/doc/clipart/with_fntright.pdf
./lib/doc/es/clipart/Resumen.pdf
./build/lib/examples/beamer-icsi-logo.pdf
./build/autotests/out-home/aastex6_defaultF.pdf
./build/autotests/out-home/de/beamer-article_defaultF.pdf
./build/autotests/out-home/de/MultilingualCaptions_defaultF.pdf
./build/autotests/out-home/de/serienbrief1_defaultF.pdf
./build/autotests/out-home/de/Math_defaultF.pdf
./build/autotests/out-home/de/serienbrief3_defaultF.pdf
./build/autotests/out-home/de/Braille_defaultF.pdf
./build/autotests/out-home/de/Tutorial_defaultF.pdf
./build/autotests/out-home/de/ItemizeBullets_defaultF.pdf
./build/autotests/out-home/de/Formelnummerierung_defaultF.pdf
./build/autotests/out-home/de/FeynmanDiagrams_defaultF.pdf
./build/autotests/out-home/de/Intro_defaultF.pdf
./build/autotests/out-home/de/Customization_defaultF.pdf
./build/autotests/out-home/de/Shortcuts_defaultF.pdf
./build/autotests/out-home/de/Additional_defaultF.pdf
./build/autotests/out-home/de/beispiel_gelyxt_defaultF.pdf
./build/autotests/out-home/de/DummyDocument1_defaultF.pdf
./build/autotests/out-home/de/beispiel_roh_defaultF.pdf
./build/autotests/out-home/de/DummyDocument2_defaultF.pdf
./build/autotests/out-home/de/EmbeddedObjects_defaultF.pdf
./build/autotests/out-home/de/Lebenslauf_defaultF.pdf
./build/autotests/out-home/de/linguistics_defaultF.pdf
./build/autotests/out-home/de/serienbrief2_defaultF.pdf
./build/autotests/out-home/de/splash_defaultF.pdf
./build/autotests/out-home/de/Dezimal_defaultF.pdf
./build/autotests/out-home/de/UserGuide_defaultF.pdf
./build/autotests/out-home/testcases_basic_defaultF.pdf
./build/autotests/out-home/mixing_inTitle_layouts_defaultF.pdf
./build/autotests/out-home/latex/arabic_simple_defaultF.pdf
./build/autotests/out-home/latex/utf8-plain-with-tex-fonts_defaultF.pdf
./build/autotests/out-home/beamer-article_defaultF.pdf
./build/autotests/out-home/achemso_defaultF.pdf
./build/autotests/out-home/serial_letter1_defaultF.pdf
./build/autotests/out-home/Formula-numbering_defaultF.pdf
./build/autotests/out-home/MultilingualCaptions_defaultF.pdf
./build/autotests/out-home/IEEEtran-CompSoc_defaultF.pdf
./build/autotests/out-home/architecture_defaultF.pdf
./build/autotests/out-home/sr/Braille_defaultF.pdf
./build/autotests/out-home/sr/splash_defaultF.pdf
./build/autotests/out-home/revtex4-1_defaultF.pdf
./build/autotests/out-home/obsolete/g-brief-en_defaultF.pdf
./build/autotests/out-home/obsolete/g-brief-de_de

CI job: Check author email in commit log

[Christian Ridderström]

Hi,

FYI, I've set up this CI job

https://ci.inria.fr/lyx/job/support/job/Check-author-email-in-commit-log/

that once a week checks the git log for commits where the author's
e-mail does not say it's from @lyx.org.

Rationale: Help us catch commits for which we wouldn't automatically be
notified via the the lyx-cvs-list.

The job's configured to e-mail lyx-devel in case it fails.

Caveat: I've not tested that the CI job triggers and notifies
properly. For that we need someone to do a commit with an incorrectly
configured author e-mail for that.

The CI job only checks the branch 'master'. I did a quick test for doing
multiple branches, which is why the list was spammed by warnings about
failed jobs. I've now set it back to just branch 'master'.
/Christian

Tip: Use shellcheck to do static analysis of sh and bash scripts

[Christian Ridderström]

Hi,

Just sharing a tip for when you write sh or bash scripts. Use a static
analysis tool like 'shellcheck' on the scripts.
https://www.shellcheck.net/

Readme at GitHub has more examples and details:
   https://github.com/koalaman/shellcheck

I would recommend that you run it on scripts you write.
Note: If our scripts didn't trigger issues (see below), we could create a
CI job to repeatedly run shellcheck and catch future issues directly.
/Christian


Below is an example of running it on one of our scripts.

$ shellcheck development/tools/count_lines_of_included_code.sh


*In development/tools/count_lines_of_included_code.sh line 5:*

l=`g++ $inc -DQT_NO_KEYWORDS -DQT_NO_STL -E 1.cpp | wc -l`

  ^-- SC2006: Use $(..) instead of legacy `..`.

   ^-- SC2086: Double quote to prevent globbing and word
splitting.



*In development/tools/count_lines_of_included_code.sh line 6:*

printf "%-40s: %d\n" $i $l

 ^-- SC2086: Double quote to prevent globbing
and word splitting.

^-- SC2086: Double quote to prevent
globbing and word splitting.

As you can see from above, the first issue is just a warning and not much
of a problem.
The second issue might be a problem depending on the actual file names. In
this case, I doubt we have source files with spaces or similar in their
names.

Below is a list of LyX scripts for which the shellcheck indicates at least
something:

HEAD @ chrid-mb $ find . -name "*.sh" -exec bash -c 'shellcheck "$1" >
/dev/null || echo "Issue found with: $1"; ' "" {} \;
Issue found with: ./3rdparty/boost/extract.sh
Issue found with: ./autogen.sh
Issue found with: ./development/attic/rename.sh
Issue found with: ./development/autotests/export-in.sh
Issue found with: ./development/autotests/run-tests.sh
Issue found with: ./development/autotests/single-test.sh
Issue found with: ./development/keystest/add_write_perms.sh
Issue found with: ./development/keystest/cache-bisect.sh
Issue found with: ./development/keystest/doNtimes.sh
Issue found with: ./development/keystest/killtest.sh
Issue found with: ./development/keystest/killtestpy.sh
Issue found with: ./development/keystest/list_all_children.sh
Issue found with: ./development/keystest/lyx_make.sh
Issue found with: ./development/keystest/main.sh
Issue found with: ./development/keystest/make_screen_shots.sh
Issue found with: ./development/keystest/maketar.sh
Issue found with: ./development/keystest/report.sh
Issue found with: ./development/keystest/report_html.sh
Issue found with: ./development/keystest/reproduce.sh
Issue found with: ./development/keystest/setup.sh
Issue found with: ./development/keystest/shared_functions.sh
Issue found with: ./development/keystest/shared_variables.sh
Issue found with: ./development/keystest/watch_keytest.sh
Issue found with: ./development/LyX-Mac-binary-release.sh
Issue found with: ./development/MacOSX/set_bundle_display_options.sh
Issue found with: ./development/tools/count_lines_of_included_code.sh
Issue found with: ./development/tools/count_total_lines_of_compiled_code.sh
Issue found with: ./development/tools/header_check.sh
Issue found with: ./development/tools/list_included_headers.sh
Issue found with: ./development/tools/separator-convert.sh
Issue found with: ./development/tools/update-po.sh
Issue found with: ./development/tools/updatelfuns.sh
Issue found with: ./development/tools/updatestats.sh

Re: 404 on manuals on lyx.org

[Christian Ridderström]

Pavel Sanda  writes:

> Yes, I will try to push our xhtml output to the web during next week or
> two so more ppl can check the output with their own eyes.
> I guess many of the bugs will rather easy-fix business.

Where/which URL(s) and with what kind of structure did you intend to
use?

Also, are you planning one using two "areas" on the web? I.e.
- An area (URL) for staging/testing/reviewing the pages
- An area (URL) for the published/deployed/reviewed pages?

I would recommend using two separate areas for robustness and
convenience. And further that we'd tell bots to not index the
staging/testing area.
/Christian

Re: 404 on manuals on lyx.org

[Christian Ridderström]

Uwe Stöhr <uwesto...@lyx.org> writes:

> El 11.08.2017 a las 08:35, Christian Ridderström escribió:
>
>> I can't do an anonymous login to ftp://ftp.lyx.de.

Uwe wrote:
> I'll try to repair it later today.

I just tested and I still cannot do anonymous login to ftp://ftp.lyx.de.
However, according to the message I get, I'm guessing anonymous isn't
allowed => so it's probably the way it should be.

>> I can   do an anonymous login to http://ftp.lyx.de.

Still works => good

>> I am _not_ able to see the subfolder /Documentation/, so it's gone, moved
>> or permissions changed.

And this now also seem to work => good

I also did a successful spot check and followed one of the links from
the wiki page 'Manuals' => good.

Thanks for fixing this Uwe.

Best regards,
Christian

FTP service on regular LyX server (Was: 404 on manuals on lyx.org)

[Christian Ridderström]

Uwe Stöhr  writes:

>> With our release frequency, it shouldn't be a problem to manually [*]
>> update PDFs and upload the official manuals.
>
> I only used ft.lyx.de because I could not upload files anymore to
> wiki.lyx.org. Is this now again possible?

Hi Uwe,

Yes, there's been a working ftp server on the regular LyX server for
some time. Don't remember for how long though.

> if so I would switch back to wiki.lyx.org.

That's kind of funny: I thought you intentionally were using a separate
FTP server to reduce the load on our regular server...?

> How do I upload there files?

Please try and follow the instructions here:

http://wiki.lyx.org/Site/AboutUploading

Also please don't hesitate to remark if the page is unclear -- or simply
fix it of course.

Regards,
Christian

PS.
I'll e-mail the FTP password separately.

Re: Changes to git branches, regarding 2.3.0 development

[Christian Ridderström]

Scott Kostyshak  writes:

> Please read the recent email I sent regarding branching 2.3.x [1]. I
> have branched and pushed the 2.3.x branch.
>
> 2.3.x will become 2.3.0, so if you are confident a commit belongs in
> 2.3.0, please go ahead and push to 2.3.x.

FYI, the CI jobs are testing branch 'master', so as it is we don't have
automatic testing of changes to branch '2.3.x'.

/Christian

> [1]
> https://www.mail-archive.com/search?l=mid=20170815074308.wq4xmkvmbeukhw23%40steph

Re: Require C++11 for 2.3?

[Christian Ridderström]

Stephan Witt  writes:

> Am 16.08.2017 um 10:47 schrieb Jean-Marc Lasgouttes :
>> Le 16 août 2017 02:51:44 GMT+02:00, c...@lyx.org a écrit :
>>
>>> Did this happen, i.e. requiring C++11?
>>> /Christian
>>
>> We require something close enough to c++11, the minimum being GCC
>> 4.6. For 2.4, we'll require a real c++11.

So no (I was really only checking [*]).

> What does this statement mean for Apple Mac clang compiler?
>
> I remember we had problems with thread_local on Mac…
>
> What does this mean for the minimum supported OS version?
>
> Perhaps one can provide a minimal hello-world example with
> all c++11 features we require for a real usable compiler
> and run-time, please?

Are you saying there might be a problem compiling LyX on an older
version of macOS?

This page
  https://trac.macports.org/wiki/XcodeVersionInfo
can be useful to map Xcode release to clang release.

According to this page
  http://clang.llvm.org/cxx_status.html
the Clang 3.3 release fully supports C++11.

If I read the pages correctly, as of Xcode 5.1 the Apple clang is
based on LLVM 3.4svn and should thus fully support C++11.
Xcode 5.1 is available for OS X 10.8, but seemingly not for OS X 10.7.

However, the above is (I think) only for the compiler. The C++ library
that's used also needs to fully support C++11.

Not sure how to check that easily. The following post (in the update) seem to
indicate that C++11 is supported as of macOS X 10.9. I could be reading
to much into the post though.
/Christian

[*] I'm trying to learn to use Gnus to work with the lists.
It's going ok but remembering keyboard shortcuts is a challenge,
and I keep forgetting that Emacs has menus that I could be using…

Re: Require C++11 for 2.3?

[Christian Ridderström]

Jean-Marc Lasgouttes  writes:

> Le 03/06/2016 à 07:24, Scott Kostyshak a écrit :
>> Dear all,
>>
>> Are we going to require C++11 starting with LyX 2.3.0? From what I
>> understand, this will make several things easier.
>
> Hello Scott,
>
> Yes, that's the plan.

Did this happen, i.e. requiring C++11?
/Christian

Re: News interface for the developers' list?

[Christian Ridderström]

On 13 August 2017 at 15:59, Christian Ridderström <c...@lyx.org> wrote:

> Hi,
>
> Some years ago lyx-devel could be accessed via a news-like interface
> through gmane. Gmane seems to have gone "belly up" since quite some
> time.
>
> Is there some other news interface?
>

Something seems to be working at nntp:news.gwene.org, and perhaps it's even
gmane.
I can e.g. see the group gmane.editors.lyx.devel with recent posts.
So perhaps gmane isn't completely dead, just it's web interface.

/Christian

News interface for the developers' list?

[Christian Ridderström]

Hi,

Some years ago lyx-devel could be accessed via a news-like interface
through gmane. Gmane seems to have gone "belly up" since quite some
time.

Is there some other news interface?
/Christian

Re: 404 on manuals on lyx.org

[Christian Ridderström]

On 11 August 2017 at 13:03, Pavel Sanda <sa...@lyx.org> wrote:

> Christian Ridderström wrote:
> > I agree HTML pages online would be nice.
> > Not sure if our manuals can (in a nice way) just be exported as HTML
> though.
>
> Last time I tried (around 2009 and "merged manuals" idea) it did not came
> out nicely.
> IIRC I reported the issues, Richard might have fixed them since then...
>
> > [*] It might make sense to have a CI job that builds PDFs etc from the
> > manuals as a separate test.
>
> The problem is you need to check them visually as well.
>

I agree that checking the generated PDFs is the difficult part.
However, in my mind the test's purpose is simply to ensure that LyX doesn't
fail when reading and exporting the manuals as PDFs.
This could catch introduced errors in HEAD, or if you let the TeX
installation update itself, breakage due to that.

> I guess once fixed we could just add one makefile rule to
> 1. create all major manuals into pdf & html from tarball
> 2. copy it to svn web tree, svn add new html figs
> 3. commit it
>
> It just needs someone to sit and spend couple hours on it...

I don't think the server autodeploys from SVN these days, but I could be
wrong.

Anyway, I didn't intend for such a CI job to also automatically publish the
manuals/PDFs.
IMHO, before publishing a new set of manuals some kind of manual review is
necessary.
/Christian

[*]
I have previously done some stuff that might be relevant/useful.
E.g. in one project we got a large log file from a build that might
somewhere contain a new error or warning message.
Or perhaps one of thousands of values in the log output had changed
although it shouldn't.
Filtering for 'errror' and 'warning' is of course one way to go.

Instead I introduced this approach:
1. Before one release, let two developers manually review the log file
2. Commit the reviewed log file as reference baseline (after fixing some
things)
3. After each new build, automatically diff the generated log to the
baseline.

The diff, if any, had to be manually reviewed and if the changes were
intentional
and as expected we converted the newly generated log into a new reference
for future builds.

This worked well IMHO and it did help us catch some subtle bugs.
However, the build situation was very different compared to that of LyX.

I also introduced this approach for other "outputs", like big automatically
generated data tables that were based on the source.
Before 'diff:ing' we sometimes had to convert to another format, but it
worked pretty well.

In theory the procedure above could work on PDFs (or perhaps PS?), or
possible PDFs converted to RTF or something else that's text based duh,
perhaps HTML :-)

It's a totally different question if the effort needed to setup the above
is worth it though, which is why I thought of a simpler test that fails if
LyX fails to export a PDF.

Re: 404 on manuals on lyx.org

[Christian Ridderström]

On 11 August 2017 at 08:35, Christian Ridderström <c...@lyx.org> wrote:

> On 8 August 2017 at 18:28, Tommaso Cucinotta <tomm...@lyx.org> wrote:
>
>> I just noticed these broken links
>>
>>   http://wiki.lyx.org/LyX/Manuals#download
>>   http://ftp.lyx.de/Documentation/en/Customization.lyx
>>   http://ftp.lyx.de/Documentation/en/Customization.pdf
>>
>> guess they're all broken.
>>
>
> I can't do an anonymous login to ftp://ftp.lyx.de.
> I can   do an anonymous login to http://ftp.lyx.de.
> I am _not_ able to see the subfolder /Documentation/, so it's gone, moved
> or permissions changed.
>
> IIRC, Uwe is related to this. CC:ing him.  (to his @lyx.org-account, which
> could be wrong)
>
> With our release frequency, it shouldn't be a problem to manually [*]
> update PDFs and upload the official manuals.
>
> Scott, are the manuals and their updating perhaps related to your notes on
> release procedure?
>

Scott wrote:
> I don't currently have any notes on this, but it's possible this was
> written somewhere and I missed it. Please let me know the exact steps
> that I should take if I should handle it, and I will add those to my
> notes.

I think I remember more now and I think Uwe does it.
I've now CCed his web.de-address thanks to Jean-Pierre.

/Christian


>
> As the problem seems to be the access, perhaps what's needed is a CI job
> that monitors that the FTP is working?
>
> There's also a seemingly outdated (from y 2009) merged manual available at:
>>
>>   http://wiki.lyx.org/LyX/Documentation#toc2
>>   https://wiki.lyx.org/uploads//LyX/Manuals/1.4.2/merged_manua
>> ls/merged_manuals-1.4.2.pdf
>>
>> However, besides fixing the links, considering the powerful conversion
>> capabilities of LyX, I'd propose to keep at least some manuals accessible
>> as HTML pages on the web, rather than, or in addition to, .lyx/.pdf
>> downloads.
>>
>
> I agree HTML pages online would be nice.
> Not sure if our manuals can (in a nice way) just be exported as HTML
> though.
>
> /Christian
>
> [*] It might make sense to have a CI job that builds PDFs etc from the
> manuals as a separate test.
>
>

What/how are converters triggered?

[Christian Ridderström]

Hi,

Do we have a description somewhere, perhaps in the source, as to what and
how the different converters are triggered?

How are files found for which converters are to be triggered?
1) File referenced by graphics insets
2) All files in certain locations
3) ?

How is the type of the file determined?
a) Only through the extension of the file
b) Through extension of file and if that's inconclusive by ...?
c) Using equivalent of 'file' command?
d) Directly looking at magic numbers in the beginning of the file
e) ?

Once a file has been converted, what decides if additional converters are
to be applied?

Do we have multiple converters defined for a single file type?
Could a user define multiple converters for a single file type?
If multiple converters could be applied, would only the "first" converter
be applied?
Or are perhaps converters applied until a conversion succeeds?

/Christian

Re: LyX manuals online?

[Christian Ridderström]

On 11 August 2017 at 00:41, Tommaso Cucinotta <tomm...@lyx.org> wrote:

> On 10/08/2017 22:27, Christian Ridderström wrote:
>
>> Do we have the LyX manuals online, either as PDF or web pages?
>>
>
> I just asked this as well, which I guess provides a partial reply to your
> Q.
>

Yes, apologies for not seeing your post before.
I've responded there instead.

Re: 404 on manuals on lyx.org

[Christian Ridderström]

On 8 August 2017 at 18:28, Tommaso Cucinotta  wrote:

> I just noticed these broken links
>
>   http://wiki.lyx.org/LyX/Manuals#download
>   http://ftp.lyx.de/Documentation/en/Customization.lyx
>   http://ftp.lyx.de/Documentation/en/Customization.pdf
>
> guess they're all broken.
>

I can't do an anonymous login to ftp://ftp.lyx.de.
I can   do an anonymous login to http://ftp.lyx.de.
I am _not_ able to see the subfolder /Documentation/, so it's gone, moved
or permissions changed.

IIRC, Uwe is related to this. CC:ing him.  (to his @lyx.org-account, which
could be wrong)

With our release frequency, it shouldn't be a problem to manually [*]
update PDFs and upload the official manuals.

Scott, are the manuals and their updating perhaps related to your notes on
release procedure?

As the problem seems to be the access, perhaps what's needed is a CI job
that monitors that the FTP is working?

There's also a seemingly outdated (from y 2009) merged manual available at:
>
>   http://wiki.lyx.org/LyX/Documentation#toc2
>   https://wiki.lyx.org/uploads//LyX/Manuals/1.4.2/merged_manua
> ls/merged_manuals-1.4.2.pdf
>
> However, besides fixing the links, considering the powerful conversion
> capabilities of LyX, I'd propose to keep at least some manuals accessible
> as HTML pages on the web, rather than, or in addition to, .lyx/.pdf
> downloads.
>

I agree HTML pages online would be nice.
Not sure if our manuals can (in a nice way) just be exported as HTML though.

/Christian

[*] It might make sense to have a CI job that builds PDFs etc from the
manuals as a separate test.

Re: Can postscript code be embedded in a LyX document?

[Christian Ridderström]

On 10 August 2017 at 11:02, Tommaso Cucinotta  wrote:

> On 09/08/2017 09:05, Guenter Milde wrote:
>
>> For EPS to PDF this means we could preferably use "repstopdf" instead of
>> "epstopdf".
>>
>>"repstopdf" is the version "whitelisted" in texmf.cnf for use with the
>>restricted shell access.
>>   On my Debian system, it is just a symlink to epstopdf:
>>
>
> Ubuntu 17.04:
>
> $ epstopdf -h | grep restricted
>   --restricted   use restricted mode(default: false)
>
> $ repstopdf -h | grep restricted
>   --restricted   use restricted mode(default: true)
>

on macOS Sierra it's as for you on Ubuntu.

I also noticed this in the help text:

  In restricted mode, options are limited to those with names and values
known to be safe; some options taking booleans, integers or fixed
names are allowed, those taking general strings are not.

/Christian


>
> Guess this is the standard. So, would we need the attached patch ?
>
> T.
>

Re: numbers/ids of posts on mail-archive.com vs. gmane.org?

[Christian Ridderström]

On 7 August 2017 at 04:53, Scott Kostyshak <skost...@lyx.org> wrote:

> On Sun, Aug 06, 2017 at 11:27:16PM +0200, Christian Ridderström wrote:
> > Hi,
> >
> > The wiki has "InterLinks", which is simply a convenient way to refer to
> > certain web pages.
> > For instance, this markup
> >
> > bug:10481
> >
> > is automatically converted into this link
> > http://www.lyx.org/trac/ticket/10481.
> >
> > Similarly, there's prefixes like LyxDevelPost and LyxUsersPost which were
> > used to refer to gmane.org. Unfortunately gmane.org doesn't seem to
> contain
> > the LyX lists anymore :-(
> >
> > I just now tested repointing the prefix "LyxDevelPost" from gmane.org to
> > mail-archive.com, so e.g.this currently works:   LyxDevelPost:200333
> >
> > There's a problem though... it seems the IDs of posts, i.e. the number,
> is
> > different for gmane.org versus mail-archive.com. My repointing would
> > therefore mean that a pre-existing link into gmane.org "evolved" from
> > initially working fine, to not working at all, to suddenly pointing to
> the
> > wrong post in mail-archive.com. *sigh*
> >
> > A lesson learned here is that perhaps we should avoid referring to posts
> on
> > e.g. the devel list.
>
> I suggest always referencing ML threads using the Message ID. For example,
> instead of referencing your email with:
>
> https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg201508.html
>
> I prefer to reference it as follows:
>
> https://www.mail-archive.com/search?l=mid=CANP4zpO6vK%
> 2BT8YQa7yMLXwMaw3fNcDUqO_%2BfcK80ypOJ0XB-Cw%40mail.gmail.com
>

Sounds pretty good to me.

Searching manually for the ID as per above caused me some problems
initially.
I can't just paste the message ID from your link as the %2B has replaced
'+' in the ID.

Note: In gmail I can see the message ID by looking at the original message,
which I do by clicking the little right downarrow next to the message and
selecting 'show original, e.g. giving me


canp4zpo6vk+t8yqa7ymlxwmaw3fncduqo_+fck80ypoj0xb...@mail.gmail.com

I've configured the wiki (edited the page Site/InterMap) to add the prefix
MsgId, which we can now use like this:

MsgId:CANP4zpO6vK%2BT8YQa7yMLXwMaw3fNcDUqO_%2BfcK80ypOJ0XB-Cw%
40mail.gmail.com

But note that %2B needs to be used to replace the '+', or it won't work.

The downside is that it's a bit longer than e.g.

 LyxDevelPost:200333

It can however be made shorter using e.g. markup like this, where
parenthesis, (), are used to hide parts of the link:

[[MsgId:CANP4zp(O6vK%2BT8YQa7yMLXwMaw3fNcDUqO_%2BfcK80ypOJ0XB-Cw%
40mail.gmail.com)]]

I just hope that the 'CANP4zp' isn't always the same...

/Christian



> Although it is longer, it contains the message ID, so if mail-archive
> removes the post, we can always convert the above link to any other mail
> archiver. All of them support referencing with message ID, if I recall
> correctly.
>
> When I'm reading email in mutt, all I do is press a shortcut (I use [ma for
> "mail archive"), and it copies the above link to my clipboard so I can
> paste
> it in my email (or on trac). If anyone is using mutt, let met know if you
> want the script.
>
> Scott
>

LyX manuals online?

[Christian Ridderström]

Hi,

Do we have the LyX manuals online, either as PDF or web pages?

If not, how come?
/Christian

numbers/ids of posts on mail-archive.com vs. gmane.org?

[Christian Ridderström]

Hi,

The wiki has "InterLinks", which is simply a convenient way to refer to
certain web pages.
For instance, this markup

bug:10481

is automatically converted into this link
http://www.lyx.org/trac/ticket/10481.

Similarly, there's prefixes like LyxDevelPost and LyxUsersPost which were
used to refer to gmane.org. Unfortunately gmane.org doesn't seem to contain
the LyX lists anymore :-(

I just now tested repointing the prefix "LyxDevelPost" from gmane.org to
mail-archive.com, so e.g.this currently works:   LyxDevelPost:200333

There's a problem though... it seems the IDs of posts, i.e. the number, is
different for gmane.org versus mail-archive.com. My repointing would
therefore mean that a pre-existing link into gmane.org "evolved" from
initially working fine, to not working at all, to suddenly pointing to the
wrong post in mail-archive.com. *sigh*

A lesson learned here is that perhaps we should avoid referring to posts on
e.g. the devel list.

But the question is what to do now.  Perhaps use a different, and unused,
prefix for devel-posts on mail-archive.com?

Thoughts,
Christian

Can postscript code be embedded in a LyX document?

[Christian Ridderström]

Hi,

Q1: Can postscript (PS) code be embedded in a LyX document in such a way
such that it's parsed when doing a preview, or exporting a document?

Q2: Can PS code only be included by embedding a graphics inset referencing
e.g. a .ps-file?

Q3: Would the PS code, in e.g. an external file, be parsed as part of
previewing, or only when exporting?

And finally:

Q4: Is PS code able to do system calls when called/parsed in some indirect
manner by LyX?

Depending on the result something should perhaps be added to the wiki page.
/Christian

Note: IIRC PS is Turing complete.

Re: Ad a section in the wiki page about LaTeX safety

[Christian Ridderström]

On 2 August 2017 at 09:30, Jean-Pierre Chrétien  wrote:

> Hello,
>
> Following Christian's suggestion, I've added a section about the subject
>
> http://wiki.lyx.org/Devel/SafetyAndSecurity#toc2
>
> Is it useful/appropriate?
>

Thanks for adding it. I read it and it does improve my understanding.
While trying to understand it better I did some editing and added a few
questions, I hope you can have a look.

FYI, I used markup like
   {+CHR-3+}
in the running text to insert a marker. And then at the end I added e.g.

- {+CHR-3+}: Bla bla bla

to describe my question or whatever.

/Christian

Re: Ad a section in the wiki page about LaTeX safety

[Christian Ridderström]

On 2 August 2017 at 09:38, Jean-Pierre Chrétien  wrote:

> Le 02/08/2017 à 09:30, Jean-Pierre Chrétien a écrit :
>
>> Hello,
>>
>> Following Christian's suggestion, I've added a section about the subject
>>
>> http://wiki.lyx.org/Devel/SafetyAndSecurity#toc2
>>
>
> I've been unable to approve the last link, none of the passwords would
> work.


I approved the link, and have e-mailed you the password.
/C

How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

On 2 August 2017 at 10:03, Scott Kostyshak  wrote:

>
> > 
> > I'm using gmail's web interface these days. This might be why I'm finding
> > it difficult to efficiently follow threads that are so long.
> > - The gmail labs thing I used for replying to parts of an e-mail is no
> > longer working.
> > - Sometimes the replies don't go to the list. *sigh*
> > - I haven't figured out how to mark a single e-mail as unread,
> >   e.g. when I feel I need to reply to it later, and instead have to mark
> > the entire thread.
> >   This does _not_ work well for long threads.
> > 
>
> I find mutt to be very helpful with emails, but it takes a long time to
> get used
> to, and I'm not sure it would solve the particular issues you reference.
>

A decade or so ago I used 'pine' which worked really well for my e-mails
and the LyX lists.
Back then I accessed multiple lists as news groups, but as 'pine' tracks
the posts you've read
through a local file, I had to run 'pine' on a single computer. So I then
ran 'pine' within a screen session on server to which I SSH:d. I needed
this because I worked from multiple computers running on different
platforms.

This worked well, and the drawback was mainly when people posted in HTML,
or included URLs which I manually had to copy and paste.

I stopped using pine, probably triggered by losing access to the server.
There were also issues related to reading regular e-mail when people
started inserting lots of HTML, images and links. I did try switching to
'alpine', and probably considered using 'mutt', but there was still the
issue of needing to run it from a single computer.

Back then I think I could switch topic by changing the subject the topic,
but I'm not sure it works so well when using gmail's web interface.

That I stopped using 'pine' and switched to gmail also resulted in my not
being able to follow the LyX lists, and this was probably the main reason I
became mainly inactive within LyX for a while.
I'm now using my c...@lyx.org account for LyX related issues, and that is
working better -- except that gmail isn't working well at all for me when
there are long threads.

I've changed the subject of this reply and I'll see how it appears in my
gmail. I.e., will it be obvious it's a new topic, or will it be joined with
the original topic.

/Christian

Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

On 2 August 2017 at 11:29, Pavel Sanda  wrote:

> Scott Kostyshak wrote:
> > Ah I did not realize you did that on purpose. I actually found it
> annoying when
> > I wanted to go up the discussion and couldn't because it was cut off
>
> +1
>

I just want to check you understood that I primarily mean a new thread when
the discussion is diverging in topic compared to the original post, i.e. to
avoid "threadjacking". For instance, would you've preferred this post to
have been in Scott's original thread?

If that's what the majority prefers I can change to that.
/C



>

Safety/security: I've received some good advice

[Christian Ridderström]

Hi,

I've consulted about LyX with someone who's an IT security professional.
I'll later post that stuff separately to developers list, and/or add it to
the wiki page.
This post is just to make sure you guys can remind if I manage to forget as
I'm tired and won't do it tonight.
/Christian

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 1 August 2017 at 21:24, Richard Heck  wrote:

> On 08/01/2017 04:54 AM, Scott Kostyshak wrote:
> > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> >> Pavel Sanda wrote:
> >>> I did not hear your reaction to it either.
> >> I see you just did that, sorry... P
> > I believe that except for Enrico and I, everyone who participated in
> > this conversation has voted.


Hi Scott,

For the record, I'm not sure why I'm not included in people who particpated
but haven't voted.
Anyway, you're the release manager so your decisions goes.


> Uwe was interested in voting, but as the
> > vote stands currently, his vote would not change the outcome. I don't
> > have indication that any other LyX developer is interested in voting.
> >
> > The results are the following:
> >
> > 1: Guillaume (hesitantly [1]), .5 Pavel
> > 2: .5 Pavel
> > 3: Kornel, JMarc, Jürgen, Richard
> >
> > Option 3 wins the vote. My decision is to go forward with Enrico's
> > latest patch for beta1.
> >
> > Because the vote was not a blowout, and because I did not represent the
> > opinions of at least Guillaume and Pavel in the options, we will have
> > another vote two weeks after beta1 is released. Each developer may
> > propose one option that will be included in the vote.
>

As a side note, my opinion was likely also not represented, but I think you
have a feeling for my general thoughts.



> > So to be clear: although we are going to release beta1 with minted
> > support, depending on the post-beta1 vote, we may decide to remove it.
>
> It's obviously up to you, Scott, as release manager, but I agree with
> Enrico, to some extent, that this isn't the best policy.



> We have spent an enormous amount of time on this and finally have come to
> some kind of resolution. It really would be best to close that part of the
> debate and spend our time figuring out how to make the needauth
> and shell-escape stuff as secure as we can make it, given the present
> framework. Time spent re-litigating the questions about the framework is
> going to be wasted time.
>

Hi Richard,

It sounds like you complain about the time spent on discussing what I
assume is not just the minted patch but safety/security in general.
However, regarding spending to much time discussing safety, I can't help
but think about LyX 2.2 where something was obviously missing in terms of
safety.
That leads me to conclude that something was missing, or went awry, in the
development process of LyX 2.2. This, combined with you advocating

 "... spend our time figuring how to make the needauth and shell-escape
stuff as secure
  as we can make it, given the present framework"

makes me a little concerned.

It sounds like you argue for a release of 2.3 regardless of the absolute
achieved level of safety, because:
a) we've anyway done all that can be done using the needauth framework.
b) it's anyway better than LyX 2.2.

Would you mind clarifying your point of view?

Is it perhaps that in your mind the needauth solution is already good
enough, or that you're confident it can be made good enough?
If so, could you perhaps expand on why?

Or do you perhaps consider safety/security less important than other
aspects?
Perhaps compare to releasing before some date, or ensuring ease-of-use for
advanced users or perhaps something else?

Best regards,
/Christian

We can always decide to remove things, especially before a major release.
> If there are serious
> problems or concerns that arise during testing, then those can be
> raised at that point. Or if one
> or more of us who preferred the present course has doubts, we can
> raise those. But simply
> having another vote (a rare enough occurrence around here) just
> because people weren't
> entirely happy with how the options were presented doesn't strike me as a
> good idea. There
> is no indication I can see that the people who voted (3) might, let alone
> would, have voted
> differently had the other options been differently described. I think
> we all understood well
> enough what the broad issue was.
>

Discussing minted/safety/etc (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

Richard wrote:

> We have spent an enormous amount of time on this ...
>

Hi,

Some thoughts regarding the discussion of safety and design of security
measures, i.e. a kind of "lessons learned" regarding the discussion aspects.

I think one thing that made things slower and more inefficient was a lack
of any self-contained description[*] of the overall design regarding safety
and security. Ideally I think the discussion would've benefited from having
one for:
- LyX version 2.2, i.e. the release with the big problems
- What was in master/HEAD, and at least was on its way to become LyX 2.3
- Intended eventual design, for LyX 2.3 or possibly a later release.

The long threads likely by now do contain a lot of separate pieces of
description, but it's unfortunately not so easy (at least for me) to find
this information among all the posts.
And sometimes I'd probably not understand to which "release" the
information pertains.

A partial "fix" might be to create a "best of"-list of posts with useful
descriptive information. Or perhaps to copy them to into one place, adding
minor markup as to what's still valid.
Or actually try to write down some design design description.

Question: Is there anyone who feels they know/understand the big picture
regarding safety/security?
Because I guess we in theory instead of writing things down could have a
designated "guru".

Actually, I'm a little worried that no one really has the big picture of
the intended design regarding safety and security.

Also, that different people have different ideas of where we are going and
what's considered needed, while we at the same time are not aware of these
differences.

For something (a "feature") much less complicated than safety/security, I
think it'd be fine for LyX if only a very small number of people know that
part, and to where they intend to go. And that everything else is in the
code. But for a complicated topic like safety, and when asking people to
vote, I think the lack of a description is a problem.

Hmm, and perhaps in case of votes, Scott should (be allowed to) weigh them
differently depending on the persons knowledge. E.g., since I don't
know/understand the (intended) safety design, why should my vote count as
much as a potential safety "guru".

On a more practical level:
Hindsight is 20-20, but regarding e.g. Enrico's patch, perhaps we should
have created a branch representing that alternative?   This would have
allowed that branch to always represent the latest improvement, thus
avoiding people testing out an older patch accidentally.
So we should keep the possibility of using branches in mind for future
discussions.

Perhaps there's other more practical things we could've done to make life
easier.

Best regards,
/Christian

[*] Regarding SW design descriptions and actually being able to understand
and review an intended design in advance, I'm probably a bit "damaged" from
when I worked with satellite software (AOCS), e.g. as SW verification and
validation manager.

In that field, lots of documentation was required which often was annoying,
but for complicated stuff like FDIR (making the SW/satellite be able to
cope with equipment failures), it really was essential to have the overall
picture. The reason being that a tiny change in one area of the system
could easily cause big problems in other areas, i.e. it's about unintended
consequences.

Here I see several similarities between FDIR and the safety/security for
LyX, e.g. that adding a neat feature like previews of Gnuplot figures
could've led to a big security hole.

Another similarity is that you can't do FDIR only at a low level, you need
a system perspective.
I think it's the same with safety/security for LyX: If we only consider
each feature separately, we're going to screw up. Two features in isolation
might be safe, but when available together it might "leak".

Another similarity is needing to define our objectives, and what we
consider "good enough" safety. If we don't understand what we're trying to
achieve, it's e.g. not possible to review a design to see if it achieves
the objectives.

Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

Richard wrote:

> We have spent an enormous amount of time on this ...
>

HI,

Regarding the discussion of LyX's safety I'd like to make a few remarks
related to ... ?list etiquette? Not sure what the correct term should be,
but it ought to be clear below.

Really long threads:
Are we really ok with threads containing upwards a hundred posts?
Perhaps at some point it's necessary to simply start the thread over?

Thread splitting:
I've tried to break off topics into separate threads, but it _seems_ like
it's mainly me doing that.
Or maybe it's how the gmail interface presents the thread to me?

Is it something we're not doing anymore.
Should thread splitting be avoided, or should we try to do it more?


I'm using gmail's web interface these days. This might be why I'm finding
it difficult to efficiently follow threads that are so long.
- The gmail labs thing I used for replying to parts of an e-mail is no
longer working.
- Sometimes the replies don't go to the list. *sigh*
- I haven't figured out how to mark a single e-mail as unread,
  e.g. when I feel I need to reply to it later, and instead have to mark
the entire thread.
  This does _not_ work well for long threads.


Are there other things we could've done to do the discussion more
efficiently?

Using a wiki page for security topics didn't seem like it's (so far) helped
anything.
Would a LyX document in git have been better?
Or a plain text file?
/Christian

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 1 August 2017 at 01:25, Pavel Sanda <sa...@lyx.org> wrote:

> Christian Ridderström wrote:
> > Please note that I'm _not_ wholly against something like needauth, I'm
> simply
> > not convinced it's good enough.  In fact, I'm still unclear on exactly
> how it
> > currently works, or perhaps rather, how it's intended to work in LyX 2.3.
>
> I already wrote you possible ways how we can cement the current situation
> more
> and IIRC you did not picked any.


Would you mind pointing me to that post?
I don't know if I simply missed it or was unable to read/consider/reply at
the time.

Meanwhile Tommaso committed patch which is
> slightly improving situation by "scary dialog". I did not hear your
> reaction to
> it either.
>

As you noticed in a later post, I did react to that.


> > > Even after all discussion I still see adding the whole needauth
> machinery
> > > as unnecessary complication of code and UI; possible future use of
> pygments
> > > still seems as made up argument for the sake of discussion rather than
> real
> > > user demand.
> >
> > Would you mind clarifying why needauth is an "unneccessary complication
> of
> > code and UI"?  (Apologies as I'm likely asking you to repeat what you've
> said
> > previously).
>
> I meant extending needauth mechanisms beyond current gnuplot/knitr usage
> for the sake of minted, because I believe minted maintainer will deliver
> us fix within couple months.
>

Ok, I understand what you mean.


> With such sight in view, I would be just fine to let minted support in
> current fashion, tell people clearly in manual that adding --shell-escape
> is dangerous, let Uwe scream at us little bit and then be done with it.
>

Ok.

Guess it comes down to the number of expected minted users. If not very
high, it's a good enough solution.
If there'd be lots of minted users, risk increase that they accidentally
leave --shell-escape in place.
/Christian

Re: Regarding safety/security, the DOCX and DOCM formats of MS Office

[Christian Ridderström]

On 1 August 2017 at 10:54, Scott Kostyshak <skost...@lyx.org> wrote:

> On Tue, Aug 01, 2017 at 02:26:52AM +0200, Christian Ridderström wrote:
>
> > Anyway, I'm not really advocating a new file extension like '.lyxm', but
> > simply trying to illustrate that MS Office tries hard to keep the
> defaults
> > safe, to the extent of not permitting macros at all.
>
> Interesting idea. I didn't know they did that.
>

FYI, I've added what I wrote to the wiki safety page, as a reference in the
future.

/C

Re: r41086 - www-user/trunk/farm/cookbook/LyX

[Christian Ridderström]

On 1 August 2017 at 10:54, Scott Kostyshak  wrote:

> On Mon, Jul 31, 2017 at 09:45:55PM +0200, sa...@lyx.org wrote:
> > Author: sanda
> > Date: Mon Jul 31 21:45:55 2017
> > New Revision: 41086
> > URL: http://www.lyx.org/trac/changeset/41086
> >
> > Log:
> > Update master translation status for web.
> >
> > 2Scott:
> > $ cd ~/lyx/master/po
> > $ make i18n.inc
> > $ mv i18n.inc ~/lyx/www/farm/cookbook/LyX/i18n_trunk.inc
> > $ cd ~/lyx/www
> > $ svn ci
>
> Thanks,
>

The above seems like it'd be useful to document somewhere.

But...

Pavel, did you run the commands on your local machine?

If so, we probably need to do something on the wiki/www server to do an
 'svn update' in some folder?

/Christian


>
> Scott
>

Re: lyx executable built with cmake will not run

[Christian Ridderström]

On 1 August 2017 at 14:47, Jean-Pierre Chrétien  wrote:

>
> It does not work, I can confirm.
>
> Here is what I ran:
>
> $ git clone g...@git.lyx.org:lyx newmaster
> $ cd newmaster
> $ automake
> $ cd ../cbuildnm
> $ cmake -DLYX_ENABLE_EXPORT_TESTS=ON -DLYX_RELEASE=ON
> -DLYX_PROGRAM_SUFFIX=ON -DLYX_ENABLE_CXX11=AUTO -DLYX_USE_QT=QT5
> -DLYX_LOCALVERSIONING=ON ../newmaster
> $ make
>

Why are you running 'automake' above, is that really needed when using
cmake?
I thought I didn't need it on my mac at least...
The commands also don't clean any contents in ../cbuildnm, should that
perhaps be done?

/Christian

What kinds of "code" can be embedded in a LyX document and "run" from LyX?

[Christian Ridderström]

Hi,

For the purpose of discussions on safety/security and
needauth/shell-escape, I would like to have, and document, a more complete
picture of the different kinds of use scenarios where LyX causes code to be
executed that was either embedded in a LyX document, or or in some external
file (script) that's referenced from a LyX document.

I've added the text below as a new section ("Execution of code form LyX")
to the wiki page on safety/security, but it contains some questions:
http://wiki.lyx.org/Devel/SafetyAndSecurity#toc4

I also wonder if it's complete, i.e. if there are additional ways in which
LyX might cause code to be executed.

---
!!! Execution of code from LyX

This section is intended to provide details of the different ways in which
LyX can cause (potentially dangerous) code to be executed.
The code in question could be stored directly in the LyX document, i.e. in
the .lyx-file, or stored in an external file, e.g. a script, that's
referenced by the LyX document.

TBC: The LyX program itself cannot directly execute any code, potentially
dangerous or otherwise. The reason is that he LyX program does not contain
any interpreter or compiler. Instad, LyX must always invoke (call) a tool
external to LyX in order for any code to actually be executed.

So In what ways can a LyX document contain code, or reference eg. an
external script, that LyX can cause to be executed?

Here's an initial attempt at listing the cases:

 Document preamble
The document preamble can contain arbitrary LaTeX code.
* Code is stored in the .lyx-file
* Execution occurs only when LyX exports the document as e.g. PDF
* LyX invokes e.g. pdflatex to execute the LaTeX code
* If option 'shell-escpae' is provided to e.g. pdflatex, execution of the
LaTeX code can be dangerous
* Q: Is it dangerous if and only if the option shell-escape is passed to
e.g. pdflatex?
* Q: Are there differences regarding safety/security for pdflatex vs luatex
vs ...?

 ERT
The document can contain ERTs which contain LaTeX code.
* Code is stored in the .lyx-file
* Execution occurs as for the preamble.
* Q: Can the LaTeX code in an ERT be arbitrary (from the point of view of
safety), assuming shell-escape is active?

 Chunk insets
The document can contain "chunk insets" that can contain code, e.g. R-code
* Code is stored in the .lyx-file
* Execution occurs only when LyX exports the document as e.g. PDF
* LyX invokes an external tool to execute the code.
* Converter settings define which tool LyX will use and with what arguments
the tool is called.
* Q: What other languages than R are relevant for "chunk insets"?

 Graphics insets

The document can contain graphics insets that reference a gnuplot script
* Code is stored in an external file (script)
* Execution occurs at preview or when  related to exporting a document.
* Converter settings define which external tool that'll be used to execute
the code ???

Besides gnuplot, what about e.g. 'Graphviz'?

 Anything else?

Q: Is there anything else that can contain code or reference code?

---

Some additional questions:

What about using the 'minted' package?   How does that fit in with the
above.
It's not code manually added by the user, but code generated from some LyX
inset.
Or is it that we're sending a code listing to minted that sends it to
pygmentize, which might "execute" the code listing?


Is pure literate programming covered by the above?

Best regards,
Christian

Regarding safety/security, the DOCX and DOCM formats of MS Office

[Christian Ridderström]

Hi,

Regarding safe behaviour or not, I happened to remember an aspect related
to MS Word and macros.

Some years ago Microsoft changed the old DOC format and as default
introduced the DOCX format. One change was of course the use of XML, but
another is that DOCX does _not_ allow macros --- at all.

Besides the confusion, it does have the advantage that typical users aren't
exposed to macros.

A user that needs [*] macros will have to save in a different format, DOCM,
and where macros are available for use.

It's been a while so I might be misremembering or confusing with other
security measured, but IIRC there were also warnings popping up when trying
to open a .docm-file from an "untrusted" source, like when a document was
received over e-mail, downloaded or opened from a network drive.
This is more a UI thing though... IIRC, e.g. Internet Explorer adds some
flag when a file is downloaded and this is what marks it as being from an
untrusted source.

Anyway, I'm not really advocating a new file extension like '.lyxm', but
simply trying to illustrate that MS Office tries hard to keep the defaults
safe, to the extent of not permitting macros at all.

/Christian

[*]
I've actually had to use DOCM and macros in a Word. In my case the macros
were used to filter out certain kinds of information from the word document
to a text file that was then imported into another system.

PS.
As an aside, I once had a very confusing experience with an Excel file from
a customer with important data where merely the act of opening the Excel
file changed the file on disk. I repeat: This happened without me saving
the file.

What happened was that the Excel file contained some OLE object which
executed when opening the file and this resulted in the file on disk
changing.  I noticed this because for safety we had separately been sent a
formal document which stated the expected hash of the excel file with data,
which verified manually when receiving the file. And then, being paranoid,
I added an automated test that verified the hash before importing the data
from the Excel file, only to find that the test failed due to the hash no
longer matched! *sigh*

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

Hi Scott,

On 31 July 2017 at 16:50, Pavel Sanda  wrote:

> Scott Kostyshak wrote:
> > I'm concerned that since this issue has left us all exhausted, there is
> > a feeling of "let's just get this over so we can move on". I encourage
> > all of us to give one more cognitive spurt and give a vote.
> >
> > From what I understand, the three options are still what I proposed
> > three weeks ago [1]:
> >
> > 1. Revert the recently added minted support.
> >
> > 2. Keep the current state of master.
> >
> > 3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
> > to lib/images. (Note that I get linker errors when I try to apply the
> > latest patch, but it might be something specific to my setup.)
> >
> > So I ask explicitly to everyone (even if you think you have already
> > voted, please give your vote again):
>

I'm just letting you know I haven't had time to look through all the recent
posts yet.

Further, and more importantly for me, the above is only for minted.
To me it doesn't make sense to not consider 'needauth' in the same context
of safety/security.

So I'm not sure I can vote as I'm not sure the alternatives make sense.

The following is likely not going to be popular, but ...

If we don't have mechanisms that keep "regular" users safe, I don't think
it matters
very much what we do with minted.

So I think a very important question is if we think needauth is
sufficiently good,
or can be made sufficiently good before release.

I do think there should, at least formally, be some kind of alternative(s)
corresponding to e.g.
removing/disabling needauth _and_ removing/disabling the automatic/silent
running of R-scripts/gnuplot,
and that it should take some "effort"/"be scary" to enable them.

IMHO it's for LyX 2.3 not ok to do a release that's unsafe just because
it's "less" unsafe
than before, i.e. the release has to be sufficiently good, not just less
bad than before.

Please note that I'm _not_ wholly against something like needauth, I'm
simply not convinced it's good enough.
In fact, I'm still unclear on exactly how it currently works, or perhaps
rather, how it's intended to work in LyX 2.3.


> Little difficult, because what I opined was not included in your list.
> To sum up I favor support of minted, which would use secure calling
> minted once it's compilation is split into separated steps as proposed
> on minted bugzilla.
> So I do not have strong opinion whether we shoould go 1 or 2 if we
> fix the issues once minted is fixed.
>
> Even after all discussion I still see adding the whole needauth machinery
> as
> unnecessary complication of code and UI; possible future use of pygments
> still seems as made up argument for the sake of discussion rather than real
> user demand.
>

Would you mind clarifying why needauth is an "unneccessary complication of
code and UI"?
(Apologies as I'm likely asking you to repeat what you've said previously).

Best regards,
/Christian


>
> So the breakdown is likely 0.5 voting points for 1 & 2.
>
> Pavel
>

Re: Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)

[Christian Ridderström]

On 31 July 2017 at 20:44, Guillaume MM  wrote:

> Le 31/07/2017 à 13:31, Jürgen Spitzmüller a écrit :
>
>> I meant it in this sense. If a vote only means "I did not have a
>> look at
>>
>> the patch but I am fed-up so let us go ahead" then it is not taking
>> responsibilities.
>>
>>
>> A vote is a vote. If the given voting will be Rates differently, this
>> will be have been the last voting I have participated on this list.
>>
>
I guess he wrote that from a iPhone, which I say because I (unwillingly)
have an iPhone and my girlfriend has remarked on the reduced quality of my
language when I use the phone.


> I am not sure I got your last sentence right :) but nobody proposed to
> discard any vote.
>

Do we have an overview somewhere (with patch reference) for the
alternatives proposed for beta1, which is then what's likely to end up in
2.3?
Note: I did just look at the wiki page but didn't see it there clearly.
/Christian

Re: [LyX/master] prefs/needauth: added warning if user tries to disable authorization for needauth converters.

[Christian Ridderström]

On 27 July 2017 at 00:06, Tommaso Cucinotta  wrote:

> commit 8a4fcd3d95ca4aeed1c46152cecadf29ed21e774
> +   _("SECURITY WARNING!"), _("Unchecking this option has the
> effect that potentially harmful converters would be run without asking your
> permission first. This is UNSAFE and NOT recommended, unless you know what
> you are doing. Are you sure you would like to proceed ? The recommended and
> safe answer is NO!"),
>

The warning as per the patch:

Unchecking this option has the effect that potentially harmful converters
would be run without asking your permission first. This is UNSAFE and NOT
recommended, unless you know what you are doing. Are you sure you would
like to proceed ? The recommended and safe answer is NO!

There's an extra space before the question mark (?). However, I would
suggest removing "unless you know what you are doing" from the message to
make it stronger, i.e.


Unchecking this option has the effect that potentially harmful converters
would be run without asking your permission first. This is UNSAFE and NOT
recommended. Are you sure you would like to proceed? The recommended and
safe answer is NO!


And actually, I wouldn't mind making the warning even stronger:


Unchecking this option is UNSAFE and NOT RECOMMENDED. With this option
unchecked LyX will NOT EVEN ASK you for permission before executing
converters that might be potentially harmful to your computer system.
Please read  for more information.
Are you sure you would like to proceed? The recommended and safe answer is
NO!


The above would of course mean that we also need to add a section to the
user guide or something.
Hmm... could we suggest asking the users' list for advice?

/Christian

Re: [patch] update PDF viewers in configure.py

[Christian Ridderström]

On 31 July 2017 at 19:52, Pavel Sanda  wrote:

> Kornel Benko wrote:
> > > It is not a demo. Only if you want to have all features like e.g.
> > > splitting PDF pages one has to buy another version. So it is the same
> as
> > > with Acrobat Reader, for the full feature set of Acrobat one has to
> pay.
>

I skimmed their manual and read the license section which says:

3. NON-COMMERCIAL USE.
You are hereby granted to use this Software for non-commercial purposes
without charge for
unlimited time on Desktop Linux.
Software can also be used for viewing and printing documents on Windows and
Mac OS X for
unlimited time without any charge.

If this is accurate, it might be useful for e.g. academic users of LyX.

Regarding licenses... the program is based on Qt, and they ship several Qt
libraries -- I just verified this after installing it on my mac.
However, the program didn't seem to acknowledge Qt or reference LGPL or
GPL, and they e.g. explicitly forbid reverse engineering so unless they've
got a commercial Qt license they're likely violating Qt's LGPL. :-(

Now, they might have acquired commercial Qt licenses, one per developer, in
which case everything could be fine.
However, as the Qt licenses aren't cheap, e.g. perhaps $1400/month for
three developers, I see a risk that they don't have a commercial license.

If we care about this, we could simply ask the company behind the program
about their licensing situation.


> > Do we really try to promote commercial programs?
>
> There is free version, with slightly less features, but otherwise if we
> support acrobat reader which is not open source either why we could
> not support another closed source but free app?
>

Market dominance is probably the reason to support Acrobat.
But it's a good point.

/Christian

Re: [patch] update PDF viewers in configure.py

[Christian Ridderström]

On 30 July 2017 at 23:47, Jean-Marc Lasgouttes  wrote:

> >Under Linux I was looking for a PDF program with which I can properly
> >fill out and submit PDF forms. I found the Program Master PDF editor
> >and
> >would therefore like to support it in LyX.
>
> To be frank I never heard of it before today. It seems to be some
> commercial program which only can be found as demo. I am not very fond of
> promoting such things.
>

+1

The reason for my reluctance is probably:
- Any risk of licensing-related issues, i.e. is Master PDF honouring SW
licenses correctly.
  E.g. I'd feel bad supporting and being associated with a SW that's
violating GPL.
- Does the program serve commercial ads and could there be anything "risky"
for the user related to that.
- Might the program be uploading "telemetry" now or in the future?
- ?

The last two aspects relates to the company's business model now, as well
as in the future.

Note that I don't know anything about Master PDF, they might be above board
(today) when it comes to all of these aspects.
Perhaps we should simply ask them (Master PDF) these questions?

Best regards,
/Christian

Re: allowing anonymous contributions to LyX's source code?

[Christian Ridderström]

On 28 June 2017 at 04:10, Joel Kulesza  wrote:

> On Tue, Jun 27, 2017 at 7:24 PM, Richard Heck  wrote:
>
>>
>> It's come up more than once, so I think it's worth writing down what
>> we've decided. Obviously, we can revisit the issue any time we like. But
>> we won't have to re-discuss it every time.
>
>
> This is why I'd like to see the approach written down as well.
>

I also thinks it makes sense to write it down as a _guideline_.

We could use a wiki page as an alternative/complement to Development.lyx.
It'd make it more open "to the world" how we deal with "unnamed
contributors" [*] that don't wish their real name published in CREDITS.
/Christian

[*] I recently read a blog or something about unnamed sources versus
anonymous sources,
which made me think it's clearer for us to refer to "venom00" as an unnamed
contributor
rather than an anonymous contributor.

Re: #10735: needauth - ask again authorization if file (or script snippet) changes

[Christian Ridderström]

Hoping/testing if a simple reply works and is added to the ticket.

I think this is an excellent improvement.
However, I'd like to suggest that instad of storing a timestamp, we should
store a hash of the document's contents.

I'm not sure if it's necessary to include the file name in what's hashed.
For instance, let's say we approve running (the content of) doc1.lyx.
Would we automatically be ok with running (the content of) an identical
document named doc_with_a_different_name.lyx?

Perhaps it'd be nice if the hash could be done only on the "code" parts.
OTOH, perhaps we can't be sure that the code parts don't depend on the
content of the non-code parts.  Similarly, in theory the code parts could
perhaps depend on the name of the document, or it's absolute path.

/C

On 29 July 2017 at 02:03, LyX Ticket Tracker  wrote:

> #10735: needauth - ask again authorization if file (or script snippet)
> changes
> -+--
>  Reporter:  t.cucinotta  |  Owner:  rgheck
>  Type:  enhancement  | Status:  new
>  Priority:  normal   |  Milestone:
> Component:  converters   |Version:  2.3.0dev
>  Severity:  normal   |   Keywords:
> -+--
>  Related to #10481.
>
>  If I authorize conversion on a file, its absolute filepath is remembered
>  by LyX, in a section of $HOME/.lyx/session. If the document is updated
>  later, e.g., saved from an e-mail, updated from git/svn/cvs, then said
>  authorization should be automatically revoked.
>  Namely, we need to store in the session also the timestamp of the last
>  authorized file, so that, when it changes, we ask authorization again.
>
> --
> Ticket URL: 
> The LyX Project 
> LyX -- The Document Processor
>

Re: Living with shell-escape: Using two LyX instances - critique invited

[Christian Ridderström]

On 27 July 2017 at 18:05, Tommaso Cucinotta  wrote:

> On 27/07/2017 17:31, Tommaso Cucinotta wrote:
>
>> I think what we might do in a relatively easy and generic way, is to allow
>> for setting individual preferences settings from the command-line, e.g.:
>>
>>alias lyx-safe='/usr/bin/lyx -Duse_converter_needauth_forbidden=true'
>>alias lyx-unsafe='/usr/bin/lyx -Duse_converter_needauth_forbidden=false
>> -Duse_converter_needauth=false'
>>
>> that would override whatever one has in the ~/.lyx/preferences config.
>>
>
I added a section to the wiki page to note that alias + options might be a
way.


>
> we have an already existing way through the LYXRC_APPLY LFUN, with a
> relatively heavy syntax:
>
> This is lyx-safe:
>
> ./src/lyx -x 'lyxrc-apply Format 22
> \use_converter_needauth_forbidden true'
>
> This is lyx-unsafe:
>
> ./src/lyx -x 'lyxrc-apply Format 22
> \use_converter_needauth_forbidden false
> \use_converter_needauth false'
>

I haven't added this to the wiki page as I don't understand the 'Format 22'
part.

Perhaps you could explain it, or add it to the page?

Would this be applied after LyX preferences/config has been read, but
before a document is opened?

/Christian

Re: Can shell-escape take advantage of needauth framework?

[Christian Ridderström]

On 25 July 2017 at 01:30, Tommaso Cucinotta <tomm...@lyx.org> wrote:

> On 18/07/2017 21:50, Christian Ridderström wrote:
>
>> I do not know how many KGB/CIA agents will be willing attend the 'hack
>> LyX' classes. How much is it worth on a spy resume ?
>>
>
> haha! something like that must have been said by someone in Redmond while
> coming out with this new brilliant and super-useful business-oriented
> automation feature called "Word Macro", before Melissa came out in 1999!
>

I'd be more interested in what people at Redmond said regarding their
design of the graphics format called Windows Metafile:
 https://en.wikipedia.org/wiki/Windows_Metafile_vulnerability
In short:
- The Windows Metafile format had a "feature" which allowed arbitrary
embedded code to be executed on affected computers without the permission
of their users.
- The feature won the 2007 Pwnie Award for "Mass 0wnage" and "Breaking the
Internet".
- Even a Unix-like system that uses Wine to emulate Windows, for example,
could be exploited
- The vulnerability is an inherent defect in the design of WMF files,
because the underlying architecture of such files includes features which
allow actual code to be executed whenever a WMF file opens

The last part kind of reminds me of Gnuplot. Up until Windows XP the
systems were pre-configured to allow WMF files to execute their code.

Anyway, I just thought it might be interesting to compare with WMF.

Personally, I wrote patent applications using LyX during one of my prior
> jobs in industry, and helped my colleagues into getting used to LyX,
> disregarding recommendations of my colleagues in that I should have used
> the "proper" .docx template... that can be conveniently edited using a much
> more secure program and OS?!?
>
> Btw, +1 on threat analysis, and np with the minefield, even simply talking
> about security risks is being a good thing for LyX, and a few mines could
> already be spotted :-)!
>

/Christian

Re: Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)

[Christian Ridderström]

On 24 July 2017 at 23:20, Tommaso Cucinotta <tomm...@lyx.org> wrote:

> On 23/07/2017 20:55, Christian Ridderström wrote:
>
>> Regarding setting something in the preference file manually: The only
>> thing I mind is that it adds a global state to LyX, as opposed to
>> starting LyX with some parameters. The global state would likely
>> affect e.g. testing.
>>
>
> the good thing is that we already have the -userdir command-line option
> that allows testing from a predictable initial state, doesn't it :-) ?
>

Good point, thanks!
/Christian



> (AFAICR, extensively used in autotests for advanced F).
>
> T.
>

Re: Any descriptions of the security aspects (related to needauth and shell-escape)?

[Christian Ridderström]

On 24 July 2017 at 23:27, Tommaso Cucinotta  wrote:
>
>
> I support the idea as well, and I'm interested in contributing to it.
>

I could help as well, at least with the outside view.


> As a starting point for the needauth stuff, I had put a recap of the
> problem and motivations within this TT:
>
>   http://www.lyx.org/trac/ticket/10481
>
> and, just as a reminder, the shell-escape was one of the mentioned
> use-cases, albeit not the very one under my hands at that moment.
>
> I'd suggest not to create a .lyx document, but rather start from a wiki
> page under
>
>   https://wiki.lyx.org/Devel/Devel
>
> with a recap of the current status and exchanged ideas, and pointers to
> the Trac TTs tracking TODOs which we agreed upon.
>

For me it doesn't matter so much where we place it.

IIRC, it's possible to password protect a wiki page. I'll see if I still
remember how to do that...
   tic=21:35, toc=21:42
Ok, not so bad. There's now this wiki page:
   http://wiki.lyx.org/Devel/SafetyAndSecurity

I'm sending the password separately to a bunch of developers, apologies for
forgetting some.

This doesn't mean I'm saying we have to, or should, use this page, I just
want us to have the option. And a wiki page might be a good place to start.
Oh... the protection of a wiki page is weak, think of it as stopping search
engines etc.
/Christian

Just a check: Ok with Qt-code in e.g. src/support/?

[Christian Ridderström]

Hi,

This is just a check. A long time ago LyX had two frontends, i.e. not
only Qt. Back then I assume the Qt-code was supposed to stay under
src/frontends/qt4/.

I just noticed some Qt code in e.g. src/support/FileMonitor.h, so I
just wanted to check that we now are ok with Qt-code in e.g.
/src/support/?

Cheers,
Christian

Curious: Why '--std=c++11' and '--std=gnu++11' in compiler options? (Was: Required C++ standard for building LyX, i.e. do we require >= C++11?)

[Christian Ridderström]

For my curiosity, I noted that in addition to '-std=c++11' among the
compiler options, there's also an option for '-std=gnu++11' in the compile
command. In my case I was using cmake and the compiler was Apples
LLVM/clang-compiler, not gcc.

Does anyone know why we use both '-std=c++11' and '-std=gnu++11'?

cd /Users/chrid/repos/lyx/m0/build-master/src/tex2lyx &&
 /Applications/Xcode.app/Contents/Developer/Toolchains/
XcodeDefault.xctoolchain/usr/bin/c++
-DBOOST_USER_CONFIG=""
-I/Users/chrid/repos/lyx/m0/build-master  ...
-Wall -Wunused-parameter
--std=c++11 -Wno-deprecated-register -fno-strict-aliasing
-Wall -Wunused-parameter
--std=c++11 -Wno-deprecated-register -fno-strict-aliasing
-O0 -g3 -D_DEBUG   -F/usr/local/Cellar/qt/4.8.7_2/lib
   -std=gnu++11
   -o CMakeFiles/tex2lyx.dir/Context.cpp.o -c
   /Users/chrid/repos/lyx/m0/src/tex2lyx/Context.cpp

/Christian

PS. There also seems to be a duplication of the '--std=c++11' as well as
the next two options.

Re: Silent/automatic execution of converter and needauth, concrete questions to clarify my understanding

[Christian Ridderström]

On 18 July 2017 at 09:06, Scott Kostyshak <skost...@lyx.org> wrote:
> On Mon, Jul 17, 2017 at 11:53:38PM +0200, Christian Ridderström wrote:
>
>> A) In LyX 2.2.x, if I open the document, no "converters" are executed. But
>> when I attempt to generate the PDF, the document could via e.g. 'R' execute
>> arbitrary code on my computer, as if it were my user account. And this
>> would happen silently, with no warning etc.
>> Correct?
>
> Yes.
>
>> But what would happen if I used LyX 2.3.0alphaX and tried to build the
>> document?
>
> Guillaume gave a more detailed answer. The quick answer is that with the
> defaults of 2.3.0alpha1-1, you would be prompted before the R code was
> run.

Thanks, it's clearer now.

Are the settings that needauth remember done:
a) per document, regardless of converter
b) per document-and-converter pair?
c) Also per snippet of code?

E.g., what happens if I'm keeping a document on say a network drive. I
put some code in the document and execute it. When asked by needauth
the first time, I say "always allow for the document".   So the next
time I execute the document I'm not asked again.

What happens now if someone else modifies the code embedded in the
document?  Will the permission(s) still be active, so that the
document executes the new code?  Am I warned in any way?

If not, perhaps a future improvement could be to be able to approve
specific code snippets to be executed.
The user-dir could e.g. contain a hash of code snippets that's
approved to be run for a certain document. Or perhaps even for all
kinds of documents.
/Christian

PS. Heh.. maybe we could use Git to store approved/disapproved code
snippets as it's a content based filesystem.

Types of LyX users (Was: Can shell-escape take advantage of needauth framework?)

[Christian Ridderström]

On 21 July 2017 at 22:12, Scott Kostyshak  wrote:
> On Tue, Jul 18, 2017 at 11:21:38AM +0200, Jean-Marc Lasgouttes wrote:
>> Le 18/07/2017 à 09:07, Scott Kostyshak a écrit :
>> > I was thinking about it from a different angle. I was only focused on
>> > what I thought was most secure, without even considering usability. As I
>> > mentioned in the thread asking for votes, I believe that we should focus
>> > completely on what is the most secure.
>>
>> Well, what is the most secure is to remove all sweave/gnuplot/minted code.
>> There is no point in looking at security without usability IMO.
>
> I see what you mean and I think most people would agree with your
> interpretation. I was taking the approach more of "under which proposal
> is the user least likely to run malicious code". In your scenario (let's
> remove all sweave/gnuplot/minted code), well sweave users would just
> never upgrade LyX and would lose any security-related improvements and
> would not have any of the protection that needauth provides. For minted
> users, they would have to do the '-shell-escape' dance and would have
> the risk of forgetting that they left a converter permanently changed.
> This is what I mean by "less secure". But I know that I'm thinking about
> things differently from others. I can understand the other perspective
> of security of "if a user uses only built-in LyX with no customizations,
> then they would be less likely to run malicious code". I just think the
> "if" in that statement is concerning.

I think Scott is partly verging towards the topic of types of users
and user scenarios.
IMHO these aspects are quite important factors when discussing
features, security, UI and what to include in a software.

What kind of user was I, perhaps the archetypical LyX user:
- Started using LyX while working on my PhD (back in 1997 or so, in
case it matters)
- Did not know LaTeX in advance, thought it might make sense with a
graphical frontend.
- Wrote articles etc in LyX, learned some LaTeX on the way, put stuff
in preamble and some ERT.
- Only added some converters to include TGIF images or something like that.
- Besides LaTeX, I never embedded code in the LyX document
- This user googles to find solutions.
- A more advanced version of this user might start asking question on
the users' list.
- LyX worked really well for what I was doing.


Another kind of user could be the "the supervisor/reviewer".
- The student has a supervisor or colleague that he'd like to review his work
- Only minor editing is expected of the reviewer, perhaps adding comments,
  perhaps fixing an error.
Note: My supervisor got printouts back then IIRC.

Related is the use case is when two or more people closely collaborate
on a document.
Perhaps they use version control to work on the same LyX document.
Or keep it on a network drive. Or on e.g. Dropbox. Doing this adds
requirements on LyX.


More advanced user:
- LyX is used to build the main document, perhaps there are child documents.
- Perhaps exporting/publishing to multiple different formats.
- The document pulls in information from external files, e.g.
.tex-files with data.
- However, .tex-files are updated externally to the use of LyX,
  so the user has to manage dependencies etc.

Then we have the kind of user for whom I don't have a name, nor know well:
- Includes LyX deeply in his work flow
- Embeds code to be executed, perhaps repeatedly, in his document
- Using LyX as an IDE to develop his "program"
- ?

Non-user:
- Like my girlfriend, who likes writing in LaTeX
- I tried to get her to use LyX but it didn't take
   - she e.g. didn't like having to go through the menus all the time
 she would've preferred being able to use the keyboard all the time.
 So using plain LaTeX was "easier".
- At some point I should get more details on this.

Perhaps there's more kinds of users?

Anyway, the connection with security and shell-escape etc is that only
one of these kinds of users would likely actually need to use
shell-escape, and that user is probably more capable. OTOH, maybe we
_want_ to make LyX a tool where using e.g. R from within LyX is a
really good experience.

If these categories are reasonable, it'd be interesting to know the
distribution of users.
/Christian

PS. These days I find myself writing work notes in Emacs' org-mode a
lot of the time, why?
- Speed. It's quicker to type/edit in LaTeX.
  Perhaps because LyX's keyboard isn't working so well for me on my
mac with a Swedish keyboard.
- I typically don't need so many formulas. Org-mode is fine for a few
formulas, including images.
- Sometimes I even embed executable code (MATLAB) in my org-mode file.
- Often I don't need pretty output.
- Very convenient with text-based files that work well with version control.
- Perhaps also related that I had to go through five years or so where
I was forced
  to write in Word, and it left me damaged.

Re: Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)

[Christian Ridderström]

On 19 July 2017 at 12:00, Jean-Marc Lasgouttes <lasgout...@lyx.org> wrote:
> Le 19/07/2017 à 07:48, Christian Ridderström a écrit :
>>
>> If user does not want all these warnings, he could disable them by
>> launching LyX with some option like "--do-not-warn-me-about-unsafe-setting".
>> Instead of having a checkbox for "don't tell me these things again".
>
>
> It has the same issues as the checkbox when one considers batch command line
> processing. People add the switch to the command line and forget about it.

IMHO a user who does batch processing is pretty advanced...

Regarding setting something in the preference file manually: The only
thing I mind is that it adds a global state to LyX, as opposed to
starting LyX with some parameters. The global state would likely
affect e.g. testing.

/Christian

> The requested operation requires the use of a converter from sweave to
> pdflatex:
> Rscript --verbose --no-save --no-restore $$s/scripts/lyxsweave.R $$p$$i
> $$p$$o $$e $$r
> This external program can execute arbitrary commands on your system,
> including dangerous ones, if instructed to do so by a maliciously crafted
> .lyx document.
> Your current preference settings forbid its execution.
> (To change this setting, go to Preferences ▹ File Handling ▹ Converters and
> uncheck Security ▹ Forbid needauth converters.)

That's a good message in my opinion, especially as it points out the
"including dangerous ones".

Living with shell-escape: Using two LyX instances - critique invited

[Christian Ridderström]

Bah, I again e-mailed only Guillaume and not the list.

On 19 July 2017 at 00:00, Guillaume MM  wrote:
>
>
> I find that it would be more cumbersome and error-prone than a good
> needauth implementation.


cumbersome:
Do you refer to using two user dirs, or perhaps having to (once?)
modify the parameter of the converter settings?
Or do you mean having two LyX windows open at the same time?

error-prone:
Do you mean that you'll open the doc^H^H^Hprogram [*] in normal mode
and when you try to build it fails?
Or do you think it'd be difficult to set this thing up?

Perhaps it's an important aspect that we're talking about a scenario
where the user is executing code embedded in his document, thus
essentially using LyX as an IDE to develop and execute programs.

>
> If I understand, what you want is visibility
> and revokability, which people already seem to agree are desirable
> improvements to make to needauth (a red status bar thingy).


Yes, visibility that I'm operating in an unusual/non-default and
potentially dangerous mode.

Revokability I'd rather say isolation/separation.

Besides wanting to allow the use of shell-escape only for a limited
set of documents (i.e. documents), I would likely also only want to
allow shell-escape when I want to run the program.  If I only need to
review/edit the program, e.g. after having received it (or had it
returned after modification) from an external source.

By only opening the unsafe LyX when I have to, I would know that
everything else I do in normal LyX is safe.  So I would only have to
be "careful" when using unsage-LyX.

Anyway, I think we should strive to allow a design where shell-escape
is not needed. And this topic is about a fallback when shell-escape is
necessary.

Guenter also wrote:
> If I got Christian right, the suggestion was intended as
>  stop-gap measure for power-users of LyX <= 2.2.x (as is my alternative 
> proposal).

Yes, it's a stop-gap measure. Depending on what happens for LyX >= 2.3
it might also be useful in the future.


/Christian

[*] They way converters are used here, the LyX document contains the
source code of something that can be executed. So it might help to
think of it as something containing source that we execute when
building (or previewing) the document. Thinking of it as running a
program might also help you think more about side effects of building
the document.   It also makes me think of providing better visibility
into exactly what code you are running, and perhaps if there could be
a way to "sign" versions of the code in the document then trust the
code when it's still signed.

Re: Any descriptions of the security aspects (related to needauth and shell-escape)?

[Christian Ridderström]

On 21 July 2017 at 22:28, Scott Kostyshak <skost...@lyx.org> wrote:

> On Wed, Jul 19, 2017 at 07:34:59PM +, Guenter Milde wrote:
> > On 2017-07-19, Christian Ridderström wrote:
> >
> > ...
> > > ... I would like to ask (not being optimistic), if there's some design
> description anywhere?
>
> > > I think this kind of information would be good to gather and store in
> some
> > > kind of design document, which could just be a text file in the repo.
> Then
> > > we could add knowledge to this document, and let if include the
> rationale
> > > behind our choices, as well as letting developers review the system
> design.
> >
> > I support the suggestion to create such a document and suppose to make it
> > a section in "Development.lyx":
> >
> > + bundled with other project policies and developer documentation
> > + write access for all developers
> > + we can use LyX's version control for to-be-reviewed parts and diverging
> >   opinions/comments
>
> +1
>
> Development.lyx does not have a rigorous structure. If anyone is
> interested in writing something more formal, when we can at reference
> that file from within Development.lyx.
>

So two suggestions for location are as follows:
- In Development.lyx
- Other document parallell to Development.lyx

A third alternative of which I was thinking was to create a separate
document that's included by Development.lyx.

Does anyone feel we should _not_ keep such a security document in the
general LyX repository?

Note: Generally speaking I'm all for being open and transparent, but such a
document might end up containing descriptions of ways in which LyX could be
compromised. It sort of goes with the territory of describing threat
scenarios/mechanisms.

/Christian

PS
Generally speaking I think it'd probably be a good experience for us
developers to collaborate on a LyX document in a Git repo. I suspect it'd
give us some ideas for improvements.

Tip: Generating/showing a big diff (related to fixing namespace comments in the code)

[Christian Ridderström]

Hi,

FYI, I did a cleanup of the comments that marks the end of a namespace in
the source.

If anyone would like to see a side-by-side diff in HTML, I put one here:

   https://chr.updog.co/fix_namespace_comments_diff.html

The details of how I did the work is in the commit message, but of main
interest is that I've manually reviewed all the changes and then verified
by building.
/Christian

PS

I generated and show the diff using two tools/methods I haven't used
before, maybe they're of interest to others:

- diff2html, https://diff2html.xyz/ - I used it to generate the
html-version of the diff.
  Pretty useful for big diffs.

- updog.co - A simple services to serve web pages stored in your Dropbox
folder

Any descriptions of the security aspects (related to needauth and shell-escape)?

[Christian Ridderström]

Hi,

When having tried to contribute to the discussion on needauth and
shell-escape I've felt that it's quite difficult to get a good picture of
things like:
- Goals of design, what are we trying to achieve
- Principle of design and system
- Assumed threat models, and perhaps list threat scenarios we _don't_ try
to protect against

The e-mail threads are ... long, sometimes confusing and I suspect contains
at least a few misunderstandings.  So I would like to ask (not being
optimistic), if there's some design description anywhere?

I wonder because IMHO security requires a system wide approach and that
it's very easy to screw up if only looking at isolated pieces. Further, it
requires continuity so you know what you initially intended to achieve and
what you consider good enough. Otherwise you might later introduce a new
feature that inadvertently opens up a security whole. Without a system
design, it's also easy to get caught in discussions trying to bandaid a
small hole while missing entire walls missing.

I think this kind of information would be good to gather and store in some
kind of design document, which could just be a text file in the repo.  Then
we could add knowledge to this document, and let if include the rationale
behind our choices, as well as letting developers review the system design.

Of course, using a design document isn't something this project is used to
so of course there's a risk it'll e.g. forgotten.  But I feel I should at
least suggest this.

Regards,
Christian

Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)

[Christian Ridderström]

On 18 July 2017 at 23:49, Jean-Marc Lasgouttes <lasgout...@lyx.org> wrote:

> Le 18/07/2017 à 23:42, Christian Ridderström a écrit :
>
>> I think the default should be secure, and that the user should have to do
>> something actively to go into a dangerous mode.
>>
>
> Well, since you consider that turning off two options is not active
> enough, I am not sure what to propose :)


The problem I see with only unchecking two check boxes are e.g.:
- Users uncheck settings all the time, it doesn't seem very "scary"
- In the settings dialog, the real implications of unchecking these options
  did not seem sufficiently clear to me.
  So calling it "Allow yourself to be shot in the foot by converters" would
help;-)
- The setting is persistent, and easily forgotten

As an aside, I think that if the user had to launch LyX from a terminal
with a parameter instead of just unchecking, the user would think he's
doing something really advanced/scary. Simply because he's not used to the
terminal.

Why does disabling something like needauth have to be done from within LyX?

If it has to be done from within LyX, then perhaps do some of the things
below to make being in unsafe mode more difficult to forget:
- When unchecking the boxes, display a dialog informing them that they're
going into dangerous territory.
- Show the warning each time LyX is started, forcing the user to
acknowledge it.
  And make it so that user with a single click can reenable needauth.
- Enable a strong/annoying visual indication/reminder that you're unsafe
mode
- Possibly show the dialog each time before building a document

If user does not want all these warnings, he could disable them by
launching LyX with some option like "--do-not-warn-me-about-unsafe-setting".
Instead of having a checkbox for "don't tell me these things again".


> So one question is how much R code or similar you have in typical
>> documents.
>>
>
> My typical documents are what I do for my data analysis course, where all
> the computation is done when typesetting the file (using Sweave). So, yes,
> I kind of rely on it.
>

Could you send me (privately) one of your docs with "a lot" of R source?
I'd like to see what it looks like and how much code we're talking about.

/Christian

>
>

Re: Can shell-escape take advantage of needauth framework?

[Christian Ridderström]

On 18 July 2017 at 22:09, Jean-Marc Lasgouttes <lasgout...@lyx.org> wrote:

> Le 18/07/2017 à 21:50, Christian Ridderström a écrit :
>
>> That you argue this way makes me sad.. and embarrassed/ashamed on behalf
>> of the project.  I could counter all your points in the paragraph above,
>> but I worry it's a waste of time and to be perfectly honest I'm a little to
>> upset right now to continue this discussion.
>>
>
> Wow, I just stepped on a mine. I'll retreat from the mine field before
> causing more harm, then.
>

My apologies for blowing up on you!
My only excuse is that we have different experiences, and .. uh... perhaps
it's related to lack of sleep due to my daughter waking every hour last
night. Yes, that might be a factor.

So please keep stomping into the minefield.

The threat model is one important aspect, but it's difficult for us to know
who uses LyX and in which industries. Or how many users there are at all.
And how many of them that use converters.  If we can achieve good security
we don't need to care about user / usage statistics though.

If we talk principles, I think we should aim for really good security and
then discuss compromises for what's not doable. But I do think we could do
a whole lot better than the current 'needauth'.

Sincere regards,
/Christian

[macOS] Behaviour when using an absolute path when doing save as

[Christian Ridderström]

Hi,

I just noticed a minor thing, and perhaps fully due to Qt and/or macOS.

Steps to reproduce:
- Start new document
- Place '/tmp/test.lyx' into your copy buffer
- Press 'Save' (Cmd-S)
- Paste filename from copy buffer, i.e. /tmp/test.lyx

Expected result:
I expected the file to be saved as /tmp/test.lyx

Discrepancy:
File was stored as :tmp:test.lyx under ~/Documents/.

Is this the expected behaviour?
/Christian

PS.
I did notice that if I type a '/' in the save dialog I get a dialog to
change folder. So this problem happened to me because I pasted in a path.

Re: Can shell-escape take advantage of needauth framework?

[Christian Ridderström]

On 18 July 2017 at 21:15, Jean-Marc Lasgouttes <lasgout...@lyx.org> wrote:

> Le 18/07/2017 à 19:46, Christian Ridderström a écrit :
>
>> I just did a test with gnuplot. In the LyX settings I had unchecked
>> 'Forbid of use of needauth converters' and unchecked 'Use needauth option'.
>> Then I opened a LyX doc with a gnuplot script. Result: LyX tried to run the
>> script due to the preview, without asking or alerting me.
>>
>> In my opinion this demonstrates a case where the security is _not_ good
>> enough, as I don't think it'd very difficult to trick someone into
>> unchecking these boxes.
>>
>
> You may want to rename one of these options "shoot yourself in the foot".
>

You're joking, but I think you're correct. The labels for these options are
not sufficiently clear, the term "needauth" isn't very ... terrifying ;-)


> Seriously, one thing I learned about security is that the size of the lock
> you use should be related to the threat you are fearing.


Yes, you are quite right about this. Another is aspect is the effort
required to implement the attack and the probability of success. Here it's
not a lot of development time required to produce a malicious payload. It
remains only to trick the user into running the document, which is
something accomplished actors are already experienced at.


> Do we really work on the scenario where someone (CIA? KGB?) will be trying
> to trick a LyX user (how many of users are a worthy target?) into changing
> its own preferences in order to –let's say– steal all the readable files on
> the user directory. IMO,  if a hacker is ready to do this (including social
> engineering), the user will have other problems than the brand new needauth
> feature of the obscure editor LyX.


> I do not know how many KGB/CIA agents will be willing attend the 'hack
> LyX' classes. How much is it worth on a spy resume ?
>
> We should start by understanding what are the reasonable threats we want
> to fight against. This discussion is becoming crazy.
>

That you argue this way makes me sad.. and embarrassed/ashamed on behalf of
the project.  I could counter all your points in the paragraph above, but I
worry it's a waste of time and to be perfectly honest I'm a little to upset
right now to continue this discussion.

/Christian

Living with shell-escape: Using two LyX instances - critique invited

[Christian Ridderström]

Hi,

If I had to use a converter that requires e.g. shell-escape perhaps the
approach below would be useful. What problems do you see with it?

1) Use two lyx user directories, one standard and one "dangerous", with
converters using shell-escape only in the dangerous lyx.

2) Create a tiny shell script or a desktop icon etc to launch "dangerous"
lyx using the "dangerous" user dir.

3) Configure dangerous lyx to have a reddish background colour.

4) By default work in the regular lyx and only use the dangerous lyx
for documents where converters are needed.

Justification:
The 1) allows me to have converters permanently configured in a dangerous
mode, and through 4) I select in which mode I work. The 4) also reduces
risk exposure as I should not open documents from strangers in the
dangerous lyx.

The 2) makes it easy for me to launch the dangerous lyx. I've actually used
this approach for a requirements tool, to open it in a special mode.

The 3) makes it clear to me in which LyX I'm working.

By working as per above I'm reducing exposure and don't have to worry about
shell-escape in normal documents.

What would the drawbacks be with this approach?
/Christian

PS. Documentation wise, I don't think it'd be that difficult to explain how
it's done.

Re: Can shell-escape take advantage of needauth framework?

[Christian Ridderström]

On 18 July 2017 at 11:32, Guillaume MM  wrote:

> Once it is in, then it
>>> has to be supported forever, I believe there is an agreement about this.
>>>
>>
>> I wouldn't say this in absolute terms, but I would agree that there's
>> lots of hesitation before removing a feature, and that hesitation only
>> increases with time. But not that we have removed features. For example,
>> we removed support for printing, even though there were still some using
>> the feature.
>>
>
> I agree, but note that for printing this did not invalidate existing
> documents.


I just did a test with gnuplot. In the LyX settings I had unchecked 'Forbid
of use of needauth converters' and unchecked 'Use needauth option'. Then I
opened a LyX doc with a gnuplot script. Result: LyX tried to run the script
due to the preview, without asking or alerting me.

In my opinion this demonstrates a case where the security is _not_ good
enough, as I don't think it'd very difficult to trick someone into
unchecking these boxes.

I therefore think we should discuss the pros/cons of needauth. If needauth
cannot be shown to be secure enough, or if we don't think we can reasonably
fix it, then my opinion is that we should discuss removing needauth.

Presumably the number of users needing R/knitr/sweave is small compared to
the total user base and I don't think it's fair to leave the majority at
risk.

At the same time I definitely think that users should be able to build
their old documents in new releases of LyX.

So, if needauth cannot be shown to be good enough, how can we support users
of R etc?
Some alternatives:

- Require that they, for these documents, stay with the 2.2.x-series, until
we have a sufficiently good security mechanism.

- Only allow the dangerous behaviour of 2.2.x if the user starts LyX 2.3.0
with a special flag.

- Force them to compile their own LyX with a special flag set.

- ?

Regards,
Christian

Silent/automatic execution of converter and needauth, concrete questions to clarify my understanding

[Christian Ridderström]

Hi,

I've gotten lots of information from Enrico and Guillaume related to the
security "gap", but I'd like to boil it down to simpler questions to make
the situation clear to me.

Assume that I've gotten a LyX document by e-mail. It was not created by me,
but let's say that the sender of the e-mail appears to be from a colleague
whom I trust, asking me to do him a favour and generate a PDF because his
computer is acting up. It's urgent of course...

A) In LyX 2.2.x, if I open the document, no "converters" are executed. But
when I attempt to generate the PDF, the document could via e.g. 'R' execute
arbitrary code on my computer, as if it were my user account. And this
would happen silently, with no warning etc.
Correct?

But what would happen if I used LyX 2.3.0alphaX and tried to build the
document?
B) Would LyX still allow the document to run arbitrary code on my computer?

C) Would the execution still happen "silently"?

D) Can the above happen with a document completely created by someone else?

Note that for all questions above I assume that the person who sent me the
document has attempted to configure it to allow the above, i.e. he's set
any flags etc he could within the document.

Regards,
Christian

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 17 July 2017 at 00:57, Enrico Forestieri <for...@lyx.org> wrote:

> On Mon, Jul 17, 2017 at 12:49:05AM +0200, Christian Ridderström wrote:
> >
> > Enrico argued that there are other (equally) dangerous converters already
> > in LyX. Then that's something to address. Does it have to be for this
> > release? If that's something to discuss, I can't say. Are many users
> > currently exposed, i.e. likely to be using it?  It's bad if we have
> > security holes, but it's not necessarily good to immediately yank
> something
> > out.  On the plus side, you as the release manager can decide what's
> needed
> > here as far as I am concerned.
>
> Dear Christian,
>
> I fear that this minted issue is a very well constructed case. At the
> moment there is no way you can risk something if not manually going through
> changing preferences. On the contrary, other features simply require a
> left click with the mouse to cause danger. It is really surprising that
> these features are not considered harmful while minted support is.
> But I am not surprised, because these are called FUD strategies and
> have always been used to muddy the waters and confound people.


Let me see if I understand this correctly, and perhaps it'll unconfuse some
others as well.

Regarding, Minted, which is an alternative to insert pretty program
listings in your document.

At the moment it takes manual (typing) work to cause security issues in
connection with minted.
The "at the moment", likely refer to e.g. LyX 2.2.2 as [1] gives a nice
illustration with screenshots on how to add the option '-shell-escape' to
the converter 'pdflatex'.  The downside with adding this option is that now
other LaTeX code in the document has the possibility of doing "bad" stuff
to my system.

Further, my LyX is now configured with this -shell-escpae that will then be
active for all other LyX documents that I build. Oops.

Note: I'd probably deal with the security issue here by using two
separately configured LyX instances that use different '-userdir':s.
It would be nice with a strong visual warning that I'm using the "unsafe"
LyX though, but i guess I could manually configure a different paper colour
in LyX.

If I've understood the proposed patches correctly, they involve making it
easier for the user to enable -shell-escape, and also easier to disable
shell-escape.  I'm torn here. Some of the proposed UI-approaches weren't
bad, but I'd probably still worry that we're making it to easy for the user
to do dangerous things.

For Minted I'd then prefer to keep the old behaviour for now and add better
integration when/if minted doesn't need shell-escape.

As for the other dangerous features, presumably related to something called
"needauth", I don't know anything...
I googled and found this [2]:


The converters definition syntax (LYX_HOME/lyxrc*) now supports a new
option, 'needauth', to prevent completely automated execution of the
converter, unless LyX acquired explicit consent by the user. This is a new
security feature, useful for converters that are capable of executing
arbitrary code, such as R scripts (used with sweave/knitr), embedded within
LyX documents. The user needs to explicitly grant per-document permission
on the first need for using the converter on each document, unless he/she
checks the "Don't ask again for this document" checkbox in the permission
dialog. The new behavior can be fine-tuned from two new options in the
preferences dialog (see their description below). These also allow for
disabling 'needauth' converters altogether, if desired (default behavior).


I don't understand if the 'needauth' is new in LyX 2.3.0 or already existed.

However, and here I'm probably offending people and stepping on mines in a
single paragraph. This seems bad to me.
The text indicates to me that it's possible for a document to store some
kind of setting that allows a converter (here external program) to be run
in an automated manner without my manual intervention or consent.
Supposedly I first had to check "Don't ask again for this document" but
consider the following example:
I create a document with some embedded code to be run by converters. It's
my document, I trust it. Then I e-mail it to a colleague or perhaps my
customer for review. Time goes by and eventually I get the document back
from review, but the review took longer than expected and my next deadline
was yesterday, so I'm in a hurry and build "my" document.   Oops, some of
the embedded code is now malicious, but the document still contains my
setting that lets the converters execute... So apologies for not knowing
the details here, but if this is being introduced in LyX 2.3.0 it sounds
like it could be pretty bad and I think the security aspects should be
discussed.  {KABOOM} is hopefully the sound effect when someone points me
t

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 17 July 2017 at 00:48, Enrico Forestieri  wrote:

> Dear Christian,
>
> I see that the operated obfuscation of issues is working with you.
>

Dear Enrico,

Don't worry, all is not lost and any operation obfuscation has not yet
succeeded in invading Sweden. I still know that I don't know this topic
well enough to have a very useful opinion.

But perhaps better to continue discussing this in a more suitable thread?
Otherwise my plan for world domination by using clang-format to obfuscate
the entire LyX code base in one fell swoop will drown in discussions that
actually matter ;-)


> At the moment, there is no security problem with minted.
>
On the contrary
> there is a big question about security of features that were either
> present or recently introduced. Of course, it was the ability by which
> these questions were treated that lead you to think that the problem
> is minted, specifically, while it is not. ATM, in no way you can risk
> something if you decide to use minted.
>

It's good to know although I'm anyway not personally at risk, unfortunately
I'm currently not writing in LyX.

Cheers,
/Christian

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 5 July 2017 at 06:59, Scott Kostyshak  wrote:

> Dear all,
>
> This is an important topic since it involves security. I'd appreciate it
> if you spent some time on understanding the issue.
>
> I see three options for what to do about the minted + shell-escape
> issue:
>
> 1. Revert the recently added minted support.
>
> 2. Keep the current state of master.
>
> 3. Apply the patch at [1] (also attached for convenience).
>

Hi Scott,

I think you're doing an excellent job as release manager and also when
summarising the issue. I therefore have to apologise that even though I
have read various postings/threads on this topic, I'm still confused.
Perhaps I can expand on that:

- What do we lose by doing 1) above?
- Is 1) necessary for users in order to use minted, or is it a convenience
feature?
- How many users do we think need / want to use minted, i.e. how many users
are affected?

Opinions seem to vary regarding what's secure and I cannot say anything to
that. My general inclination is to prefer that the user to go through a
bunch of manual steps in order to use dangerous features. If they remain in
place, so be it. But perhaps we could warn the user when he's configured
converters to use shell-escape, or if LyX was started with some global
option enabling shell-escape. Perhaps by making the LyX window "look
dangerous" like private browser windows, or more realistically flash a
warning before each build.

As an aside, I've used org-mode documents in Emacs to invoke MATLAB on
snippets from within the document and it's very nice, except the really
annoying part where you for each snippet have to approve that it's run.
Each time. A big bug in org-mode is that it's not properly showing the
snippet that's about to be executed before you approve it to be run...
Perhaps I'm a masochist...

Enrico argued that there are other (equally) dangerous converters already
in LyX. Then that's something to address. Does it have to be for this
release? If that's something to discuss, I can't say. Are many users
currently exposed, i.e. likely to be using it?  It's bad if we have
security holes, but it's not necessarily good to immediately yank something
out.  On the plus side, you as the release manager can decide what's needed
here as far as I am concerned.

I'm sorry for not being of more help, but hopefully my comments will
trigger someone else to contribute something more helpful.

/Christian

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 16 July 2017 at 22:45, Jean-Marc Lasgouttes  wrote:

> What I mean is that my absolute priority these days is to have 2.3.0 out.


Fully understood.


> The cleanups I proposed where chosen to have a minimal effect on release
> date. Anything that requires too much thinking is a bit too much for me.
> Currently minted and hyphen are blocking us, and we should work towards
> solving this (even though these are touchy subjects).


I just went through a large chunk of the minted postings and I still don't
have a clear idea about my preference, and I'm therefore not sure what to
write that'd contribute.

I'm generally inclined towards security and backwards compatibility.
Perhaps it's because I experienced a directed attack at a previous
workplace. Or perhaps it's an occupational hazard from previously working
with satellite software as e.g. verification and validation manager. But
even for that SW we took into account if there was a urgent and necessary
need for e.g. intermediate release, assuming we had a realistic plan for
fixing issues in a coming release.

For minted/hyphen, I'm e.g. not clear on the need for the features. The
minted thing seems more optional, whereas the hyphen thing seems like a
blocker. I haven't read the hyphen stuff yet, but my baseline is that I'd
really hate it if I wasn't able to compile documents I wrote a long time
ago.

Security scenarios/threat models for minted could be expanded upon. I mean
that these days it's not just about me creating and editing my own
document, instead authors collaborate and share the documents via e-mail
and Dropbox etc. Theoretically some agency could intercept the document in
transit and inject malicious code that they hope you'll execute on your
computer.  Further, you might not be the direct target and instead its
someone whose computer is on the same network as you. For instance, I've
seen research papers from a swedish defence research institute written in
LaTeX, or perhaps LyX, who knows. But if it was LyX then it could well be
worth it for an adversary (Russia..) to compromise that author's computer.
I'd better stop writing now.

Cheers,
Christian

PS.
Scott, if you see this, let me take the opportunity to praise your work as
release manager -- you're doing an excellent job from what I've read!  It's
a shame you can't just setup a teleconference to discuss the issues.

Jean-Marc, in the best of all worlds there'd be a developer meeting planned
for the near future where these things could've been discussed face to
face.

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 16 July 2017 at 21:39, Jean-Marc Lasgouttes  wrote:

> Le 16/07/2017 à 21:34, Kornel Benko a écrit :
>
>> If not now, then probably never. There is no optimal start, except
>> at start of a project.
>>
>
> Not necessarily. We do not have much spare time to do it right, and this
> kind of thing needs to be correct on first run.
>

I'm not sure what you mean here.

Regarding formatting choices, I don't believe they have to be correct on
the first run. Or rather, I don't think they have to be "complete" on the
first run. You _do_ want to avoid large sets of changes that you later
revert. But I don't see it as a big drawback to e.g. initially use
clang-format without reformatting very long lines (we currently have >3500
lines that exceed 80 characters). At a later stage, we could then modify
the formatting configuration to get rid of long lines, or just do it
manually. Or leave them.


> We can discuss this in the 2.4 cycle, even if we decide to apply it only
> at the end of the cycle. Making sure that the style we choose can be
> obtained with everyone editor of choice is not a small affair.
>

Could you expand on the editor aspect?

I'm used to invoking clang-format from within Emacs on either the entire
buffer or on a specific region. Basically using clang-format has changed
how I code. These days while writing I've  stopped caring about whitespace
formatting. Instead I intermittently I do 'clang-format-buffer' to make it
pretty, e.g. before reviewing what I've done and definitely before commits.
I know people map some TAB-shortcut to invoke 'clang-format-region' easily.

I also know there's integration for clang-format for at least Emacs, Vim,
BBEdit, Visual Studio, XCode and Atom. Clang-format can of course also be
invoked from the command line. And it can be applied using e.g. 'git
clang-format' on only the parts that are being committed.

However, I'm _not_ expecting that most developers have to use clang-format.
Or even that they will...  If developers that regularly commit were to
start using it that'd be great.

I hope the info above helped.
/Christian

PS.
Note: I could create a CI job that warns if we start getting a lot of bad
formatting in commits. Or if you want to be tyrannical (not my
recommendation), you can have a pre-commit hook.




>
> JMarc
>
>

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 16 July 2017 at 21:15, Jean-Marc Lasgouttes  wrote:

> Le 16/07/2017 à 20:51, Scott Kostyshak a écrit :
>
>> "no debate" is good, but we want even more than that. We want a few more
>> "yes let's go for it!" before we impose a new style. I think we got
>> support from Kornel, but we would want more to go forward.  There are
>> other costs to changing many lines of the source code, such as "git
>> blame" not being as easy/helpful.
>>
>
> Indeed, my original message was about doing things that had a moderate
> price tag attached to them. Imposing a style is IMO too expensive for us at
> this point.


I'm definitely not looking to impose a new style and I'm spending time to
get close to the "existing" formatting style.
I used quotations marks around "existing" as there's a bit of variation in
what's currently used.

Here's my thinking/planning:

Once I'm close with a .clang-format file, I planned on assessing/presenting
how much impact applying clang-format would have on the code base, as well
as show what it'd look like. But also give some alternatives for
implementing it, e.g.:

- applying to all files at once

- for now only apply to source that haven't been modified in a long time

- for now only apply to source where there's only minor/trivial formatting
change
  (this is conceptually similar to Richard's removal of trailing whitespace)

- only add the .clang-format, letting developers use it as they please when
preparing patches

- suggest use of clang-format as part of a pre-commit hook, either on an
entire source file or only on the changed parts.
  (there's a cute command: git clang-format)

I need to think/read a little more on these alternatives, and see how
they'd interact with the code before I could recommend one.

Speaking of reading/thinking, Scott's right about that there can be a cost
associated:
- the time I'm putting in, which is quite a few hours actually but I'm
enjoying it and learning things
- effect on e.g. "git blame"
- risk of introducing errors (I think I've got that covered though)
- cost for developers with pending work in their own branches
- cost of issues related to differences between versions of clang-format
  (I'm currently looking at this, by comparing the result of using ver. 3.8
vs. 3.9 vs 4.0 vs. 5.0)
- ?

/Christian

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 15 July 2017 at 19:06, Jean-Marc Lasgouttes <lasgout...@lyx.org> wrote:

> Le 15/07/2017 à 18:55, Christian Ridderström a écrit :
>
>> In my opinion, if we don't reach consensus easily on formatting issues,
>> we should at least for now refrain from using a .clang-format.
>>
>
> This is probably what is going to take too much time...  I am not sure
> that we should have such a debate now.
>

I hope we can at least see if a debate is necessary, we might get lucky.
Perhaps the current source code is mainly consistent, in which case we
could align to that format. I can at least compile a list of topics.

Btw, what (if any) version of clang-format do you have on your system?
/Christian

PS.

>
> The trick is to avoid using tabs for space alignment, like emacs'
> smart-tabs-mode does:
> https://www.emacswiki.org/emacs/SmartTabs
>
> I have to admit that I have not done it yes, and therefore I do that by
> hand currently.


Clang format uses TAB for block indentation and spaces for the rest, i.e.
Emacs smart-tabs, through the setting

UseTabStyle: ForIndentation

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 7 July 2017 at 04:37, Scott Kostyshak  wrote:

> > What do others think?
>
> ^ If you get support from other LyX devs, and you are willing to take
> care of everything, then I'm find with it. My only other criterion is
> that I don't want to personally spend any time on this. We have other
> issues and debates going on where I want to spend my time. So the hard
> part is convincing other LyX devs.
>

Hi Scott,

I'm willing to take care of it and have been busy demonstrating it's safe
to use clang-format. The "simple" task of verifying that a binary based on
reformatted code is identical took much longer than expected and turned out
to be non-trivial. E.g., building LyX twice in a row from the same source
and from the same location didn't result the same binary on my mac.  So I
had to come up with an alternative approach based on doing pair-wise
comparisons of the disassembled code of object files.

The principle is that if two sets of source code results in object files
with the same assembler code, then the two sets are equivalent code wise. I
think it's a pretty solid approach. I'm going to post the details
separately to let developers assess if they think the approach sufficiently
well demonstrate that it's safe to use clang-format.

I used the approach only on macOS and in essence I produced two sets of
source code files and then built them separately. Then for each pair of
object files I did the following:
  - Strip symbols (strip -u -r) from the two object files
  - Disassemble (objdump -d) the two object files into assembly code
  - Compare the assembly code of the two object files

It was educational and I had to do some workarounds, but my conclusion is
that it's sufficiently safe to apply clang-format, with the configuration I
created.
If deemed necessary I could repeat the process on Linux, but probably not
on Windows.

Regarding code formatting, I think a style used consistently and
_automatically_ over the source code is much more important than a
particular style choice [*].

In 2007 Bo Peng added a uncrustify.cfg which _perhaps_ represents the
canonical "LyX coding style", but who knows. I will look at that for
guidance, and also at the existing code. However, sometimes I
might/should/ought to ask for consensus regarding a choice between
formatting styles. Especially as developers can have surprisingly strong
opinions here in my experience. So there's a risk of time consuming
discussions... but it's up to you to stay away ;-)

In my opinion, if we don't reach consensus easily on formatting issues, we
should at least for now refrain from using a .clang-format.

Also regarding formatting, the available versions of clang-format might
result in formatting restrictions [**].

With consensus on the format, and thus a different configuration file for
clang-format, I would repeat the check that globally applying clang-format
does not change the code. One reason for this is that I did notice that
changing the order of include files did result in slightly different
results (the difference might have been benign of course).

Best regards,
/Christian

[*] Having done a few diffs when testing with clang-format I do find the
use of TAB for indentation annoying. The reason is that sometimes the diff
output in my terminal isn't aligned properly, i.e. what in reality is
nicely aligned isn't aligned in my terminal. This was even after I
configured Git to invoke 'less' with a tab size of width 4. So for this
reason I'd actually prefer spaces to be used for indentation, but as tabs
are what's currently used, that's a strong reason to stay with tabs.

[**] I'll ask developers what version of clang-format they have available,
or could easily install. Even though later versions of clang-format have
some neat features, it's bad for us if the project's .clang-format cannot
be used by all developers, or if it doesn't produce the same result. This
matters for the developers what would actually be using clang-format.

On my mac, 'brew install clang-format' gave me ver. 4.0.0, but doing  'brew
upgrade clang-format' gave me version 5.0.0 which is the latest version.

Re: Deprecated functions used in some Objective C code

[Christian Ridderström]

> > PS. Should we create a Trac issue for this, to help remember we need to
> fix it in the future?
>
> I don’t need a ticket for it, the compiler is nagging every time I build
> LyX :)


It's fine by me. In case you wondered, some reasons I thought of creating
an issue:

- gives background, eg why we get the warning which someone else might
wonder about in the future

- is the warning "bad"? (answer: not at the moment, does not prevent
release)

- gives developers/release manager not on macOS info that the issue exists

- explains why we currently can't switch to the new functions etc

- ...

Anyway, all these aspects are (now) covered by this thread, so eg someone
googling the warning ought to find this.

Cheers,
Christian

Re: Deprecated functions used in some Objective C code

[Christian Ridderström]

On 8 July 2017 at 22:01, Stephan Witt  wrote:

> I’m aware of these warnings. Recently I tried to replace the deprecated
> calls in os_unix.cpp. But that’s not easy.
> 1. The replacements have to be available at least with 10.7 or we have to
> de-support old systems.
> 2. The replacements have to be provide the same functionality as the
> deprecated functions.
>
> The change d568846e0331adc9a879c68b00c7dff901692dc7 eliminated two
> deprecated function warnings.
> The change b5c2859a92b2e29c88156a82abc0b314265f761d reverts this because
> the new functions are available for 10.10 and newer only.
>

Ok. Well, we ought to be able to live with it until the functions actually
vanish in some future macOS version.

/Christian

PS. Should we create a Trac issue for this, to help remember we need to fix
it in the future?

Deprecated functions used in some Objective C code

[Christian Ridderström]

Hi,

While compiling on macOS Sierra, 10.12,5, I noticed warnings about
deprecated calls from an Objective C file. Below is a log excerpt, there
are probably more warnings.

What's the policy is here... should I e.g. create a Trac issue?

It's not obvious to me that we at this time would want to switch to
whatever is replacing the deprecated API.

Cheers,
/Christian


cd /Users/chrid/repos/lyx/m0/build-master2/src/support &&

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
  -DBOOST_USER_CONFIG=""
 -I/Users/chrid/repos/lyx/m0/build-master2 -I/Users/chrid/repos/lyx/m0/src
 -I/Users/chrid/repos/lyx/m0/3rdparty/boost
-I/Users/chrid/repos/lyx/m0/src/support
 -I/Users/chrid/repos/lyx/m0/build-master2/src/support
-I/usr/local/Cellar/qt/4.8.7_2/include
 -I/usr/local/Cellar/qt/4.8.7_2/lib/QtCore.framework/Headers
 -I/usr/local/Cellar/qt/4.8.7_2/mkspecs/default
-I/usr/local/Cellar/qt/4.8.7_2/include/QtCore
 -I/usr/local/Cellar/qt/4.8.7_2/include/QtGui
-I/usr/local/Cellar/qt/4.8.7_2/include/QtDesigner
 -I/usr/local/Cellar/qt/4.8.7_2/include/QtNetwork
-I/usr/local/Cellar/qt/4.8.7_2/include/QtOpenGL
 -I/usr/local/Cellar/qt/4.8.7_2/include/QtSql
-I/usr/local/Cellar/qt/4.8.7_2/include/QtXml
 -I/usr/local/Cellar/qt/4.8.7_2/include/QtSvg
-I/usr/local/Cellar/qt/4.8.7_2/include/QtUiTools
 -I/usr/local/Cellar/qt/4.8.7_2/include/QtTest
 -g   -F/usr/local/Cellar/qt/4.8.7_2/lib
 -std=c99
 -o CMakeFiles/support.dir/linkback/LinkBackServer.m.o
 -c
 /Users/chrid/repos/lyx/m0/src/support/linkback/LinkBackServer.m

/Users/chrid/repos/lyx/m0/src/support/linkback/LinkBackServer.m:175:11:
warning: 'NSRunCriticalAlertPanel' is deprecated: first deprecated in macOS
10.10 - Use NSAlert instead
  [-Wdeprecated-declarations]
result = NSRunCriticalAlertPanel(title, @"%@", ok, urlstr, nil,
msg) ;
 ^
/System/Library/Frameworks/AppKit.framework/Headers/NSPanel.h:42:25: note:
'NSRunCriticalAlertPanel' has been explicitly marked deprecated here
APPKIT_EXTERN NSInteger NSRunCriticalAlertPanel(NSString *title, NSString
*msgFormat, NSString *defaultButton, NSString *alternateButton, NSString
*otherButton, ...) NS_FORMAT_FUNCTION(2...
^
/Users/chrid/repos/lyx/m0/src/support/linkback/LinkBackServer.m:176:6:
warning: 'NSAlertAlternateReturn' is deprecated: first deprecated in macOS
10.10 - Use NSAlertFirstButtonReturn, etc
  instead [-Wdeprecated-declarations]
if (NSAlertAlternateReturn == result) {
^
/System/Library/Frameworks/AppKit.framework/Headers/NSPanel.h:72:5: note:
'NSAlertAlternateReturn' has been explicitly marked deprecated here
NSAlertAlternateReturn NS_ENUM_DEPRECATED_MAC(10_0, 10_10, "Use
NSAlertFirstButtonReturn, etc instead") = 0,
^
2 warnings generated.

Required C++ standard for building LyX, i.e. do we require >= C++11?

[Christian Ridderström]

Hi,

I was trying to confirm that compiling LyX requires at least C++11, but so
far I've only seen the following in INSTALL:

Requirements


First of all, you will need a recent C++ compiler, where recent means
that the compilers are close to C++11 standard conforming like gcc (at
least 4.6) or clang.

The text says "close to C++11", why don't we say that we require a compiler
conformant to C++11?

Is there some other place where we''re stating, or should be stating, the
requirements to build LyX?

Cheers
/Christian

PS. Background to my question.

If we require C++11, then C++03 template code like this (from LaTeX.cpp):

   stack  > child;

can now be reformatted as

   stack > child;

It's a minor thing and just reminded me to check what the requirements
really are.

Re: Cleanup before 2.3.0?

[Christian Ridderström]

Hi Scott,

On 6 July 2017 at 22:20, Scott Kostyshak  wrote:

> > - Now (before release) would be a good time to start using clang-format
>
> Why? One reason is that it might make it easier to backport fixes from
> master to 2.3.x.


This was my reason, as comparison of code becomes easier through 2.3.x.
Other than that we can certainly do it at a later stage.


> A reason against such a change is that the benefit is
> more long-run than short-run, as there is always the risk that unwanted
> functionality could change.


We should certainly consider the risk of changed functionality, but we are
actually quite safe:

- With one exception (see below) clang-format is designed to only change
   "whitespace" of the source code, i.e. adding/removing whitespace.

- We can compare the built executable before and after running clang-format,
  the executables should be identical.

- The one exception is if clang-format is allowed to sort the order of
"includes".
  But we disallow this (SortIncludes: false), so it won't happen.
  Note: Good source code should not depend on the order of the includes,
but ...

- Strictly speaking the behaviour of the program can change through a macro
  like __LINE__, if we have some crazy code that depends on a piece of
  code being located at some specific line number.  Otherwise we're talking
about
  that line numbers in a possible log output might be different. Just as if
someone
  had manually added or removed a few source code lines.

- If other programs read the LyX source code and depend on certain
  things being located on certain lines, or at certain indentation levels,
this
  could affect the overall functionality.
  So Doxygen would need to be checked that it still works - if we actually
use it for anything?


> Even if no functionality is changed in
> theory, we could somehow expose a compiler bug (this is true even if
> just changing whitespace).
>

I'd expect this to in practice be covered by binary comparison of the
executable.
In theory we could of course still be hit by a compiler bug on a platform
where we don't compare the executable.

Also, anyone doing the same whitespace change manually would trigger such a
bug...  in theory the removal of trailing whitespace from lines could
trigger it.

What do others think?


I'm attaching a file that should be saved as ".clang_format" in e.g. src/
or a in a higher directory.
This will then automatically be used by clang-format. To apply clang-format
to all files in src/, run:
  cd src
  clang-format -i *.cpp *.h

Note: The same would have to be done for source code in the subfolders as
well.

Note: From within Emacs you can do 'M-x clang-format-buffer'.

Note: I've tested with clang-format version 4.0. I don't know what the
minimum required version number is for the configuration settings I'm using.

Note: I created the attached .clang-format to try and minimise the number
of changed source code lines.
This configuration should be seen as a first step, to let people see what
we're talking about.
In particular, it's currently typically not fixing lines that are to long.

/Christian

PS.
As I see it, after we're using clang-format, a next step could be to use
'clang-tidy' to do some static analysis.


clang-format-custom
Description: Binary data

Re: Cleanup before 2.3.0?

[Christian Ridderström]

On 3 July 2017 at 11:26, Jean-Marc Lasgouttes  wrote:

> Since we are approaching major release, I think it is a good time to do
> some mechanical clean-ups. The idea is that it is better to do it now
> instead of at the beginning of a cycle in order to ease backporting of
> patches to stable.
>

I posted what's below in a different thread, but it's better of here:

-
If we're open to doing to some cleanup of the code I strongly suggest we
use a tool called 'clang-format' to automatically take care of code
formatting. I've introduced this tool at my work and it's _really_ nice to
not have to care about the formatting in most cases.  Tonight I'll create a
"format" file, ".clang-format", that attempts to replicate the formatting
style of the typical LyX code. If that goes well we can then discuss if it
makes sense to use/apply clang-format to the LyX source code. It might well
be that it results in too many changes for us to want to apply them.

Note: I have seen there's a config file for uncrustify in the repository,
but I got some error messages when trying to run uncrustify on the LyX
code.  Running clang-format on the LyX code works fine though. The
difficult part is tailoring the formatting rules to the current formatting
style to minimise the number of changes, and to avoid tedious discussions
about how the code ought to be formatted.

In case you're interested in clang-format, there's plenty of information
online. Chandler Carruth who leads the C++, Clang, and LLVM teams at
Google has a nice talk that includes a live demo of using clang-format in
this video: https://www.youtube.com/watch?v=cX_GhJ6BuWI. The demo starts as
about 29 minutes, but I'd suggest looking from 23 minutes. Fun detail:
Chandler references TeX and the issue of where to break lines, clang-format
needs to do something similar to the source code.
-

Expanding on the above.
- IMHO it's really nice to use clang-format
- Now (before release) would be a good time to start using clang-format
- It has integrations for various IDEs and editors, but can also be run
from the command line.
- It's some work to start using it
  - Trying to replicate the "nominal" formatting style that the old code
uses.
 I will make an attempt here tonight.
  - Realising that a lot of the old code didn't actually use your nominal
formatting style.

And something really important.  I think Scott, as release manager, has
absolute power to veto the use of clang-format before a release. Or to
limit the extent to which it is used. Especially if a never-ending
discussion on code formatting starts...

Cheers,
Christian

Re: Build failed in Jenkins: Build branch "master" » ubuntu-xenial-qt4-autotools-extended #296

[Christian Ridderström]

The failure (below) was due to the CI worker (lyx-linux1) running out of
disk space.
This CI worker only has 20 GB on the attached disk for building, and it's
on the small side - the other CI workers have 40 GB.
I've now deleted old workspaces and it should be fine again.
Cheers,
Christian

On 5 July 2017 at 20:08,  wrote:

> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-
> xenial-qt4-autotools-extended/296/--
> 
> [...truncated 2443 lines...]
>   CXX  buffer_funcs.o
>   CXX  BufferList.o
>   CXX  BufferParams.o
>   CXX  BufferView.o
>   CXX  Bullet.o
>   CXX  Changes.o
>   CXX  Chktex.o
>   CXX  CiteEnginesList.o
>   CXX  CmdDef.o
>   CXX  Color.o
>   CXX  ConverterCache.o
>   CXX  Converter.o
>   CXX  CoordCache.o
>   CXX  Counters.o
>   CXX  Cursor.o
>   CXX  CursorSlice.o
>   CXX  CutAndPaste.o
>   CXX  DepTable.o
>   CXX  DocIterator.o
>   CXX  Encoding.o
>   CXX  BufferEncodings.o
>   CXX  ErrorList.o
>   CXX  Exporter.o
>   CXX  factory.o
>   CXX  Floating.o
>   CXX  FloatList.o
>   CXX  FontInfo.o
>   CXX  FontList.o
>   CXX  Font.o
>   CXX  Format.o
>   CXX  FuncRequest.o
>   CXX  FuncStatus.o
>   CXX  Graph.o
>   CXX  IndicesList.o
>   CXX  InsetIterator.o
>   CXX  InsetList.o
>   CXX  Intl.o
>   CXX  KeyMap.o
>   CXX  KeySequence.o
>   CXX  Language.o
>   CXX  LaTeX.o
>   CXX  LaTeXFeatures.o
>   CXX  LaTeXPackages.o
>   CXX  LayoutFile.o
>   CXX  LayoutModuleList.o
>   CXX  Length.o
>   CXX  lengthcommon.o
>   CXX  Lexer.o
>   CXX  LyX.o
>   CXX  LyXAction.o
>   CXX  lyxfind.o
>   CXX  LyXRC.o
>   CXX  LyXVC.o
>   CXX  MetricsInfo.o
>   CXX  ModuleList.o
>   CXX  Mover.o
>   CXX  output_docbook.o
>   CXX  output.o
>   CXX  output_latex.o
>   CXX  output_xhtml.o
>   CXX  OutputParams.o
>   CXX  output_plaintext.o
>   CXX  Paragraph.o
>   CXX  ParagraphMetrics.o
>   CXX  ParagraphParameters.o
>   CXX  ParIterator.o
>   CXX  PDFOptions.o
>   CXX  Row.o
>   CXX  RowPainter.o
>   CXX  Server.o
>   CXX  ServerSocket.o
>   CXX  sgml.o
>   CXX  Session.o
>   CXX  Spacing.o
>   CXX  TexRow.o
>   CXX  texstream.o
>   CXX  Text.o
>   CXX  Text2.o
>   CXX  Text3.o
>   CXX  TextClass.o
>   CXX  TextMetrics.o
>   CXX  TocBackend.o
>   CXX  TocBuilder.o
>   CXX  Trans.o
>   CXX  Undo.o
>   CXX  VCBackend.o
>   CXX  version.o
>   CXX  VSpace.o
>   CXX  WordList.o
>   CXX  Layout.o
>   AR   liblyxcore.a
>   CXX  graphics/epstools.o
>   CXX  graphics/GraphicsCache.o
>   CXX  graphics/GraphicsCacheItem.o
>   CXX  graphics/GraphicsConverter.o
>   CXX  graphics/GraphicsLoader.o
>   CXX  graphics/GraphicsParams.o
>   CXX  graphics/PreviewImage.o
>   CXX  graphics/PreviewLoader.o
>   AR   liblyxgraphics.a
>   CXX  mathed/InsetMathAMSArray.o
>   CXX  mathed/InsetMathArray.o
>   CXX  mathed/InsetMathBig.o
>   CXX  mathed/InsetMathBoldSymbol.o
>   CXX  mathed/InsetMathBox.o
>   CXX  mathed/InsetMathBrace.o
>   CXX  mathed/InsetMath.o
>   CXX  mathed/InsetMathCancel.o
>   CXX  mathed/InsetMathCancelto.o
>   CXX  mathed/InsetMathCases.o
>   CXX  mathed/InsetMathChar.o
>   CXX  mathed/InsetMathClass.o
>   CXX  mathed/InsetMathColor.o
>   CXX  mathed/InsetMathCommand.o
>   CXX  mathed/InsetMathComment.o
>   CXX  mathed/InsetMathDecoration.o
>   CXX  mathed/InsetMathDelim.o
>   CXX  mathed/InsetMathDiff.o
>   CXX  mathed/InsetMathDots.o
>   CXX  mathed/InsetMathEnsureMath.o
>   CXX  mathed/InsetMathEnv.o
>   CXX  mathed/InsetMathExFunc.o
>   CXX  mathed/InsetMathExInt.o
>   CXX  mathed/InsetMathFont.o
>   CXX  mathed/InsetMathFontOld.o
>   CXX  mathed/InsetMathFrac.o
>   CXX  mathed/InsetMathGrid.o
>   CXX  mathed/InsetMathHull.o
>   CXX  mathed/InsetMathKern.o
>   CXX  mathed/InsetMathLefteqn.o
>   CXX  mathed/InsetMathLim.o
>   CXX  mathed/InsetMathMacro.o
>   CXX  mathed/InsetMathMacroArgument.o
>   CXX  mathed/InsetMathMacroTemplate.o
>   CXX  mathed/InsetMathMatrix.o
>   CXX  mathed/InsetMathNest.o
>   CXX  mathed/InsetMathNumber.o
>   CXX  mathed/InsetMathOverset.o
>   CXX  mathed/InsetMathPar.o
>   CXX  mathed/InsetMathPhantom.o
>   CXX  mathed/InsetMathRef.o
>   CXX  mathed/InsetMathRoot.o
>   CXX  mathed/InsetMathScript.o
>   CXX  mathed/InsetMathSideset.o
>   CXX  mathed/InsetMathSize.o
>   CXX  mathed/InsetMathSpace.o
>   CXX  mathed/InsetMathSpecialChar.o
>   CXX  mathed/InsetMathSplit.o
>   CXX  mathed/InsetMathSqrt.o
>   CXX  mathed/InsetMathStackrel.o
>  

Re: Build failed in Jenkins: Build branch "master" » ubuntu-latest-qt5-cmake #279

[Christian Ridderström]

The failure (below) was due to the CI worker (lyx-linux1) running out of
disk space.
This CI worker only has 20 GB on the attached disk for building, and it's
on the small side - the other CI workers have 40 GB.
I've now deleted old workspaces and it should be fine again.
Cheers,
Christian

On 6 July 2017 at 19:10,  wrote:

> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-
> latest-qt5-cmake/279/--
> Started by an SCM change
> Building remotely on lyx-linux1 (linux) in workspace <
> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-
> latest-qt5-cmake/ws/>
> [WS-CLEANUP] Deleting project workspace...
> [WS-CLEANUP] Done
> java.io.IOException: Failed to mkdirs:  build-master-head/job/ubuntu-latest-qt5-cmake/ws/>
> at hudson.FilePath.mkdirs(FilePath.java:1191)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:
> 1267)
> at hudson.model.AbstractBuild$AbstractBuildExecution.
> defaultCheckout(AbstractBuild.java:604)
> at jenkins.scm.SCMCheckoutStrategy.checkout(
> SCMCheckoutStrategy.java:86)
> at hudson.model.AbstractBuild$AbstractBuildExecution.run(
> AbstractBuild.java:529)
> at hudson.model.Run.execute(Run.java:1741)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(
> ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:410)
> Build does not meet criteria for workspace archiving - result is not at
> least SUCCESS.
>

Re: [LyX/master] Fix trailing whitespace in cpp files.

[Christian Ridderström]

On 3 July 2017 at 21:42, Scott Kostyshak  wrote:

> On Mon, Jul 03, 2017 at 07:53:56PM +0200, Richard Heck wrote:
> > commit 75bfed55079cab6b73fbea6ce4ae3f10d1af3b91
> > Author: Richard Heck 
> > Date:   Mon Jul 3 13:53:14 2017 -0400
> >
> > Fix trailing whitespace in cpp files.
>
>
Hi Richard,

I'm not sure how you removed trailing whitespace, but if it was done using
a tool that doesn't "know" C++, it could in theory change the code in a
highly unlikely corner case. Namely if the code is using a raw string
literal that occupies multiple lines in which a line ends with a trailing
space. E.g. like in this code:

const char *p = R"(a\
b
c)";  // Note that there are four spaces after the 'b' before the newline

assert(std::strcmp(p, "a\\\nb\nc") == 0);


I think it's _very_ unlikely we have this kind of corner case in the LyX
code, but it might anyway be good to be aware of it.
It's also good to know about raw string literals, especially when defining
strings with regular expressions since you won't have to escape backslashes.

As an aside, if we're open to doing to some cleanup of the code I strongly
suggest we use a tool called 'clang-format' to automatically take care of
code formatting. I've introduced this tool at my work and it's _really_
nice to not have to care about the formatting in most cases.  Tonight I'll
create a "format" file, ".clang-format", that attempts to replicate the
formatting style of the typical LyX code. If that goes well we can then
discuss if it makes sense to use/apply clang-format to the LyX source code.
It might well be that it results in too many changes for us to want to
apply them.

Note: I have seen there's a config file for uncrustify in the repository,
but I got some error messages when trying to run uncrustify on the LyX
code.  Running clang-format on the LyX code works fine though. The
difficult part is tailoring the formatting rules to the current formatting
style to minimise the number of changes, and to avoid tedious discussions
about how the code ought to be formatted.

In case you're interested in clang-format, there's plenty of information
online. Chandler Carruth who leads the C++, Clang, and LLVM teams at
Google has a nice talk that includes a live demo of using clang-format in
this video: https://www.youtube.com/watch?v=cX_GhJ6BuWI. The demo starts as
about 29 minutes, but I'd suggest looking from 23 minutes. Fun detail:
Chandler references TeX and the issue of where to break lines, clang-format
needs to do something similar to the source code.

Cheers,
Christian

Re: allowing anonymous contributions to LyX's source code?

[Christian Ridderström]

On 26 June 2017 at 10:31, Jean-Marc Lasgouttes  wrote:

>  From time to time, we receive patches from people who prefer to remain
>> anonymous (e.g. use a nickname and not their real name). It would be
>> nice if we had a clear policy on whether we are OK with this.
>>
>
> We cannot be sure that we know who is author of a patch. But we can have a
> " best effort" policy. We already have in the repo a contributor (venom00)
> who disclosed his real identity only privately. I think this is a good
> trade-off.
>

+1
/Christian

Re: Server rebooted

[Christian Ridderström]

Richard wrote:

> These look correct to me, but I usually use iptables via shorewall.
>
> That said, those are massive subnets to block. I guess if we're blocking
> legitimate users, we'll hear about it.
>

If the blocks seem to work I was thinking of announcing to users list the
blocked ranges, together with an explanation.

Separately, I was thinking of unblocking some of the ranges to see if the
server still works after a few days. We still don't know why we run low on
memory, ie if its related to Trac, a wiki or git or something else. So if
the server starts losing memory, I can see the IP of the culprit and then
perhaps figure out which of our services that has the memory leak.

Cheers,
Christian


>
> Richard
>
>

Re: Server rebooted

[Christian Ridderström]

On 19 June 2017 at 10:35, Jean-Marc Lasgouttes  wrote:

> Le 18/06/2017 à 16:29, Kornel Benko a écrit :
>
>> Server was out of memory. Stopping/starting apache seems to have resolved
>>> it.
>>>
>>
>> Yes, just tried 'git pull'. Immediate response.
>>
>
> I restarted it again since it was running on virtual memory. Sigh.
>

Do you mean it was using a lot of swap?


>  We have to do something.


I've activated some rules in IP-tables that blocks a few sets of chinese
IP-adresses which I've seen crawling over our server.
Let's see if this keeps the server happy. See below for the rules [1].

It might be that the blocks [1] are enough. I suggest we run the server
with these blocks in place for a while and see if it's still good after say
a week. As the server has recently needed a reboot every day or so, that
should be a strong indication that the issue is related to the subnets I
blocked.

Note: The rules for the IP-tables are _not_ saved permanently. If you
restart the server, the blocks are gone.


> There were talks about limiting the number of connection from a same
> source, or something.


IIRC, I did some tests/notes in order to limit connections from the same
source but I don't think I got all the way there.


> Another solution would be to use the honeypot's project http:BL API with
> something like mod_honeypot. But this is well beyond my skills.
>

Speaking of skills, I don't actually know how to use IP-tables, so if
anyone notices a problem with the rules etc [1], please let me know.


>
> http://www.projecthoneypot.org/services_overview.php
> http://www.miim.com/software/linux/modhoneypot.html
> http://www.projecthoneypot.org/httpbl_download.php
>
> I do not see many people advocating it, though. It might be that
> blacklisting by hand is effective enough for our needs.
>
> JMarc
>

/Christian

[1]

[chr@lyx iptables_backup_and_notes]$ cat config_iptables_2017-06-19.modified
# Created by CHR to block chinese IP-adresses, hoping it will fix issues
with server
# Install these rules e.g. as follows
#   cd ~lyx/iptables_backup_and_notes
#   sudo ./iptables_config.sh restore
~lyx/iptables_backup_and_notes/config_iptables_2017-06-19.modified
# Note: The command above does not save rules permanently.
#   If server is reset, the rules are note restored.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s 42.156.0.0/255.255.0.0 -j DROP
-A INPUT -s 42.120.0.0/255.255.0.0 -j DROP
-A INPUT -s 106.11.0.0/255.255.0.0 -j DROP
-A INPUT -s 106.38.0.0/255.255.0.0 -j DROP
-A INPUT -s 220.181.0.0/255.255.0.0 -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3690 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9418 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# End of rules.

Re: Server rebooted

[Christian Ridderström]

On 18 June 2017 at 14:40, Christian Ridderström <c...@lyx.org> wrote:

>
> On Sun, 18 Jun 2017 at 12:25, Kornel Benko <kor...@lyx.org> wrote:
>
>> Am Samstag, 17. Juni 2017 um 12:58:34, schrieb Jean-Marc Lasgouttes <
>> lasgout...@lyx.org>
>> > Le 17/06/2017 à 12:57, Christian Ridderström a écrit :
>> > > Good decision.
>> > >
>> > We have to find a way to get rid of these chinese robots...
>>
>> Slow again here.
>
>
> I'll be home in an hour and can then login and see if I notice anything
> odd. Perhaps enable IP-tables. Otherwise I'll try a reboot again.
>

Server was out of memory. Stopping/starting apache seems to have resolved
it.
Haven't looked at IP-tables yet, but after the reboot there shouldn't be
any special blocking going on - that currently needs to be added manually.
I can do that later tonight.
/Christian

Re: Server rebooted

[Christian Ridderström]

On Sun, 18 Jun 2017 at 12:25, Kornel Benko <kor...@lyx.org> wrote:

> Am Samstag, 17. Juni 2017 um 12:58:34, schrieb Jean-Marc Lasgouttes <
> lasgout...@lyx.org>
> > Le 17/06/2017 à 12:57, Christian Ridderström a écrit :
> > > Good decision.
> > >
> > We have to find a way to get rid of these chinese robots...
>
> Slow again here.


I'll be home in an hour and can then login and see if I notice anything
odd. Perhaps enable IP-tables. Otherwise I'll try a reboot again.

Christian





>
> Kornel

FYI, recently failed builds were due to slow LyX (git) server. (Was: Build failed in Jenkins: Build branch "master" » ubuntu-xenial-qt4-autotools-extended #276

[Christian Ridderström]

FYI, this CI job (and others) failed because the LyX server was slow (JMarc
has fixed it by rebooting the server). This can be seen from the full log
of the CI job:



Cloning repository git://git.lyx.org/lyx.git

 > git init 
 > /builds/workspace/build-master-head/ubuntu-xenial-qt4-autotools-extended
# timeout=10

Fetching upstream changes from git://git.lyx.org/lyx.git

 > git --version # timeout=10

 > git fetch --no-tags --progress git://git.lyx.org/lyx.git
+refs/heads/*:refs/remotes/origin/* --depth=1

ERROR: Timeout after 10 minutes

ERROR : Error
cloning remote repo 'origin'

hudson.plugins.git.GitException
:
Command "git fetch --no-tags --progress git://git.lyx.org/lyx.git
+refs/heads/*:refs/remotes/origin/* --depth=1" returned status code
143:


Unfortunately the truncated log sent to this list doesn't include that
initial part. Instead we see part of the progress, which looks like this:

remote: Compressing objects:  94% (21243/22585)
remote: Compressing objects:  94% (21255/22585)
remote: Compressing objects:  94% (21262/22585)
remote: Compressing objects:  94% (21268/22585)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.
launchCommandIn(CliGitAPIImpl.java:1772)

at hudson.model.Executor.run(Executor.java:410)
ERROR: null


Interpretation: The CI job had spent ten minutes waiting for the LyX (git)
server to compressing 94% and then timed out.

/Christian

On 17 June 2017 at 11:35,  wrote:

> https://ci.inria.fr/lyx/job/build-master-head/job/ubuntu-
> xenial-qt4-autotools-extended/276/--
> 

Re: Asked for username/password when going to ftp.lyx.org from www.lyx.org/Download. (Was: can I get a username and password to download lyx?(

[Christian Ridderström]

On 18 May 2017 at 09:09, Stephan Witt  wrote:

> For me - on Mac - it has always been like this. It’s not a big deal IMO to
> choose anonymous and proceed. Therefore I didn’t complain :)
> OTOH how would you connect in case you want using a name/password if
> you’re not asked for?
>

In Fan's case, he seems to be on Firefox / Windows and without a dialog
option to login in anonymously, so I can understand why it's confusing for
him.

Scott, which OS are you on?  (you tested with Chromium and Firefox, and
didn't have a problem).

In summary we seem to have:

Windows? / Firefox   (Fan): Only asked for user/password, no anonymous
option
?/Chromium  (Scott):Automatically logged in anonymously.
? / Firefox (Scott):Automatically logged in anonymously.
macOS  / Chrome(CHR):Automatically logged in anonymously.
macOS / Safari(CHR): Dialog with choice to login as guest.
   Assuming Stephan's also on
Mac/Safari.

So at least on macOS it does depend on the browser.

/Christian

PS
Fan, did it work for you if you gave "anonymous" as the username?

Asked for username/password when going to ftp.lyx.org from www.lyx.org/Download. (Was: can I get a username and password to download lyx?(

[Christian Ridderström]

On 17 May 2017 at 23:02, Christian Ridderström <c...@lyx.org> wrote:

> On 17 May 2017 at 22:47, Fan Zhang <fan.zh...@skhms.com> wrote:
>
>> I was asked for a username and password from ftp://ftp.lyx.org
>>
>> I don’t know how I can get that. Can you please help?
>>
>
> Hi Fan,
>
> Thanks for letting us know you still have the problem.
>

Can anyone say if what's described below (regarding name/password) is a new
behaviour, or if it's always been like this?

/Christian


> I think I was able to reproduce your experience/problem when trying to
> download LyX by doing the following:
>
> * Browse to https://www.lyx.org/Download
> * Click on the link that refers to FTP site
> <ftp://ftp.lyx.org/pub/lyx/bin/2.2.3>  ( ftp://ftp.lyx.org/pub/lyx/
> bin/2.2.3 )
>
> I was then asked for a user name / password, just like you say.
> This is of course confusing, so I'm CC:ing this e-mail to the developers'
> list for further discussion even thought there might not actually be
> anything we can do about it directly.
>
> However, what's helpful for you to know is that you can log in to the FTP
> site _anonymously_.
>
> I'm currently on macOS, so in my case when I click the FTP-link, Safari
> asks me if I want to open the address in "Finder" (like Explorer on
> Windows). That's when I'm asked for a name and password. However, at this
> point I can select to connect as "Guest", and I then don't need to enter
> any name or password.
>
> How you should do this depends on from what platform you're connecting and
> what FTP-client you're using, but the bottom line is that you should be
> able to log in anonymously / guest. I have a vague memory that you can just
> give the name as "anonymous" and simply give e.g. your e-mail address as a
> password, and you'll get in.
>