Re: Server rebooted

[Richard Heck]

On 06/19/2017 02:03 PM, Christian Ridderström wrote:
> On 19 June 2017 at 10:35, Jean-Marc Lasgouttes  > wrote:
>
> Le 18/06/2017 à 16:29, Kornel Benko a écrit :
>
> Server was out of memory. Stopping/starting apache seems
> to have resolved
> it.
>
>
> Yes, just tried 'git pull'. Immediate response.
>
>
> I restarted it again since it was running on virtual memory. Sigh.
>
>
> Do you mean it was using a lot of swap?
>  
>
>  We have to do something. 
>
>
> I've activated some rules in IP-tables that blocks a few sets of
> chinese IP-adresses which I've seen crawling over our server.
> Let's see if this keeps the server happy. See below for the rules [1].
>
> It might be that the blocks [1] are enough. I suggest we run the
> server with these blocks in place for a while and see if it's still
> good after say a week. As the server has recently needed a reboot
> every day or so, that should be a strong indication that the issue is
> related to the subnets I blocked.
>
> Note: The rules for the IP-tables are _not_ saved permanently. If you
> restart the server, the blocks are gone.
>  
>
> There were talks about limiting the number of connection from a
> same source, or something. 
>
>
> IIRC, I did some tests/notes in order to limit connections from the
> same source but I don't think I got all the way there.
>  
>
> Another solution would be to use the honeypot's project http:BL
> API with something like mod_honeypot. But this is well beyond my
> skills.
>
>
> Speaking of skills, I don't actually know how to use IP-tables, so if
> anyone notices a problem with the rules etc [1], please let me know.
>  
>
>
> http://www.projecthoneypot.org/services_overview.php
> 
> http://www.miim.com/software/linux/modhoneypot.html
> 
> http://www.projecthoneypot.org/httpbl_download.php
> 
>
> I do not see many people advocating it, though. It might be that
> blacklisting by hand is effective enough for our needs.
>
> JMarc
>
>
> /Christian 
>
> [1]
>
> [chr@lyx iptables_backup_and_notes]$ cat
> config_iptables_2017-06-19.modified
> # Created by CHR to block chinese IP-adresses, hoping it will fix
> issues with server
> # Install these rules e.g. as follows
> #   cd ~lyx/iptables_backup_and_notes
> #   sudo ./iptables_config.sh restore
> ~lyx/iptables_backup_and_notes/config_iptables_2017-06-19.modified
> # Note: The command above does not save rules permanently.
> #   If server is reset, the rules are note restored.
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> -A INPUT -s 42.156.0.0/255.255.0.0  -j DROP
> -A INPUT -s 42.120.0.0/255.255.0.0  -j DROP
> -A INPUT -s 106.11.0.0/255.255.0.0  -j DROP
> -A INPUT -s 106.38.0.0/255.255.0.0  -j DROP
> -A INPUT -s 220.181.0.0/255.255.0.0 
> -j DROP
> -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
> -A INPUT -p icmp -j ACCEPT 
> -A INPUT -i lo -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 3690 -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 9418 -j ACCEPT 
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
> -A INPUT -j REJECT --reject-with icmp-host-prohibited 
> -A FORWARD -j REJECT --reject-with icmp-host-prohibited 
> COMMIT
> # End of rules.

These look correct to me, but I usually use iptables via shorewall.

That said, those are massive subnets to block. I guess if we're blocking
legitimate users, we'll hear about it.

Richard

Re: [LyX/master] Use otexstringstream for the captions of InsetCaptionables

[Enrico Forestieri]

On Tue, Jun 20, 2017 at 02:58:17AM +0200, Guillaume MM wrote:

> Le 11/06/2017 à 01:08, Enrico Forestieri a écrit :
> > On Mon, Oct 17, 2016 at 00:16:21AM +0200, Guillaume Munch wrote:
> > 
> > > commit 676a0639c505d52336e3228c44c2515ccbbaf34a
> > > Author: Guillaume Munch 
> > > Date:   Sat Sep 24 00:49:00 2016 +0200
> > > 
> > >  Use otexstringstream for the captions of InsetCaptionables
> > >  * Enable TexRow for InsetListings caption.
> > >  * Move getCaption* from InsetText to InsetCaptionable.
> > >  * Clean-up caption generation for InsetFloat.
> > [...]
> > > @@ -420,7 +417,11 @@ docstring InsetListings::getCaption(OutputParams 
> > > const & runparams) const
> > >   // NOTE that } is not allowed in blah2.
> > >   regex const reg("(.*)label\\{(.*?)\\}(.*)");
> > >   string const new_cap("$1$3},label={$2");
> > > - return from_utf8(regex_replace(to_utf8(cap), reg, new_cap));
> > > + // TexString validity: the substitution preserves the number of 
> > > newlines.
> > > + // Moreover we assume that $2 does not contain newlines, so that the 
> > > texrow
> > > + // information remains accurate.
> > > + cap.str = from_utf8(regex_replace(to_utf8(cap.str), reg, new_cap));
> > > + return cap;
> > >   }
> > 
> > This commit broke the caption handling in InsetListings. For example,
> > the document attached at #9382:
> > https://www.lyx.org/trac/raw-attachment/ticket/9382/problem.20_nopreamble.lyx
> > now fails to compile. It can be compiled again with the attached patch,
> > which is not the right one, of course.
> > 
> 
> 
> Dear Enrico,
> 
> 
> I had a look at this issue 8 days ago since you pointed at my commit. I
> realised then that my commit did nothing to worsen or improve this
> issue (the commit is meant to leave the output unchanged, and the bug
> appears in LyX 2.2.2 where this line of commits has never been). Sorry
> for the wait: I did not have much time on my hands and it was not a
> regression for me so I decided to postpone my reply until I had more time.
> 
> I now see that you are proposing a patch at
> . I have several questions about
> your patch.
> 
> Given that you had located a regression (you even point to a specific
> part of the code), have you tried to see what was causing the regression
> in the commit while coming up with a patch? I think that trying to do so
> would show you that the commit you are pointing to actually leaves the
> output unchanged.
> 
> In your bug report you notice that the bug is present in stable and you
> say to Scott that the patch is valid for stable. It does not apply. Why
> did you let Scott think that you have tested your patch on stable? I
> think that if you had, then you could have noticed that the commit was
> never backported.
> 
> As you know, you are writing to a developers list where people are
> willing to put in their valuable time to help you, and expect accurate
> technical informations. It happens to make a mistake, but it does not
> make sense to me that with enough care you did not find an occasion to
> correct yourself.
> 
> I want to be able to trust technical claims that are made on the list in
> the future, and so do other developers probably. I try to always
> double-check technical claims I make and sometimes it helps me catch
> errors at the last moment. I would like to be sure that we agree on
> everybody putting in the same care in their contributions.

Yes, you are right.

-- 
Enrico

Re: [LyX/master] Fix bug #9101

[Enrico Forestieri]

On Tue, Jun 20, 2017 at 02:45:31AM +0200, Guillaume MM wrote:

> Le 08/06/2017 à 04:28, Enrico Forestieri a écrit :
> > 
> > Actually it was simpler than expected.
> > 
> 
> Thanks.

Yes, you are right.

-- 
Enrico

Re: [LyX/2.2.x] Fix bugs #9598 and #10650

[Enrico Forestieri]

On Tue, Jun 20, 2017 at 02:45:19AM +0200, Guillaume MM wrote:

> Le 08/06/2017 à 02:07, Enrico Forestieri a écrit :
> > On Thu, Jun 08, 2017 at 12:50:19AM +0200, Guillaume MM wrote:
> > 
> > > Le 05/06/2017 à 23:15, Enrico Forestieri a écrit :
> > > > commit 59c22bd7b604a3ba9e0e78f7c51cb601f08d0192
> > > > Author: Enrico Forestieri
> > > > Date:   Mon Jun 5 23:14:48 2017 +0200
> > > > 
> > > >   Fix bugs #9598 and #10650
> > > > ---
> > > 
> > > > +// gcc < 4.8.0 and msvc < 2015 do not support C++11 thread_local
> > > > +#if defined(__GNUC__) && ((__GNUC__ == 4 && __GNUC_MINOR__ < 8) || 
> > > > __cplusplus < 201103L)
> > > > +#define THREAD_LOCAL_STATIC static __thread
> > > > +#elif defined(_MSC_VER) && ((_MSC_VER < 1900) || __cplusplus < 201103L)
> > > > +#define THREAD_LOCAL_STATIC static __declspec(thread)
> > > > +#else
> > > > +#define THREAD_LOCAL_STATIC thread_local static
> > > > +#endif
> > > > +
> > > 
> > > 
> > > According to Stephan in this discussion:
> > > 
> > > https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg196176.html
> > > 
> > > it is unfortunately not possible to use thread_local on Mac before some
> > > time, it seems. I would actually be happy to hear the contrary.
> > 
> > Note that I also added the check for __cplusplus < 201103L, which should
> > assure full compliance with C++11.
> 
> Only in a perfect world. Note also that for clang you are always taking
> the first branch so the value of __cplusplus is irrelevant. If
> intentional, I suggest that you make it clear in the comment.
> 
> > If thread_local is not supported, the
> > compiler should not set __cplusplus to such value or higher and thus
> > should use the fallback definition.
> 
> Sorry, this does not make sense to me. First, all of your definitions
> use some form of thread-local storage. The problem referred to above is
> that there is no implementation of it at all, even called __thread or
> something else. You do not acknowledge that there is an issue, so it
> is not clear to me that you have read the discussion mentioned above
> carefully. To the best of our knowledge, in the best case your current
> patch requires a discussion about dropping the support of Xcode < 8.
> 
> 
> > 
> > > (Also, it had been decided that LyX requires msvc ≥ 2015 so the second
> > > branch would not be necessary.)
> > 
> > I don't think that LyX really cannot be compiled with older versions.
> > Are there reports in this regard?
> > 
> 
> There was a discussion at
> 
> and parent messages, and maybe someplace else as well. Since my patch on
> Unicode string literals alluded to in that message is not going to be
> committed before 2.4 at least, it could make sense to support MSVC 2013
> in 2.3 if it currently works. What worries me most is lack of support
> for 
> ("magic statics") for which your criteria “does it compile?” is not
> going to be enough.

Yes, you are right.

-- 
Enrico

Re: Can shell-escape take advantage of needauth framework?

[Enrico Forestieri]

On Tue, Jun 20, 2017 at 02:43:58AM +0200, Guillaume MM wrote:

> Le 19/06/2017 à 15:39, Enrico Forestieri a écrit :
> > On Mon, Jun 19, 2017 at 06:39:22AM +0200, Jürgen Spitzmüller wrote:
> > 
> > > Am Sonntag, den 18.06.2017, 19:56 +0200 schrieb Enrico Forestieri:
> > > > > I think we need to provide an option to add -shell-escape only to
> > > > > specific documents and only on the given machine. This prevents
> > > > > sending
> > > > > documents with -shell-escape (main problem of a document setting).
> > > > 
> > > > This is contradictory. We avoid sending documents with -shell-escape
> > > > but then add it to specific documents. So, it is the same thing.
> > > 
> > > No, it isn't. I didn't propose a document property, but a per-document
> > > session setting. This is a completely different thing.
> > 
> > Sorry, it was not clear to me what you meant. Here is a patch following
> > this strategy.
> > 
> > - We never store in the document the need for -shell-escape.
> > - When the user checks the toolbar button and then runs a latex backend,
> >he is alerted that the backend will be allowed to run external programs.
> > - At this point, he can decide to let the backend run (and be asked again
> >next time), or to always allow execution with -shell-escape for this doc.
> > - If the user chooses to always allow -shell-escape for the current 
> > document,
> >the document path is stored in the session file, so that next time it is
> >loaded on the current machine, the toolbar button will be automatically
> >toggled and no question will be asked.
> > - If the user manually toggles the toolbar button so that to disallow the
> >-shell-escape option for an authorized document, the document is
> >automatically removed from the list of authorized documents.
> > 
> > This patch does not introduce a format change, because nothing is recorded
> > in the document (the document status is only recorded in the session file).
> > 
> 
> 
> If I understand correctly, this is the latest proposal for hard-wiring
> the "-shell-escape" option when running the child latex processes, so I
> will comment on this one. But I could write almost the same for all the
> other proposals I have read so far.
> 
> Let me summarize.
> 
> Pygments: a python software that does some valuable processing on
> certain text inputs.
> 
> minted.sty: a LaTeX interface for Pygments. For some convenience reason,
> minted.sty calls Pygments using the \write18 command. \write18 lets one
> execute arbitrary commands on the console so it is disabled by default.
> The -shell-escape option is necessary for using minted, it enables
> \write18 but then \write18 can be used anywhere in the document.
> 
> Enrico's patch: add the -shell-escape option when some conditions are
> met, overriding the values shown to the user in the converter
> preferences. The idea is that the user has the ability to "trust" a
> document (roughly a different name for needauth, with small differences
> in the details such as the ability to revoke the trust given, which is
> an improvement compared to needauth).
> 
> First I note that minted.sty executes an external tool contrary to the
> LaTeX tradition. External tools are used all the time, but usually the
> user is responsible for calling the external tools themselves before
> calling LaTeX again. And the small convenience gain of doing it the
> minted way is irrelevant to LyX users given that LyX is capable of
> automating the process of calling Pygments. Here I tell you nothing new
> about LaTeX and LyX. And in fact, there already exists an interface to
> Pygments that works the LaTeX way: pygmentex.sty. (There also were
> difficulties in guessing whether minted.sty will be able to find
> Pygments, this is also no longer an issue if LyX is in charge of calling
> Pygments.)
> 
> Second, the design seems to be based on elaborate assumptions about the
> user's usage and their behaviour. I would like to see the arguments
> being are in principles of secure usability, which is a topic of
> academic research with available articles and textbooks. See e.g.:
> 
> https://dl.acm.org/citation.cfm?id=687663
> http://shop.oreilly.com/product/9780596008277.do
> 
> At a cursory glance, the proposed mechanism violates several principles:
> 
> * Prompting the user to give up security before anything happens. This
> is equivalent to having no dialog at all the second or third time it
> appears if the user depends on it, because they will only think about it
> the first time if at all (think of "security exceptions" in your browser),
> * Running child processes with more privileges than necessary,
> * Forcing all-or-nothing choices (e.g. one needs to trust the whole
> document instead of just minted),
> * What is trusted can change over time.
> 
> I am further less convinced because the relationship between usability
> issues and security has already been pointed out by Helge and I would
> have like to see his points 

Re: dash patch for stable

[Enrico Forestieri]

On Tue, Jun 20, 2017 at 02:59:00AM +0200, Guillaume MM wrote:

> Le 03/06/2017 à 01:08, Enrico Forestieri a écrit :
> > On Fri, Jun 02, 2017 at 07:30:34PM +, Guenter Milde wrote:
> > > 
> > > Note that the "ERT patch" is only for *stable*, i.e. 2.2.x.
> > > For 2.2, ERT is the only way to ensure full backwards compatibility.
> > 
> > A solution should have been thought and done before 2.2.0 was released.
> > Now it's too late.
> > 
> 
> Enrico, I also have to criticise this sort of comment of yours. I am
> sorry to say that to me it looks disconnected with the understanding of
> the problem at the time and gratuitous.

Yes, you are right.

-- 
Enrico

Re: [LyX/master] Use otexstringstream for the captions of InsetCaptionables

[Guillaume MM]

Le 11/06/2017 à 01:08, Enrico Forestieri a écrit :

On Mon, Oct 17, 2016 at 00:16:21AM +0200, Guillaume Munch wrote:


commit 676a0639c505d52336e3228c44c2515ccbbaf34a
Author: Guillaume Munch 
Date:   Sat Sep 24 00:49:00 2016 +0200

 Use otexstringstream for the captions of InsetCaptionables
 
 * Enable TexRow for InsetListings caption.
 
 * Move getCaption* from InsetText to InsetCaptionable.
 
 * Clean-up caption generation for InsetFloat.

[...]

@@ -420,7 +417,11 @@ docstring InsetListings::getCaption(OutputParams const & 
runparams) const
// NOTE that } is not allowed in blah2.
regex const reg("(.*)label\\{(.*?)\\}(.*)");
string const new_cap("$1$3},label={$2");
-   return from_utf8(regex_replace(to_utf8(cap), reg, new_cap));
+   // TexString validity: the substitution preserves the number of 
newlines.
+   // Moreover we assume that $2 does not contain newlines, so that the 
texrow
+   // information remains accurate.
+   cap.str = from_utf8(regex_replace(to_utf8(cap.str), reg, new_cap));
+   return cap;
  }
  
  


This commit broke the caption handling in InsetListings. For example,
the document attached at #9382:
https://www.lyx.org/trac/raw-attachment/ticket/9382/problem.20_nopreamble.lyx
now fails to compile. It can be compiled again with the attached patch,
which is not the right one, of course.




Dear Enrico,


I had a look at this issue 8 days ago since you pointed at my commit. I
realised then that my commit did nothing to worsen or improve this
issue (the commit is meant to leave the output unchanged, and the bug
appears in LyX 2.2.2 where this line of commits has never been). Sorry
for the wait: I did not have much time on my hands and it was not a
regression for me so I decided to postpone my reply until I had more time.

I now see that you are proposing a patch at
. I have several questions about
your patch.

Given that you had located a regression (you even point to a specific
part of the code), have you tried to see what was causing the regression
in the commit while coming up with a patch? I think that trying to do so
would show you that the commit you are pointing to actually leaves the
output unchanged.

In your bug report you notice that the bug is present in stable and you
say to Scott that the patch is valid for stable. It does not apply. Why
did you let Scott think that you have tested your patch on stable? I
think that if you had, then you could have noticed that the commit was
never backported.

As you know, you are writing to a developers list where people are
willing to put in their valuable time to help you, and expect accurate
technical informations. It happens to make a mistake, but it does not
make sense to me that with enough care you did not find an occasion to
correct yourself.

I want to be able to trust technical claims that are made on the list in
the future, and so do other developers probably. I try to always
double-check technical claims I make and sometimes it helps me catch
errors at the last moment. I would like to be sure that we agree on
everybody putting in the same care in their contributions.


Guillaume

Re: dash patch for stable

[Guillaume MM]

Le 03/06/2017 à 01:08, Enrico Forestieri a écrit :

On Fri, Jun 02, 2017 at 07:30:34PM +, Guenter Milde wrote:


Note that the "ERT patch" is only for *stable*, i.e. 2.2.x.
For 2.2, ERT is the only way to ensure full backwards compatibility.


A solution should have been thought and done before 2.2.0 was released.
Now it's too late.



Enrico, I also have to criticise this sort of comment of yours. I am
sorry to say that to me it looks disconnected with the understanding of
the problem at the time and gratuitous.

Guillaume

Re: [LyX/master] Fix bug #9101

[Guillaume MM]

Le 08/06/2017 à 04:28, Enrico Forestieri a écrit :


Actually it was simpler than expected.



Thanks.

Re: [LyX/2.2.x] Fix bugs #9598 and #10650

[Guillaume MM]

Le 08/06/2017 à 02:07, Enrico Forestieri a écrit :

On Thu, Jun 08, 2017 at 12:50:19AM +0200, Guillaume MM wrote:


Le 05/06/2017 à 23:15, Enrico Forestieri a écrit :

commit 59c22bd7b604a3ba9e0e78f7c51cb601f08d0192
Author: Enrico Forestieri
Date:   Mon Jun 5 23:14:48 2017 +0200

  Fix bugs #9598 and #10650
---



+// gcc < 4.8.0 and msvc < 2015 do not support C++11 thread_local
+#if defined(__GNUC__) && ((__GNUC__ == 4 && __GNUC_MINOR__ < 8) || __cplusplus 
< 201103L)
+#define THREAD_LOCAL_STATIC static __thread
+#elif defined(_MSC_VER) && ((_MSC_VER < 1900) || __cplusplus < 201103L)
+#define THREAD_LOCAL_STATIC static __declspec(thread)
+#else
+#define THREAD_LOCAL_STATIC thread_local static
+#endif
+



According to Stephan in this discussion:

https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg196176.html

it is unfortunately not possible to use thread_local on Mac before some
time, it seems. I would actually be happy to hear the contrary.


Note that I also added the check for __cplusplus < 201103L, which should
assure full compliance with C++11.


Only in a perfect world. Note also that for clang you are always taking
the first branch so the value of __cplusplus is irrelevant. If
intentional, I suggest that you make it clear in the comment.


If thread_local is not supported, the
compiler should not set __cplusplus to such value or higher and thus
should use the fallback definition.


Sorry, this does not make sense to me. First, all of your definitions
use some form of thread-local storage. The problem referred to above is
that there is no implementation of it at all, even called __thread or
something else. You do not acknowledge that there is an issue, so it
is not clear to me that you have read the discussion mentioned above
carefully. To the best of our knowledge, in the best case your current
patch requires a discussion about dropping the support of Xcode < 8.





(Also, it had been decided that LyX requires msvc ≥ 2015 so the second
branch would not be necessary.)


I don't think that LyX really cannot be compiled with older versions.
Are there reports in this regard?



There was a discussion at

and parent messages, and maybe someplace else as well. Since my patch on
Unicode string literals alluded to in that message is not going to be
committed before 2.4 at least, it could make sense to support MSVC 2013
in 2.3 if it currently works. What worries me most is lack of support
for 
("magic statics") for which your criteria “does it compile?” is not
going to be enough.

Re: Can shell-escape take advantage of needauth framework?

[Guillaume MM]

Le 19/06/2017 à 15:39, Enrico Forestieri a écrit :

On Mon, Jun 19, 2017 at 06:39:22AM +0200, Jürgen Spitzmüller wrote:


Am Sonntag, den 18.06.2017, 19:56 +0200 schrieb Enrico Forestieri:

I think we need to provide an option to add -shell-escape only to
specific documents and only on the given machine. This prevents
sending
documents with -shell-escape (main problem of a document setting).


This is contradictory. We avoid sending documents with -shell-escape
but then add it to specific documents. So, it is the same thing.


No, it isn't. I didn't propose a document property, but a per-document
session setting. This is a completely different thing.


Sorry, it was not clear to me what you meant. Here is a patch following
this strategy.

- We never store in the document the need for -shell-escape.
- When the user checks the toolbar button and then runs a latex backend,
   he is alerted that the backend will be allowed to run external programs.
- At this point, he can decide to let the backend run (and be asked again
   next time), or to always allow execution with -shell-escape for this doc.
- If the user chooses to always allow -shell-escape for the current document,
   the document path is stored in the session file, so that next time it is
   loaded on the current machine, the toolbar button will be automatically
   toggled and no question will be asked.
- If the user manually toggles the toolbar button so that to disallow the
   -shell-escape option for an authorized document, the document is
   automatically removed from the list of authorized documents.

This patch does not introduce a format change, because nothing is recorded
in the document (the document status is only recorded in the session file).




If I understand correctly, this is the latest proposal for hard-wiring
the "-shell-escape" option when running the child latex processes, so I
will comment on this one. But I could write almost the same for all the
other proposals I have read so far.

Let me summarize.

Pygments: a python software that does some valuable processing on
certain text inputs.

minted.sty: a LaTeX interface for Pygments. For some convenience reason,
minted.sty calls Pygments using the \write18 command. \write18 lets one
execute arbitrary commands on the console so it is disabled by default.
The -shell-escape option is necessary for using minted, it enables
\write18 but then \write18 can be used anywhere in the document.

Enrico's patch: add the -shell-escape option when some conditions are
met, overriding the values shown to the user in the converter
preferences. The idea is that the user has the ability to "trust" a
document (roughly a different name for needauth, with small differences
in the details such as the ability to revoke the trust given, which is
an improvement compared to needauth).

First I note that minted.sty executes an external tool contrary to the
LaTeX tradition. External tools are used all the time, but usually the
user is responsible for calling the external tools themselves before
calling LaTeX again. And the small convenience gain of doing it the
minted way is irrelevant to LyX users given that LyX is capable of
automating the process of calling Pygments. Here I tell you nothing new
about LaTeX and LyX. And in fact, there already exists an interface to
Pygments that works the LaTeX way: pygmentex.sty. (There also were
difficulties in guessing whether minted.sty will be able to find
Pygments, this is also no longer an issue if LyX is in charge of calling
Pygments.)

Second, the design seems to be based on elaborate assumptions about the
user's usage and their behaviour. I would like to see the arguments
being are in principles of secure usability, which is a topic of
academic research with available articles and textbooks. See e.g.:

https://dl.acm.org/citation.cfm?id=687663
http://shop.oreilly.com/product/9780596008277.do

At a cursory glance, the proposed mechanism violates several principles:

* Prompting the user to give up security before anything happens. This
is equivalent to having no dialog at all the second or third time it
appears if the user depends on it, because they will only think about it
the first time if at all (think of "security exceptions" in your browser),
* Running child processes with more privileges than necessary,
* Forcing all-or-nothing choices (e.g. one needs to trust the whole
document instead of just minted),
* What is trusted can change over time.

I am further less convinced because the relationship between usability
issues and security has already been pointed out by Helge and I would
have like to see his points being taken into account in the discussion.

Until these needauth mechanisms (or whatever they are called) are
designed with the needs of users in mind following established
principles, their purpose will be for the developer to shift blame to
the user in case something bad happens. (For developers they also make
it safer to open random lyx files from 

Re: Can shell-escape take advantage of needauth framework?

[Enrico Forestieri]

On Mon, Jun 19, 2017 at 09:00:33PM +0200, Enrico Forestieri wrote:

> On Mon, Jun 19, 2017 at 08:57:00PM +0200, Jürgen Spitzmüller wrote:
> 
> > Am Montag, den 19.06.2017, 20:33 +0200 schrieb Enrico Forestieri:
> > > Because we don't know whether it's needed?
> > 
> > Why not? Can't we define that?
> 
> Are you able to tell that you need -shell-escape for the attached
> document?

Now attached.

-- 
Enrico
#LyX 2.2 created this file. For more info see http://www.lyx.org/
\lyxformat 508
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\begin_preamble
\usepackage[miktex,siunitx]{gnuplottex}
\usepackage{siunitx}

% Using pdflatex, the produced eps file is automatically converted
% to pdf using epstopdf by default. Unfortunately, the cairolatex
% terminal of gnuplot inserts a couple comment lines just before
% the line "%!PS-Adobe-3.0 EPSF-3.0" that characterizes an eps file.
% This confuses the epstopdf program that comes with MikTeX, which
% issues a "Invalid binary DOS header" error and produces an empty
% pdf file. The solution is using ps2pdf.
\usepackage{epstopdf}
\epstopdfDeclareGraphicsRule{.eps}{pdf}{.pdf}{%
ps2pdf -dEPSCrop #1 \OutputFile
}
\end_preamble
\use_default_options true
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding auto
\fontencoding global
\font_roman "default" "default"
\font_sans "default" "default"
\font_typewriter "default" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\graphics default
\default_output_format pdf2
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize 12
\spacing single
\use_hyperref false
\papersize default
\use_geometry true
\use_package amsmath 1
\use_package amssymb 1
\use_package cancel 1
\use_package esint 1
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 1
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 1
\index Index
\shortcut idx
\color #008000
\end_index
\leftmargin 2cm
\topmargin 2cm
\rightmargin 2cm
\bottommargin 2cm
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Standard
\begin_inset Preview

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
begin{gnuplot}[terminal=cairolatex,terminaloptions=lw 3]
\end_layout

\begin_layout Plain Layout

set title "The sinc function"
\end_layout

\begin_layout Plain Layout

set xlabel "Abscissa"
\end_layout

\begin_layout Plain Layout

set ylabel "Ordinate"
\end_layout

\begin_layout Plain Layout

set samples 200
\end_layout

\begin_layout Plain Layout

set zeroaxis
\end_layout

\begin_layout Plain Layout

set border 31 lw 1
\end_layout

\begin_layout Plain Layout

plot sin(pi*x)/(pi*x) lw 2 title "sinc(x)"
\end_layout

\begin_layout Plain Layout


\backslash
end{gnuplot}
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\end_body
\end_document

Re: Can shell-escape take advantage of needauth framework?

[Enrico Forestieri]

On Mon, Jun 19, 2017 at 08:57:00PM +0200, Jürgen Spitzmüller wrote:

> Am Montag, den 19.06.2017, 20:33 +0200 schrieb Enrico Forestieri:
> > Because we don't know whether it's needed?
> 
> Why not? Can't we define that?

Are you able to tell that you need -shell-escape for the attached
document?

-- 
Enrico

Re: Can shell-escape take advantage of needauth framework?

[Jürgen Spitzmüller]

Am Montag, den 19.06.2017, 20:33 +0200 schrieb Enrico Forestieri:
> Because we don't know whether it's needed?

Why not? Can't we define that?

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: Can shell-escape take advantage of needauth framework?

[Enrico Forestieri]

On Mon, Jun 19, 2017 at 07:54:03PM +0200, Jürgen Spitzmüller wrote:
> 
> Again: Why do we need the toolbar button? Why not let the document
> itself ask for shell-escaping, depending on the need for that (e.g.,
> minted)?

Because we don't know whether it's needed?

-- 
Enrico

Re: Server rebooted

[Christian Ridderström]

On 19 June 2017 at 10:35, Jean-Marc Lasgouttes  wrote:

> Le 18/06/2017 à 16:29, Kornel Benko a écrit :
>
>> Server was out of memory. Stopping/starting apache seems to have resolved
>>> it.
>>>
>>
>> Yes, just tried 'git pull'. Immediate response.
>>
>
> I restarted it again since it was running on virtual memory. Sigh.
>

Do you mean it was using a lot of swap?


>  We have to do something.


I've activated some rules in IP-tables that blocks a few sets of chinese
IP-adresses which I've seen crawling over our server.
Let's see if this keeps the server happy. See below for the rules [1].

It might be that the blocks [1] are enough. I suggest we run the server
with these blocks in place for a while and see if it's still good after say
a week. As the server has recently needed a reboot every day or so, that
should be a strong indication that the issue is related to the subnets I
blocked.

Note: The rules for the IP-tables are _not_ saved permanently. If you
restart the server, the blocks are gone.


> There were talks about limiting the number of connection from a same
> source, or something.


IIRC, I did some tests/notes in order to limit connections from the same
source but I don't think I got all the way there.


> Another solution would be to use the honeypot's project http:BL API with
> something like mod_honeypot. But this is well beyond my skills.
>

Speaking of skills, I don't actually know how to use IP-tables, so if
anyone notices a problem with the rules etc [1], please let me know.


>
> http://www.projecthoneypot.org/services_overview.php
> http://www.miim.com/software/linux/modhoneypot.html
> http://www.projecthoneypot.org/httpbl_download.php
>
> I do not see many people advocating it, though. It might be that
> blacklisting by hand is effective enough for our needs.
>
> JMarc
>

/Christian

[1]

[chr@lyx iptables_backup_and_notes]$ cat config_iptables_2017-06-19.modified
# Created by CHR to block chinese IP-adresses, hoping it will fix issues
with server
# Install these rules e.g. as follows
#   cd ~lyx/iptables_backup_and_notes
#   sudo ./iptables_config.sh restore
~lyx/iptables_backup_and_notes/config_iptables_2017-06-19.modified
# Note: The command above does not save rules permanently.
#   If server is reset, the rules are note restored.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s 42.156.0.0/255.255.0.0 -j DROP
-A INPUT -s 42.120.0.0/255.255.0.0 -j DROP
-A INPUT -s 106.11.0.0/255.255.0.0 -j DROP
-A INPUT -s 106.38.0.0/255.255.0.0 -j DROP
-A INPUT -s 220.181.0.0/255.255.0.0 -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3690 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9418 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# End of rules.

Re: Can shell-escape take advantage of needauth framework?

[Jürgen Spitzmüller]

Am Montag, den 19.06.2017, 15:39 +0200 schrieb Enrico Forestieri:
> Sorry, it was not clear to me what you meant. Here is a patch
> following
> this strategy.
> 
> - We never store in the document the need for -shell-escape.
> - When the user checks the toolbar button and then runs a latex
> backend,
>   he is alerted that the backend will be allowed to run external
> programs.
> - At this point, he can decide to let the backend run (and be asked
> again
>   next time), or to always allow execution with -shell-escape for
> this doc.
> - If the user chooses to always allow -shell-escape for the current
> document,
>   the document path is stored in the session file, so that next time
> it is
>   loaded on the current machine, the toolbar button will be
> automatically
>   toggled and no question will be asked.
> - If the user manually toggles the toolbar button so that to disallow
> the
>   -shell-escape option for an authorized document, the document is
>   automatically removed from the list of authorized documents.
> 
> This patch does not introduce a format change, because nothing is
> recorded
> in the document (the document status is only recorded in the session
> file).

Again: Why do we need the toolbar button? Why not let the document
itself ask for shell-escaping, depending on the need for that (e.g.,
minted)?

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: Can shell-escape take advantage of needauth framework?

[Enrico Forestieri]

On Mon, Jun 19, 2017 at 06:39:22AM +0200, Jürgen Spitzmüller wrote:

> Am Sonntag, den 18.06.2017, 19:56 +0200 schrieb Enrico Forestieri:
> > > I think we need to provide an option to add -shell-escape only to
> > > specific documents and only on the given machine. This prevents
> > > sending
> > > documents with -shell-escape (main problem of a document setting).
> > 
> > This is contradictory. We avoid sending documents with -shell-escape
> > but then add it to specific documents. So, it is the same thing.
> 
> No, it isn't. I didn't propose a document property, but a per-document
> session setting. This is a completely different thing.

Sorry, it was not clear to me what you meant. Here is a patch following
this strategy.

- We never store in the document the need for -shell-escape.
- When the user checks the toolbar button and then runs a latex backend,
  he is alerted that the backend will be allowed to run external programs.
- At this point, he can decide to let the backend run (and be asked again
  next time), or to always allow execution with -shell-escape for this doc.
- If the user chooses to always allow -shell-escape for the current document,
  the document path is stored in the session file, so that next time it is
  loaded on the current machine, the toolbar button will be automatically
  toggled and no question will be asked.
- If the user manually toggles the toolbar button so that to disallow the
  -shell-escape option for an authorized document, the document is
  automatically removed from the list of authorized documents.

This patch does not introduce a format change, because nothing is recorded
in the document (the document status is only recorded in the session file).

-- 
Enrico
diff --git a/lib/ui/stdtoolbars.inc b/lib/ui/stdtoolbars.inc
index 9da37ecf75..794325665a 100644
--- a/lib/ui/stdtoolbars.inc
+++ b/lib/ui/stdtoolbars.inc
@@ -103,6 +103,7 @@ ToolbarSet
Item "Update" "buffer-update"
Item "View master document" "master-buffer-view"
Item "Update master document" "master-buffer-update"
+   Item "Allow running external programs" 
"buffer-toggle-shell-escape"
Item "Enable Forward/Reverse Search" "buffer-toggle-output-sync"
Separator
StickyPopupMenu "view-others" "View other formats"
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index 61b89200c1..b66c2522fe 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -55,6 +55,7 @@
 #include "ParagraphParameters.h"
 #include "ParIterator.h"
 #include "PDFOptions.h"
+#include "Session.h"
 #include "SpellChecker.h"
 #include "sgml.h"
 #include "texstream.h"
@@ -980,6 +981,11 @@ int Buffer::readHeader(Lexer & lex)
errorList.push_back(ErrorItem(_("Document header error"), s));
}
 
+   std::set & shellescape_files =
+   theSession().shellescapeFiles().shellescapeFiles();
+   if (shellescape_files.find(absFileName()) != shellescape_files.end())
+   params().shell_escape = true;
+
params().makeDocumentClass();
 
return unknown_tokens;
@@ -2613,6 +2619,11 @@ bool Buffer::getStatus(FuncRequest const & cmd, 
FuncStatus & flag)
break;
}
 
+   case LFUN_BUFFER_TOGGLE_SHELL_ESCAPE: {
+   flag.setOnOff(params().shell_escape);
+   break;
+   }
+
case LFUN_BUFFER_TOGGLE_OUTPUT_SYNC: {
flag.setOnOff(params().output_sync);
break;
@@ -2888,6 +2899,18 @@ void Buffer::dispatch(FuncRequest const & func, 
DispatchResult & dr)
params().compressed = !params().compressed;
break;
 
+   case LFUN_BUFFER_TOGGLE_SHELL_ESCAPE:
+   params().shell_escape = !params().shell_escape;
+   if (!params().shell_escape) {
+   std::set & shellescape_files =
+   
theSession().shellescapeFiles().shellescapeFiles();
+   std::set::iterator it =
+   shellescape_files.find(absFileName());
+   if (it != shellescape_files.end())
+   shellescape_files.erase(it);
+   }
+   break;
+
case LFUN_BUFFER_TOGGLE_OUTPUT_SYNC:
undo().recordUndoBufferParams(CursorData());
params().output_sync = !params().output_sync;
diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 38ca643400..8b282171ef 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -459,6 +459,7 @@ BufferParams::BufferParams()
html_css_as_file = false;
display_pixel_ratio = 1.0;
 
+   shell_escape = false;
output_sync = false;
use_refstyle = true;
use_minted = false;
diff --git a/src/BufferParams.h b/src/BufferParams.h
index 9f20ce14c6..aa33b9a61e 100644
--- a/src/BufferParams.h
+++ b/src/BufferParams.h
@@ -535,6 +535,8 @@ 

Re: Server rebooted

[Jean-Marc Lasgouttes]

Le 18/06/2017 à 16:29, Kornel Benko a écrit :

Server was out of memory. Stopping/starting apache seems to have resolved
it.


Yes, just tried 'git pull'. Immediate response.


I restarted it again since it was running on virtual memory. Sigh.

 We have to do something. There were talks about limiting the number of 
connection from a same source, or something. Another solution would be 
to use the honeypot's project http:BL API with something like 
mod_honeypot. But this is well beyond my skills.


http://www.projecthoneypot.org/services_overview.php
http://www.miim.com/software/linux/modhoneypot.html
http://www.projecthoneypot.org/httpbl_download.php

I do not see many people advocating it, though. It might be that 
blacklisting by hand is effective enough for our needs.


JMarc