Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 09, 2017 at 04:49:57PM -0400, Scott Kostyshak wrote:
> On Wed, Aug 09, 2017 at 08:13:01PM +0200, Enrico Forestieri wrote:
> > On Wed, Aug 09, 2017 at 05:35:35PM +0200, Jürgen Spitzmüller wrote:
> > > Am Mittwoch, den 09.08.2017, 17:11 +0200 schrieb Enrico Forestieri:
> > > > Please, can you check if the attached patch addresses your concern?
> > > 
> > > Yes, it does. Thanks.
> > 
> > I don't know whether this counts as an explicit consent to apply. Scott?
> 
> Please commit. I did a quick test and it worked well.

I committed a sligthly refined version at 40e9dea5.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Wed, Aug 09, 2017 at 08:13:01PM +0200, Enrico Forestieri wrote:
> On Wed, Aug 09, 2017 at 05:35:35PM +0200, Jürgen Spitzmüller wrote:
> 
> > Am Mittwoch, den 09.08.2017, 17:11 +0200 schrieb Enrico Forestieri:
> > > Please, can you check if the attached patch addresses your concern?
> > 
> > Yes, it does. Thanks.
> 
> I don't know whether this counts as an explicit consent to apply. Scott?

Please commit. I did a quick test and it worked well.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 09, 2017 at 05:35:35PM +0200, Jürgen Spitzmüller wrote:

> Am Mittwoch, den 09.08.2017, 17:11 +0200 schrieb Enrico Forestieri:
> > Please, can you check if the attached patch addresses your concern?
> 
> Yes, it does. Thanks.

I don't know whether this counts as an explicit consent to apply. Scott?

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am Mittwoch, den 09.08.2017, 17:11 +0200 schrieb Enrico Forestieri:
> Please, can you check if the attached patch addresses your concern?

Yes, it does. Thanks.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Aug 08, 2017 at 04:01:40PM +0200, Jürgen Spitzmüller wrote:
> Am Dienstag, den 25.07.2017, 21:19 +0200 schrieb Enrico Forestieri:
> > On Tue, Jul 25, 2017 at 11:11:36AM +0200, Pavel Sanda wrote:
> > 
> > > Scott Kostyshak wrote:
> > > > It did feel strange that the "OK" and "Apply" buttons were not
> > > > enabled
> > > > when checking the "Allow running external programs" checkbox, but
> > > > I got
> > > > used to it quickly.
> > > 
> > > Likely not feature but just forgotten connect with signal of box
> > > being checked.
> > 
> > No, this was done by design. The status of this checkbox is not saved
> > in
> > the document, so the document should not be marked as dirty.
> 
> I find it irritating as well. Since this dialog in general follows an
> OK/Apply paradigm, I think this should be done for this checkbox as
> well, even if the change does not survive sessions (and at the cost of
> a dirty buffer).

Please, can you check if the attached patch addresses your concern?

-- 
Enrico
diff --git a/src/frontends/qt4/GuiDocument.cpp 
b/src/frontends/qt4/GuiDocument.cpp
index 8889fdf677..04e78d1ae6 100644
--- a/src/frontends/qt4/GuiDocument.cpp
+++ b/src/frontends/qt4/GuiDocument.cpp
@@ -664,7 +664,8 @@ void LocalLayout::validatePressed() {
 
 GuiDocument::GuiDocument(GuiView & lv)
: GuiDialog(lv, "document", qt_("Document Settings")),
- biblioChanged_(false), nonModuleChanged_(false)
+ biblioChanged_(false), nonModuleChanged_(false),
+ modulesChanged_(false), shellescapeChanged_(false)
 {
setupUi(this);
 
@@ -1542,9 +1543,16 @@ void GuiDocument::change_adaptor()
 
 void GuiDocument::shellescapeChanged()
 {
-   // This is treated specially as the change is automatically applied
-   // and the document isn't marked as dirty. Visual feedback is given
-   // by the appearance/disappearance of a red icon in the status bar.
+   shellescapeChanged_ = true;
+   changed();
+}
+
+
+void GuiDocument::updateShellEscape()
+{
+   if (!shellescapeChanged_)
+   return;
+
bp_.shell_escape = outputModule->shellescapeCB->isChecked();
if (!bp_.shell_escape)
theSession().shellescapeFiles().remove(buffer().absFileName());
@@ -1552,8 +1560,28 @@ void GuiDocument::shellescapeChanged()
theSession().shellescapeFiles().insert(buffer().absFileName());
Buffer & buf = const_cast(buffer());
buf.params().shell_escape = bp_.shell_escape;
-   BufferView * bv = const_cast(bufferview());
-   bv->processUpdateFlags(Update::Force);
+}
+
+
+void GuiDocument::slotApply()
+{
+   updateShellEscape();
+   bool wasclean = !nonModuleChanged_ && buffer().isClean();
+   GuiDialog::slotApply();
+   if (wasclean && !modulesChanged_)
+   buffer().markClean();
+   modulesChanged_ = false;
+}
+
+
+void GuiDocument::slotOK()
+{
+   updateShellEscape();
+   bool wasclean = !nonModuleChanged_ && buffer().isClean();
+   GuiDialog::slotOK();
+   if (wasclean && !modulesChanged_)
+   buffer().markClean();
+   modulesChanged_ = false;
 }
 
 
@@ -2662,7 +2690,7 @@ void GuiDocument::modulesChanged()
 {
modulesToParams(bp_);
 
-   if (applyPB->isEnabled() && nonModuleChanged_) {
+   if (applyPB->isEnabled() && (nonModuleChanged_ || shellescapeChanged_)) 
{
int const ret = Alert::prompt(_("Unapplied changes"),
_("Some changes in the dialog were not yet 
applied.\n"
"If you do not apply now, they will be lost 
after this action."),
@@ -2671,6 +2699,7 @@ void GuiDocument::modulesChanged()
applyView();
}
 
+   modulesChanged_ = true;
bp_.makeDocumentClass();
paramsToDialog();
changed();
@@ -3276,6 +3305,7 @@ void GuiDocument::applyView()
 
// reset tracker
nonModuleChanged_ = false;
+   shellescapeChanged_ = false;
 }
 
 
@@ -3865,6 +3895,7 @@ void GuiDocument::paramsToDialog()
 
// reset tracker
nonModuleChanged_ = false;
+   shellescapeChanged_ = false;
 }
 
 
diff --git a/src/frontends/qt4/GuiDocument.h b/src/frontends/qt4/GuiDocument.h
index 5efde8eee3..37cecbfe60 100644
--- a/src/frontends/qt4/GuiDocument.h
+++ b/src/frontends/qt4/GuiDocument.h
@@ -86,6 +86,10 @@ public:
 
 public Q_SLOTS:
void onBufferViewChanged();//override
+   // OK button clicked
+   void slotOK();
+   // Apply button clicked
+   void slotApply();
 
 private Q_SLOTS:
void updateNumbering();
@@ -274,6 +278,8 @@ private:
///
void loadModuleInfo();
///
+   void updateShellEscape();
+   ///
void updateUnknownBranches();
///
void updateTexFonts();
@@ -303,6 +309,10 @@ private:
bool biblioChanged_;
/// Track if a non-module document param changed
bool nonModuleChanged_;
+   /// Track if 

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am Dienstag, den 25.07.2017, 21:19 +0200 schrieb Enrico Forestieri:
> On Tue, Jul 25, 2017 at 11:11:36AM +0200, Pavel Sanda wrote:
> 
> > Scott Kostyshak wrote:
> > > It did feel strange that the "OK" and "Apply" buttons were not
> > > enabled
> > > when checking the "Allow running external programs" checkbox, but
> > > I got
> > > used to it quickly.
> > 
> > Likely not feature but just forgotten connect with signal of box
> > being checked.
> 
> No, this was done by design. The status of this checkbox is not saved
> in
> the document, so the document should not be marked as dirty.

I find it irritating as well. Since this dialog in general follows an
OK/Apply paradigm, I think this should be done for this checkbox as
well, even if the change does not survive sessions (and at the cost of
a dirty buffer).

Or maybe the document dialog is not the right place for this widget
(but I don't know a better one).

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Paul A. Rubin]

On 08/05/2017 10:49 AM, mn wrote:

On 05.08.17 16:26, Paul A. Rubin wrote:


FWIW, I subscribe through GMail, using a filter to put the dev and user
lists in separated virtual folders, but access my GMail account in
Thunderbird (IMAP). The messages are individual. I don't know if there's
a way to maintain threads in T-bird (haven't tried), but I can always
use a "Quick Filter" to isolate a thread.


If you want to try, these ideas might help:
https://support.mozilla.org/en-US/questions/1079651

greetings
Mike
Oops -- turns out I was having an extended senior moment. There's a 
column header in T-bird that sorts messages into threads if you click 
it. I completely forgot it was there. (Until I subscribed to the LyX 
lists, I never really had a use for threading in my mail client.)


And yes, once messages are threaded, you can tag or mark individual 
messages without tagging/marking the full thread.


Paul

Re: How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Paul A. Rubin]

On 08/05/2017 10:49 AM, mn wrote:

On 05.08.17 16:26, Paul A. Rubin wrote:


FWIW, I subscribe through GMail, using a filter to put the dev and user
lists in separated virtual folders, but access my GMail account in
Thunderbird (IMAP). The messages are individual. I don't know if there's
a way to maintain threads in T-bird (haven't tried), but I can always
use a "Quick Filter" to isolate a thread.


If you want to try, these ideas might help:
https://support.mozilla.org/en-US/questions/1079651

greetings
Mike

Thanks!

Paul

Re: How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[mn]

On 05.08.17 16:26, Paul A. Rubin wrote:

> FWIW, I subscribe through GMail, using a filter to put the dev and user 
> lists in separated virtual folders, but access my GMail account in 
> Thunderbird (IMAP). The messages are individual. I don't know if there's 
> a way to maintain threads in T-bird (haven't tried), but I can always 
> use a "Quick Filter" to isolate a thread.


If you want to try, these ideas might help:
https://support.mozilla.org/en-US/questions/1079651

greetings
Mike

Re: How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Paul A. Rubin]

On 08/04/2017 10:02 PM, Scott Kostyshak wrote:

On Fri, Aug 04, 2017 at 08:22:40AM +0200, Christian Ridderström wrote:

On 2 August 2017 at 10:03, Scott Kostyshak  wrote:



I'm using gmail's web interface these days. This might be why I'm finding
it difficult to efficiently follow threads that are so long.
- The gmail labs thing I used for replying to parts of an e-mail is no
longer working.
- Sometimes the replies don't go to the list. *sigh*
- I haven't figured out how to mark a single e-mail as unread,
   e.g. when I feel I need to reply to it later, and instead have to mark
the entire thread.
   This does _not_ work well for long threads.


I find mutt to be very helpful with emails, but it takes a long time to
get used
to, and I'm not sure it would solve the particular issues you reference.


A decade or so ago I used 'pine' which worked really well for my e-mails
and the LyX lists.
Back then I accessed multiple lists as news groups, but as 'pine' tracks
the posts you've read
through a local file, I had to run 'pine' on a single computer. So I then
ran 'pine' within a screen session on server to which I SSH:d. I needed
this because I worked from multiple computers running on different
platforms.

This worked well, and the drawback was mainly when people posted in HTML,
or included URLs which I manually had to copy and paste.

HTML is still (probably even worse) a problem. Whenever I receive HTML-only
emails, I respond and explain why I would prefer a multi-part MIME email. In the
end, it's not really a problem: Mutt can automatically convert the HTML to text,
using e.g. w3m. Or you can press the keys "vm" and it will view the email in
your browser.

As for links, you can set up mutt to pipe to urlview, which is quite useful.


I stopped using pine, probably triggered by losing access to the server.
There were also issues related to reading regular e-mail when people
started inserting lots of HTML, images and links. I did try switching to
'alpine', and probably considered using 'mutt', but there was still the
issue of needing to run it from a single computer.

Back then I think I could switch topic by changing the subject the topic,
but I'm not sure it works so well when using gmail's web interface.

That I stopped using 'pine' and switched to gmail also resulted in my not
being able to follow the LyX lists, and this was probably the main reason I
became mainly inactive within LyX for a while.
I'm now using my c...@lyx.org account for LyX related issues, and that is
working better -- except that gmail isn't working well at all for me when
there are long threads.

I've changed the subject of this reply and I'll see how it appears in my
gmail. I.e., will it be obvious it's a new topic, or will it be joined with
the original topic.

Regarding breaking threads, I will try to remember to break threads when a
discussion I wish to start is completely new. Please feel free to remind me or
point out where I could have done that. I can see how it would be easier for you
and others.

Scott
FWIW, I subscribe through GMail, using a filter to put the dev and user 
lists in separated virtual folders, but access my GMail account in 
Thunderbird (IMAP). The messages are individual. I don't know if there's 
a way to maintain threads in T-bird (haven't tried), but I can always 
use a "Quick Filter" to isolate a thread.


Paul

Re: How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Scott Kostyshak]

On Fri, Aug 04, 2017 at 08:22:40AM +0200, Christian Ridderström wrote:
> On 2 August 2017 at 10:03, Scott Kostyshak  wrote:
> 
> >
> > > 
> > > I'm using gmail's web interface these days. This might be why I'm finding
> > > it difficult to efficiently follow threads that are so long.
> > > - The gmail labs thing I used for replying to parts of an e-mail is no
> > > longer working.
> > > - Sometimes the replies don't go to the list. *sigh*
> > > - I haven't figured out how to mark a single e-mail as unread,
> > >   e.g. when I feel I need to reply to it later, and instead have to mark
> > > the entire thread.
> > >   This does _not_ work well for long threads.
> > > 
> >
> > I find mutt to be very helpful with emails, but it takes a long time to
> > get used
> > to, and I'm not sure it would solve the particular issues you reference.
> >
> 
> A decade or so ago I used 'pine' which worked really well for my e-mails
> and the LyX lists.
> Back then I accessed multiple lists as news groups, but as 'pine' tracks
> the posts you've read
> through a local file, I had to run 'pine' on a single computer. So I then
> ran 'pine' within a screen session on server to which I SSH:d. I needed
> this because I worked from multiple computers running on different
> platforms.
> 
> This worked well, and the drawback was mainly when people posted in HTML,
> or included URLs which I manually had to copy and paste.

HTML is still (probably even worse) a problem. Whenever I receive HTML-only
emails, I respond and explain why I would prefer a multi-part MIME email. In the
end, it's not really a problem: Mutt can automatically convert the HTML to text,
using e.g. w3m. Or you can press the keys "vm" and it will view the email in
your browser.

As for links, you can set up mutt to pipe to urlview, which is quite useful.

> I stopped using pine, probably triggered by losing access to the server.
> There were also issues related to reading regular e-mail when people
> started inserting lots of HTML, images and links. I did try switching to
> 'alpine', and probably considered using 'mutt', but there was still the
> issue of needing to run it from a single computer.
> 
> Back then I think I could switch topic by changing the subject the topic,
> but I'm not sure it works so well when using gmail's web interface.
> 
> That I stopped using 'pine' and switched to gmail also resulted in my not
> being able to follow the LyX lists, and this was probably the main reason I
> became mainly inactive within LyX for a while.
> I'm now using my c...@lyx.org account for LyX related issues, and that is
> working better -- except that gmail isn't working well at all for me when
> there are long threads.
> 
> I've changed the subject of this reply and I'll see how it appears in my
> gmail. I.e., will it be obvious it's a new topic, or will it be joined with
> the original topic.

Regarding breaking threads, I will try to remember to break threads when a
discussion I wish to start is completely new. Please feel free to remind me or
point out where I could have done that. I can see how it would be easier for you
and others.

Scott


signature.asc
Description: PGP signature

Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Pavel Sanda]

Christian Ridderström wrote:
> On 2 August 2017 at 11:29, Pavel Sanda  wrote:
> 
> > Scott Kostyshak wrote:
> > > Ah I did not realize you did that on purpose. I actually found it
> > annoying when
> > > I wanted to go up the discussion and couldn't because it was cut off
> >
> > +1
> >
> 
> I just want to check you understood that I primarily mean a new thread when
> the discussion is diverging in topic compared to the original post, i.e. to
> avoid "threadjacking". For instance, would you've preferred this post to
> have been in Scott's original thread?

No, not this post as its completely unrelated to the technical issue of
minted, shell-escape and needauth.

On contrary completely new thread "Living with shell-escape: Using two LyX
instances" in which you pick several quotes from other email and start anew
makes it hard to track the discussion properly within the context later on.

Pavel

Re: Discussing minted/safety/etc (Was: Options for resolving the minted + shell-escape issue)

[Tommaso Cucinotta]

On 02/08/2017 01:00, Christian Ridderström wrote:

The long threads likely by now do contain a lot of separate pieces of 
description, but it's unfortunately not so easy (at least for me) to find this 
information among all the posts. [...] Or perhaps to copy them to into one 
place, adding minor markup as to what's still valid.


I think this is already a great improvement

  http://wiki.lyx.org/Devel/SafetyAndSecurity

we just need more eyes to go through it, so to add possible important missing 
bits.

T.

Re: Options for resolving the minted + shell-escape issue

[Tommaso Cucinotta]

On 02/08/2017 17:45, Richard Heck wrote:

do even better. [...] People who have the expertise to design a new solution, 
and then to implement it, whether via AppArmor or whatever, should feel free to 
make it a priority for 2.4 and create a feature branch for it.


+1, I'll take some time to rebase my old patch and double-check it has a switch 
to turn it off on other OSes and Linux distros w/out AA.

T.

How to access the list efficiently? Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

On 2 August 2017 at 10:03, Scott Kostyshak  wrote:

>
> > 
> > I'm using gmail's web interface these days. This might be why I'm finding
> > it difficult to efficiently follow threads that are so long.
> > - The gmail labs thing I used for replying to parts of an e-mail is no
> > longer working.
> > - Sometimes the replies don't go to the list. *sigh*
> > - I haven't figured out how to mark a single e-mail as unread,
> >   e.g. when I feel I need to reply to it later, and instead have to mark
> > the entire thread.
> >   This does _not_ work well for long threads.
> > 
>
> I find mutt to be very helpful with emails, but it takes a long time to
> get used
> to, and I'm not sure it would solve the particular issues you reference.
>

A decade or so ago I used 'pine' which worked really well for my e-mails
and the LyX lists.
Back then I accessed multiple lists as news groups, but as 'pine' tracks
the posts you've read
through a local file, I had to run 'pine' on a single computer. So I then
ran 'pine' within a screen session on server to which I SSH:d. I needed
this because I worked from multiple computers running on different
platforms.

This worked well, and the drawback was mainly when people posted in HTML,
or included URLs which I manually had to copy and paste.

I stopped using pine, probably triggered by losing access to the server.
There were also issues related to reading regular e-mail when people
started inserting lots of HTML, images and links. I did try switching to
'alpine', and probably considered using 'mutt', but there was still the
issue of needing to run it from a single computer.

Back then I think I could switch topic by changing the subject the topic,
but I'm not sure it works so well when using gmail's web interface.

That I stopped using 'pine' and switched to gmail also resulted in my not
being able to follow the LyX lists, and this was probably the main reason I
became mainly inactive within LyX for a while.
I'm now using my c...@lyx.org account for LyX related issues, and that is
working better -- except that gmail isn't working well at all for me when
there are long threads.

I've changed the subject of this reply and I'll see how it appears in my
gmail. I.e., will it be obvious it's a new topic, or will it be joined with
the original topic.

/Christian

Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

On 2 August 2017 at 11:29, Pavel Sanda  wrote:

> Scott Kostyshak wrote:
> > Ah I did not realize you did that on purpose. I actually found it
> annoying when
> > I wanted to go up the discussion and couldn't because it was cut off
>
> +1
>

I just want to check you understood that I primarily mean a new thread when
the discussion is diverging in topic compared to the original post, i.e. to
avoid "threadjacking". For instance, would you've preferred this post to
have been in Scott's original thread?

If that's what the majority prefers I can change to that.
/C



>

Re: Options for resolving the minted + shell-escape issue

[Guenter Milde]

Dear Scott, dear Enrico, dear LyX developers,

On 2017-08-02, Enrico Forestieri wrote:
> On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:

>> I expected the point about a second vote being unfair, but since I did not
>> consider the first vote fair (due to my poor structuring of the options), I
>> chose to do a second vote.

> I must add that your fairness was lacking well before, trying to treat
> differently dangerous converters and minted. 

IMV, Scott did a good job in trying to keep a balanced view in face of the
opinions varying so widely and the heated discussion.


There are basic differences between minted support and the unsafe converters
now handled with needauth. This makes it difficult to apply a commonly
accepted "threat score".

* Minted package support is not dangerous itself

* However, minted support is of no use unless the "pdflatex", "lualatex"
  or "xelatex" converters are made insecure by calling them with
  --shell-escape.

* --shell-escape affects not only some short code snippets but the complete
  LaTeX code of the document and all included files.
  It is rather difficult to detect a trojan in a definition file
  included by a package required by the document class and fetched from a
  search path comprising hundreds of files.

* Due to the larger user base, the incentive to program a virus or trojan
  using LaTeX may be higher than for a "niche language" like gnuplot.


> But maybe you were skilfully deceived to remove minted support but not
> the needauth one.

I understand the frustration of every developer forced to remove changes
that are rejected by co-developers due to reasons hard to comprehend.

However, there is no need to suspect a conspiracy. From the release
manager perspective (as well as the user perspective), it is not about
*removing* minted support but about adding minted support in 2.3,
postponing the inclusion, or looking for alternatives.

OTOH, the needauth mechanism is an alternative to removing support for
dangerous converters present in 2.2.

Günter

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Thu, Aug 03, 2017 at 01:11:47PM +0200, Enrico Forestieri wrote:
> On Thu, Aug 03, 2017 at 11:04:01AM +0200, Enrico Forestieri wrote:
> 
> > On Wed, Aug 02, 2017 at 12:01:58PM -0400, Richard Heck wrote:
> > 
> > > On 08/02/2017 07:25 AM, Enrico Forestieri wrote:
> > > > On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:
> > > >> [[So the situation is as Richard described it:]] we can of course 
> > > >> decide to remove something before the final release.
> > > > Moreover, this last sentence means that you still want a sword of 
> > > > Damocles hanging on a feature that would have not been added without 
> > > > your interest for it.
> > > 
> > > I don't think that is entirely fair. Scott was simply echoing a remark
> > > *I* had made (I've restored something clipped), which is totally banal:
> > > We can ALWAYS decide to remove something if we feel we have to do so. I
> > > don't think that either Scott or I see any reason to do that now or
> > > expect that there will be any such reason later. I, at least, just meant
> > > to point out that deciding not to take a second vote doesn't mean we
> > > can't act later if we feel we have to do so. Obviously.
> > > 
> > > And just to be clear, the kind of reason that could possibly lead to
> > > removal of needauth, say, would NOT lie in the sorts of abstact
> > > considerations that have dominated recent discussion. Rather, it would
> > > emerge from testing, i.e., from problems encountered by users. So there
> > > is no suggestion here, either, that we should re-litigate the issue.
> > 
> > Thanks for the clarification. I just wanted to be sure this was the
> > correct interpretation. I must say that I had a hard time trying to
> > correctly interpret some decisions.
> 
> I have now committed the shell escape patch. Note that it introduces
> some string changes. It can be further refined and improved, of course.

Did a quick test and it looks good.

> Also note that I will be unavailable the next couple of days.

Thanks for the notice.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Thu, Aug 03, 2017 at 10:58:43AM +0200, Enrico Forestieri wrote:
> On Tue, Aug 01, 2017 at 04:57:33AM -0400, Scott Kostyshak wrote:
> > On Mon, Jul 31, 2017 at 11:43:04AM +0200, Enrico Forestieri wrote:
> > 
> > > Corrected at 44babaf6.
> > 
> > Thanks, that works well. The only other comment I have is to possibly
> > instruct the user to reconfigure after installing the module. If I
> > recall, that's what we say in similar contexts.
> 
> This is a bit different. We don't detect the python module because it
> is difficult doing so. The user can be using either Python 2 or 3, but
> the module can be for the "other" python version. It is the driver command
> that knows what to do. It is usually named "pygmentize" but it is not
> mandatory. For example, a distribution switches to Python 3 but still
> makes Python 2 available. A user installs the pygments module for Python 2
> whose driver command is maybe named pygmentize2 to avoid collision with
> the Python 3 version, which is also available.
> At the moment we only check for pygmentize but, according to users reports
> we can add other common names for the driver command. I really don't know
> what distributions will do. Maybe there will not be any implementation where
> the driver is called something different than pygmentize. In this case we
> can revisit the current strategy and simply disable the check box.

I see, that all makes sense.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Thu, Aug 03, 2017 at 11:04:01AM +0200, Enrico Forestieri wrote:

> On Wed, Aug 02, 2017 at 12:01:58PM -0400, Richard Heck wrote:
> 
> > On 08/02/2017 07:25 AM, Enrico Forestieri wrote:
> > > On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:
> > >> [[So the situation is as Richard described it:]] we can of course decide 
> > >> to remove something before the final release.
> > > Moreover, this last sentence means that you still want a sword of 
> > > Damocles hanging on a feature that would have not been added without your 
> > > interest for it.
> > 
> > I don't think that is entirely fair. Scott was simply echoing a remark
> > *I* had made (I've restored something clipped), which is totally banal:
> > We can ALWAYS decide to remove something if we feel we have to do so. I
> > don't think that either Scott or I see any reason to do that now or
> > expect that there will be any such reason later. I, at least, just meant
> > to point out that deciding not to take a second vote doesn't mean we
> > can't act later if we feel we have to do so. Obviously.
> > 
> > And just to be clear, the kind of reason that could possibly lead to
> > removal of needauth, say, would NOT lie in the sorts of abstact
> > considerations that have dominated recent discussion. Rather, it would
> > emerge from testing, i.e., from problems encountered by users. So there
> > is no suggestion here, either, that we should re-litigate the issue.
> 
> Thanks for the clarification. I just wanted to be sure this was the
> correct interpretation. I must say that I had a hard time trying to
> correctly interpret some decisions.

I have now committed the shell escape patch. Note that it introduces
some string changes. It can be further refined and improved, of course.
Also note that I will be unavailable the next couple of days.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 02, 2017 at 12:01:58PM -0400, Richard Heck wrote:

> On 08/02/2017 07:25 AM, Enrico Forestieri wrote:
> > On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:
> >> [[So the situation is as Richard described it:]] we can of course decide 
> >> to remove something before the final release.
> > Moreover, this last sentence means that you still want a sword of Damocles 
> > hanging on a feature that would have not been added without your interest 
> > for it.
> 
> I don't think that is entirely fair. Scott was simply echoing a remark
> *I* had made (I've restored something clipped), which is totally banal:
> We can ALWAYS decide to remove something if we feel we have to do so. I
> don't think that either Scott or I see any reason to do that now or
> expect that there will be any such reason later. I, at least, just meant
> to point out that deciding not to take a second vote doesn't mean we
> can't act later if we feel we have to do so. Obviously.
> 
> And just to be clear, the kind of reason that could possibly lead to
> removal of needauth, say, would NOT lie in the sorts of abstact
> considerations that have dominated recent discussion. Rather, it would
> emerge from testing, i.e., from problems encountered by users. So there
> is no suggestion here, either, that we should re-litigate the issue.

Thanks for the clarification. I just wanted to be sure this was the
correct interpretation. I must say that I had a hard time trying to
correctly interpret some decisions.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Aug 01, 2017 at 04:57:33AM -0400, Scott Kostyshak wrote:
> On Mon, Jul 31, 2017 at 11:43:04AM +0200, Enrico Forestieri wrote:
> 
> > Corrected at 44babaf6.
> 
> Thanks, that works well. The only other comment I have is to possibly
> instruct the user to reconfigure after installing the module. If I
> recall, that's what we say in similar contexts.

This is a bit different. We don't detect the python module because it
is difficult doing so. The user can be using either Python 2 or 3, but
the module can be for the "other" python version. It is the driver command
that knows what to do. It is usually named "pygmentize" but it is not
mandatory. For example, a distribution switches to Python 3 but still
makes Python 2 available. A user installs the pygments module for Python 2
whose driver command is maybe named pygmentize2 to avoid collision with
the Python 3 version, which is also available.
At the moment we only check for pygmentize but, according to users reports
we can add other common names for the driver command. I really don't know
what distributions will do. Maybe there will not be any implementation where
the driver is called something different than pygmentize. In this case we
can revisit the current strategy and simply disable the check box.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Mittwoch, 2. August 2017 um 16:11:42, schrieb Guillaume MM 
> Le 01/08/2017 à 10:50, Scott Kostyshak a écrit :
> > On Mon, Jul 31, 2017 at 09:07:11PM +0200, Guillaume MM wrote:
> > 
> >> I am sure that Scott meant to include in some way the option that I have
> >> been advocating constantly from the beginning, which I understand is
> >> probably 1. (Otherwise, I do not see what the option 1. refers to nor
> >> who proposed it, and I would opt for not taking part in the vote.)
> > 
> > Yes my intention was to represent your opinion in a simplified way. I'm
> > sorry to have failed that. You will have an opportunity to write your
> > own option for the next vote (see separate email).
> 
> 
> Hi Scott,
> 
> That's alright, I think my opinion was represented well enough. Patch
> 0002 was disabling minted as per 1., and patches 0001 and 0003 at
> 
> answer your more general question "how can we make LyX the most secure
> for 2.3?". One can see it like that. 0001 takes various comments that
> have been made about Enrico's patch by various people into account and
> 0003 fixes a small display issue.

Nice, but ..

> It just misses for now the enabling of
> shell-escape via the improved needauth mechanism but this looks simple
> enough to do, and after that it can be proposed as an improvement to
> Enrico's mechanism.

I think that this is essential for testing.
Could you please provide a complete patch?
FWIW, Enrico's last patch is working well, so for a comparison it would be nice
to have this one complete too.

> Guillaume
> 

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 08/02/2017 10:11 AM, Guillaume MM wrote:
>  Please do not let Enrico's new attacks affect you.

Can we please put an end to this kind of public comment? If you want to
make
such remarks privately, that is totally reasonable. But this sort of
snark is totally
uncalled for, and it is not helping right now.

Richard

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 08/02/2017 07:25 AM, Enrico Forestieri wrote:
> On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:
>> [[So the situation is as Richard described it:]] we can of course decide to 
>> remove something before the final release.
> Moreover, this last sentence means that you still want a sword of Damocles 
> hanging on a feature that would have not been added without your interest for 
> it.

I don't think that is entirely fair. Scott was simply echoing a remark
*I* had made (I've restored something clipped), which is totally banal:
We can ALWAYS decide to remove something if we feel we have to do so. I
don't think that either Scott or I see any reason to do that now or
expect that there will be any such reason later. I, at least, just meant
to point out that deciding not to take a second vote doesn't mean we
can't act later if we feel we have to do so. Obviously.

And just to be clear, the kind of reason that could possibly lead to
removal of needauth, say, would NOT lie in the sorts of abstact
considerations that have dominated recent discussion. Rather, it would
emerge from testing, i.e., from problems encountered by users. So there
is no suggestion here, either, that we should re-litigate the issue.

Richard

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 08/01/2017 07:13 PM, Christian Ridderström wrote:
> On 1 August 2017 at 21:24, Richard Heck  > wrote:
>
> Hi Richard,
>
> It sounds like you complain about the time spent on discussing what I
> assume is not just the minted patch but safety/security in general.
> However, regarding spending to much time discussing safety, I can't
> help but think about LyX 2.2 where something was obviously missing in
> terms of safety.
> That leads me to conclude that something was missing, or went awry, in
> the development process of LyX 2.2. This, combined with you advocating
>
>  "... spend our time figuring how to make the needauth
> and shell-escape stuff as secure
>   as we can make it, given the present framework" 
>
> makes me a little concerned.
>
> It sounds like you argue for a release of 2.3 regardless of the
> absolute achieved level of safety, because:
> a) we've anyway done all that can be done using the needauth framework.
> b) it's anyway better than LyX 2.2.
>
> Would you mind clarifying your point of view?

No, I don't mean to complain about our discussing this in such detail.
Such discussions are tremendously important, since security issues are
tremendously important. I simply meant to point out that we *have*
discussed this in great detail. At some point we have to make a
decision. We could continue discussing the general issue---and the wiki
page, etc, that you've created are a good step in that direction---but
the overall view recently has been that the present discussion was not
changing anyone's mind. It was just getting people upset.

Since we have had to take a vote, we know it's a decision with which not
everyone will be happy. But once a decision has been made, we need to do
our best, *as a team*, to implement the decision that was made as best
we can. Continuing to discuss the question that's already been decided,
by rare vote, is not productive.

That said, it does matter to me that the needauth framework, so far as I
can see, makes things *better* than they were in LyX 2.2. LyX 2.3, in my
opinion, will be *more secure* than LyX 2.2 was. If I thought that
security was being *reduced*, that would be an entirely different matter.

I'm sure it's true that we could, in principle, do even better. That is
why it's not unreasonable to continue this discussion on a somewhat
different level. People who have the expertise to design a new solution,
and then to implement it, whether via AppArmor or whatever, should feel
free to make it a priority for 2.4 and create a feature branch for it.
(I'm guessing this would involve format changes.) Maybe that would even
be a reason to do an early release of 2.4. I don't mean to be
pre-judging that.

Richard

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 02, 2017 at 04:11:42PM +0200, Guillaume MM wrote:
> 
> The first thing we will remember about your handling of this discussion
> is your commitment of rationality. Please do not let Enrico's new
> attacks affect you.

I have really no words able to correctly describe you.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Guillaume MM]

Le 01/08/2017 à 10:50, Scott Kostyshak a écrit :

On Mon, Jul 31, 2017 at 09:07:11PM +0200, Guillaume MM wrote:


I am sure that Scott meant to include in some way the option that I have
been advocating constantly from the beginning, which I understand is
probably 1. (Otherwise, I do not see what the option 1. refers to nor
who proposed it, and I would opt for not taking part in the vote.)


Yes my intention was to represent your opinion in a simplified way. I'm
sorry to have failed that. You will have an opportunity to write your
own option for the next vote (see separate email).



Hi Scott,

That's alright, I think my opinion was represented well enough. Patch
0002 was disabling minted as per 1., and patches 0001 and 0003 at

answer your more general question "how can we make LyX the most secure
for 2.3?". One can see it like that. 0001 takes various comments that
have been made about Enrico's patch by various people into account and
0003 fixes a small display issue. It just misses for now the enabling of
shell-escape via the improved needauth mechanism but this looks simple
enough to do, and after that it can be proposed as an improvement to
Enrico's mechanism.

The first thing we will remember about your handling of this discussion
is your commitment of rationality. Please do not let Enrico's new
attacks affect you.

Guillaume

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:

> I expected the point about a second vote being unfair, but since I did not
> consider the first vote fair (due to my poor structuring of the options), I
> chose to do a second vote.

I must add that your fairness was lacking well before, trying to treat
differently dangerous converters and minted. But maybe you were skilfully
deceived to remove minted support but not the needauth one.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Jean-Marc Lasgouttes]

Agreed too. I will add that we should not neglect the harm we are inflicting to 
our smallish team by making people increasingly pissed off at each other. 
Security is important, but bikeshedding has a cost, and I am not sure that we 
can afford it.

We have to take into account that the minted problem is supposed to eventually 
fix itself, meaning that none of our choices will paint us in a corner. 
Therefore we can afford to be wrong, instead of looking for an elusive perfect 
solution.

JMarc

Le 2 août 2017 13:17:55 GMT+02:00, "Jürgen Spitzmüller"  a écrit 
>
>I agree.

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Aug 02, 2017 at 06:05:19AM -0400, Scott Kostyshak wrote:
> 
> I am going to take the advice of the senior developers and to cancel the
> second vote.

Dear Scott, you can try to remove the nail but the hole remains.

> we can of course decide to remove something before the final release.

Moreover, this last sentence means that you still want a sword of Damocles
hanging on a feature that would have not been added without your interest
for it.

I am trying to analyze your behavior in this matter in order to take
a decision about what to do with my future involvement with LyX.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am 01.08.2017 9:24 nachm. schrieb "Richard post-beta1 vote, we may decide
to remove it.


It's obviously up to you, Scott, as release manager, but I agree with
Enrico, to some extent,
that this isn't the best policy. We have spent an enormous amount of
time on this and finally
have come to some kind of resolution. It really would be best to close
that part of the debate
and spend our time figuring out how to make the needauth and
shell-escape stuff as secure as
we can make it, given the present framework. Time spent re-litigating
the questions about the
framework is going to be wasted time.


I agree.

Jürgen


.

Richard

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Christian Ridderström wrote:
> On 1 August 2017 at 01:25, Pavel Sanda  wrote:
> 
> Would you mind pointing me to that post?  I don't know if I simply missed it
> or was unable to read/consider/reply at the time.

I can, but would you mind start using some reasonable technical solution rather
than gmail if you want to participate in more complex threads on this email 
list?

Pavel

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Wed, Aug 02, 2017 at 11:15:23AM +0200, Pavel Sanda wrote:
> Richard Heck wrote:
> > It's obviously up to you, Scott, as release manager, but I agree with 
> > Enrico,
> > to some extent, that this isn't the best policy. We have spent an enormous
> > amount of time on this and finally have come to some kind of resolution. It
> > really would be best to close that part of the debate and spend our time
> > figuring out how to make the needauth and shell-escape stuff as secure as we
> > can make it, given the present framework. Time spent re-litigating the
> > questions about the framework is going to be wasted time.
> 
> +1

I did not expect such strong opinions against a second vote. I expected the
point about a second vote being unfair, but since I did not consider the first
vote fair (due to my poor structuring of the options), I chose to do a second
vote. But it seems I did not correctly estimate the cost of a second vote, not
just in terms of time, but I think in other dimensions as well. I am going to
take the advice of the senior developers and to cancel the second vote.

In this decision to cancel the second vote, in addition to experience of the
developers who gave their opinions (on the list as well as through private
email), it also made a difference that Pavel, who did not vote for option 3, and
who was part of the reason I decided to have a second vote (because his was one
of the opinions not represented as an option), also supports canceling the
second vote.

Further, I do remember asking for one final cognitive spurt. And I really
appreciate those who gave it.

So the situation is as Richard described it: we can of course decide to remove
something before the final release. But as of now, there is no future vote
planned on this issue.

Thanks to everyone for your help and advice,

Scott


signature.asc
Description: PGP signature

Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Pavel Sanda]

Scott Kostyshak wrote:
> Ah I did not realize you did that on purpose. I actually found it annoying 
> when
> I wanted to go up the discussion and couldn't because it was cut off

+1

Pavel

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Richard Heck wrote:
> It's obviously up to you, Scott, as release manager, but I agree with Enrico,
> to some extent, that this isn't the best policy. We have spent an enormous
> amount of time on this and finally have come to some kind of resolution. It
> really would be best to close that part of the debate and spend our time
> figuring out how to make the needauth and shell-escape stuff as secure as we
> can make it, given the present framework. Time spent re-litigating the
> questions about the framework is going to be wasted time.

+1

Pavel

Re: Discussing minted/safety/etc (Was: Options for resolving the minted + shell-escape issue)

[Scott Kostyshak]

On Wed, Aug 02, 2017 at 01:00:47AM +0200, Christian Ridderström wrote:

> Hmm, and perhaps in case of votes, Scott should (be allowed to) weigh them
> differently depending on the persons knowledge. E.g., since I don't
> know/understand the (intended) safety design, why should my vote count as
> much as a potential safety "guru".

I will not be assigning weights to votes. We can restrict who votes (e.g. those
who participate in a discussion), but assigning subjective weights does not seem
like a good idea to me. There are ways to improve the vote for next time, and I
will write some ideas down and ask for feedback on them.

From what I see, you have great knowledge and experience with security. If the
above comment is because I did not reference you when I counted the votes, all I
can say is that it was a mistake.

> On a more practical level:
> Hindsight is 20-20, but regarding e.g. Enrico's patch, perhaps we should
> have created a branch representing that alternative?   This would have
> allowed that branch to always represent the latest improvement, thus
> avoiding people testing out an older patch accidentally.
> So we should keep the possibility of using branches in mind for future
> discussions.

This is a good idea, I think. We do have a features repository. I think part of
it is that I did not expect it to be such a long discussion. It snowballed.

> Perhaps there's other more practical things we could've done to make life
> easier.

Thanks for starting this discussion. I would like to spend more time on
reflection, but I also want to spend time on addressing issues needed to get
beta1 out.

> [*] Regarding SW design descriptions and actually being able to understand
> and review an intended design in advance, I'm probably a bit "damaged" from
> when I worked with satellite software (AOCS), e.g. as SW verification and
> validation manager.
> 
> In that field, lots of documentation was required which often was annoying,
> but for complicated stuff like FDIR (making the SW/satellite be able to
> cope with equipment failures), it really was essential to have the overall
> picture. The reason being that a tiny change in one area of the system
> could easily cause big problems in other areas, i.e. it's about unintended
> consequences.
> 
> Here I see several similarities between FDIR and the safety/security for
> LyX, e.g. that adding a neat feature like previews of Gnuplot figures
> could've led to a big security hole.
> 
> Another similarity is that you can't do FDIR only at a low level, you need
> a system perspective.
> I think it's the same with safety/security for LyX: If we only consider
> each feature separately, we're going to screw up. Two features in isolation
> might be safe, but when available together it might "leak".
> 
> Another similarity is needing to define our objectives, and what we
> consider "good enough" safety. If we don't understand what we're trying to
> achieve, it's e.g. not possible to review a design to see if it achieves
> the objectives.

I'm glad you bring this experience to the discussion. You can provide a unique
view that the rest of us might not see.

Scott


signature.asc
Description: PGP signature

Re: Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Scott Kostyshak]

On Wed, Aug 02, 2017 at 12:20:24AM +0200, Christian Ridderström wrote:
> Richard wrote:
> 
> > We have spent an enormous amount of time on this ...
> >
> 
> HI,
> 
> Regarding the discussion of LyX's safety I'd like to make a few remarks
> related to ... ?list etiquette? Not sure what the correct term should be,
> but it ought to be clear below.
> 
> Really long threads:
> Are we really ok with threads containing upwards a hundred posts?
> Perhaps at some point it's necessary to simply start the thread over?
> 
> Thread splitting:
> I've tried to break off topics into separate threads, but it _seems_ like
> it's mainly me doing that.

Ah I did not realize you did that on purpose. I actually found it annoying when
I wanted to go up the discussion and couldn't because it was cut off, but that's
probably because I don't have much experience with thread-splitting. I'd be
happy to change my workflow if others preferred it.

> Or maybe it's how the gmail interface presents the thread to me?
> 
> Is it something we're not doing anymore.
> Should thread splitting be avoided, or should we try to do it more?

It might be useful to know what other mailing lists do and if they have related
policies.

> 
> I'm using gmail's web interface these days. This might be why I'm finding
> it difficult to efficiently follow threads that are so long.
> - The gmail labs thing I used for replying to parts of an e-mail is no
> longer working.
> - Sometimes the replies don't go to the list. *sigh*
> - I haven't figured out how to mark a single e-mail as unread,
>   e.g. when I feel I need to reply to it later, and instead have to mark
> the entire thread.
>   This does _not_ work well for long threads.
> 

I find mutt to be very helpful with emails, but it takes a long time to get used
to, and I'm not sure it would solve the particular issues you reference.

> Are there other things we could've done to do the discussion more
> efficiently?
> 
> Using a wiki page for security topics didn't seem like it's (so far) helped
> anything.
> Would a LyX document in git have been better?
> Or a plain text file?

Good questions, I don't know the answer.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Wed, Aug 02, 2017 at 01:13:14AM +0200, Christian Ridderström wrote:
> On 1 August 2017 at 21:24, Richard Heck  wrote:
> 
> > On 08/01/2017 04:54 AM, Scott Kostyshak wrote:
> > > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> > >> Pavel Sanda wrote:
> > >>> I did not hear your reaction to it either.
> > >> I see you just did that, sorry... P
> > > I believe that except for Enrico and I, everyone who participated in
> > > this conversation has voted.
> 
> 
> Hi Scott,
> 
> For the record, I'm not sure why I'm not included in people who particpated
> but haven't voted.

I meant to mention you and just did not. I'm sorry about that. I really do
appreciate your participation in the discussion.

> Anyway, you're the release manager so your decisions goes.

No, thanks for pointing that out.

Scott

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Tue, Aug 01, 2017 at 03:24:48PM -0400, Richard Heck wrote:
> On 08/01/2017 04:54 AM, Scott Kostyshak wrote:
> > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> >> Pavel Sanda wrote:
> >>> I did not hear your reaction to it either.
> >> I see you just did that, sorry... P
> > I believe that except for Enrico and I, everyone who participated in
> > this conversation has voted. Uwe was interested in voting, but as the
> > vote stands currently, his vote would not change the outcome. I don't
> > have indication that any other LyX developer is interested in voting.
> >
> > The results are the following:
> >
> > 1: Guillaume (hesitantly [1]), .5 Pavel
> > 2: .5 Pavel
> > 3: Kornel, JMarc, Jürgen, Richard
> >
> > Option 3 wins the vote. My decision is to go forward with Enrico's
> > latest patch for beta1.
> >
> > Because the vote was not a blowout, and because I did not represent the
> > opinions of at least Guillaume and Pavel in the options, we will have
> > another vote two weeks after beta1 is released. Each developer may
> > propose one option that will be included in the vote.
> >
> > So to be clear: although we are going to release beta1 with minted
> > support, depending on the post-beta1 vote, we may decide to remove it.
> 
> It's obviously up to you, Scott, as release manager, but I agree with
> Enrico, to some extent,
> that this isn't the best policy. We have spent an enormous amount of
> time on this and finally
> have come to some kind of resolution.

I understand your points and took a long time to make that decision. And since I
have so much respect for Enrico and you, especially given your long experience
as the stable release manager, I will reflect on the decision I made much more
(probably long past the 2.3.0 release).

> It really would be best to close
> that part of the debate
> and spend our time figuring out how to make the needauth and
> shell-escape stuff as secure as
> we can make it, given the present framework. Time spent re-litigating
> the questions about the
> framework is going to be wasted time.
> We can always decide to remove things, especially before a major
> release. If there are serious
> problems or concerns that arise during testing, then those can be raised
> at that point. Or if one
> or more of us who preferred the present course has doubts, we can raise
> those.

> But simply
> having another vote (a rare enough occurrence around here) just because
> people weren't
> entirely happy with how the options were presented doesn't strike me as
> a good idea. There
> is no indication I can see that the people who voted (3) might, let
> alone would, have voted
> differently had the other options been differently described. I think we
> all understood well
> enough what the broad issue was.

If you feel that further discussion is a waste of time, then just don't
participate in the discussion and keep your vote the same. Others who feel
similarly can do the same. That way there is not much time lost for those who
choose to do that. If those who voted in favor of the patch just decide that
they are sure that is the right way forward and no other points or alternatives
would change their mind, then you can all keep your vote there.

This way, we at least move on towards beta1.

Scott

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 1 August 2017 at 21:24, Richard Heck  wrote:

> On 08/01/2017 04:54 AM, Scott Kostyshak wrote:
> > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> >> Pavel Sanda wrote:
> >>> I did not hear your reaction to it either.
> >> I see you just did that, sorry... P
> > I believe that except for Enrico and I, everyone who participated in
> > this conversation has voted.


Hi Scott,

For the record, I'm not sure why I'm not included in people who particpated
but haven't voted.
Anyway, you're the release manager so your decisions goes.


> Uwe was interested in voting, but as the
> > vote stands currently, his vote would not change the outcome. I don't
> > have indication that any other LyX developer is interested in voting.
> >
> > The results are the following:
> >
> > 1: Guillaume (hesitantly [1]), .5 Pavel
> > 2: .5 Pavel
> > 3: Kornel, JMarc, Jürgen, Richard
> >
> > Option 3 wins the vote. My decision is to go forward with Enrico's
> > latest patch for beta1.
> >
> > Because the vote was not a blowout, and because I did not represent the
> > opinions of at least Guillaume and Pavel in the options, we will have
> > another vote two weeks after beta1 is released. Each developer may
> > propose one option that will be included in the vote.
>

As a side note, my opinion was likely also not represented, but I think you
have a feeling for my general thoughts.



> > So to be clear: although we are going to release beta1 with minted
> > support, depending on the post-beta1 vote, we may decide to remove it.
>
> It's obviously up to you, Scott, as release manager, but I agree with
> Enrico, to some extent, that this isn't the best policy.



> We have spent an enormous amount of time on this and finally have come to
> some kind of resolution. It really would be best to close that part of the
> debate and spend our time figuring out how to make the needauth
> and shell-escape stuff as secure as we can make it, given the present
> framework. Time spent re-litigating the questions about the framework is
> going to be wasted time.
>

Hi Richard,

It sounds like you complain about the time spent on discussing what I
assume is not just the minted patch but safety/security in general.
However, regarding spending to much time discussing safety, I can't help
but think about LyX 2.2 where something was obviously missing in terms of
safety.
That leads me to conclude that something was missing, or went awry, in the
development process of LyX 2.2. This, combined with you advocating

 "... spend our time figuring how to make the needauth and shell-escape
stuff as secure
  as we can make it, given the present framework"

makes me a little concerned.

It sounds like you argue for a release of 2.3 regardless of the absolute
achieved level of safety, because:
a) we've anyway done all that can be done using the needauth framework.
b) it's anyway better than LyX 2.2.

Would you mind clarifying your point of view?

Is it perhaps that in your mind the needauth solution is already good
enough, or that you're confident it can be made good enough?
If so, could you perhaps expand on why?

Or do you perhaps consider safety/security less important than other
aspects?
Perhaps compare to releasing before some date, or ensuring ease-of-use for
advanced users or perhaps something else?

Best regards,
/Christian

We can always decide to remove things, especially before a major release.
> If there are serious
> problems or concerns that arise during testing, then those can be
> raised at that point. Or if one
> or more of us who preferred the present course has doubts, we can
> raise those. But simply
> having another vote (a rare enough occurrence around here) just
> because people weren't
> entirely happy with how the options were presented doesn't strike me as a
> good idea. There
> is no indication I can see that the people who voted (3) might, let alone
> would, have voted
> differently had the other options been differently described. I think
> we all understood well
> enough what the broad issue was.
>

Discussing minted/safety/etc (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

Richard wrote:

> We have spent an enormous amount of time on this ...
>

Hi,

Some thoughts regarding the discussion of safety and design of security
measures, i.e. a kind of "lessons learned" regarding the discussion aspects.

I think one thing that made things slower and more inefficient was a lack
of any self-contained description[*] of the overall design regarding safety
and security. Ideally I think the discussion would've benefited from having
one for:
- LyX version 2.2, i.e. the release with the big problems
- What was in master/HEAD, and at least was on its way to become LyX 2.3
- Intended eventual design, for LyX 2.3 or possibly a later release.

The long threads likely by now do contain a lot of separate pieces of
description, but it's unfortunately not so easy (at least for me) to find
this information among all the posts.
And sometimes I'd probably not understand to which "release" the
information pertains.

A partial "fix" might be to create a "best of"-list of posts with useful
descriptive information. Or perhaps to copy them to into one place, adding
minor markup as to what's still valid.
Or actually try to write down some design design description.

Question: Is there anyone who feels they know/understand the big picture
regarding safety/security?
Because I guess we in theory instead of writing things down could have a
designated "guru".

Actually, I'm a little worried that no one really has the big picture of
the intended design regarding safety and security.

Also, that different people have different ideas of where we are going and
what's considered needed, while we at the same time are not aware of these
differences.

For something (a "feature") much less complicated than safety/security, I
think it'd be fine for LyX if only a very small number of people know that
part, and to where they intend to go. And that everything else is in the
code. But for a complicated topic like safety, and when asking people to
vote, I think the lack of a description is a problem.

Hmm, and perhaps in case of votes, Scott should (be allowed to) weigh them
differently depending on the persons knowledge. E.g., since I don't
know/understand the (intended) safety design, why should my vote count as
much as a potential safety "guru".

On a more practical level:
Hindsight is 20-20, but regarding e.g. Enrico's patch, perhaps we should
have created a branch representing that alternative?   This would have
allowed that branch to always represent the latest improvement, thus
avoiding people testing out an older patch accidentally.
So we should keep the possibility of using branches in mind for future
discussions.

Perhaps there's other more practical things we could've done to make life
easier.

Best regards,
/Christian

[*] Regarding SW design descriptions and actually being able to understand
and review an intended design in advance, I'm probably a bit "damaged" from
when I worked with satellite software (AOCS), e.g. as SW verification and
validation manager.

In that field, lots of documentation was required which often was annoying,
but for complicated stuff like FDIR (making the SW/satellite be able to
cope with equipment failures), it really was essential to have the overall
picture. The reason being that a tiny change in one area of the system
could easily cause big problems in other areas, i.e. it's about unintended
consequences.

Here I see several similarities between FDIR and the safety/security for
LyX, e.g. that adding a neat feature like previews of Gnuplot figures
could've led to a big security hole.

Another similarity is that you can't do FDIR only at a low level, you need
a system perspective.
I think it's the same with safety/security for LyX: If we only consider
each feature separately, we're going to screw up. Two features in isolation
might be safe, but when available together it might "leak".

Another similarity is needing to define our objectives, and what we
consider "good enough" safety. If we don't understand what we're trying to
achieve, it's e.g. not possible to review a design to see if it achieves
the objectives.

Long threads? / list etiquette? (Was: Options for resolving the minted + shell-escape issue)

[Christian Ridderström]

Richard wrote:

> We have spent an enormous amount of time on this ...
>

HI,

Regarding the discussion of LyX's safety I'd like to make a few remarks
related to ... ?list etiquette? Not sure what the correct term should be,
but it ought to be clear below.

Really long threads:
Are we really ok with threads containing upwards a hundred posts?
Perhaps at some point it's necessary to simply start the thread over?

Thread splitting:
I've tried to break off topics into separate threads, but it _seems_ like
it's mainly me doing that.
Or maybe it's how the gmail interface presents the thread to me?

Is it something we're not doing anymore.
Should thread splitting be avoided, or should we try to do it more?


I'm using gmail's web interface these days. This might be why I'm finding
it difficult to efficiently follow threads that are so long.
- The gmail labs thing I used for replying to parts of an e-mail is no
longer working.
- Sometimes the replies don't go to the list. *sigh*
- I haven't figured out how to mark a single e-mail as unread,
  e.g. when I feel I need to reply to it later, and instead have to mark
the entire thread.
  This does _not_ work well for long threads.


Are there other things we could've done to do the discussion more
efficiently?

Using a wiki page for security topics didn't seem like it's (so far) helped
anything.
Would a LyX document in git have been better?
Or a plain text file?
/Christian

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

On 1 August 2017 at 01:25, Pavel Sanda  wrote:

> Christian Ridderström wrote:
> > Please note that I'm _not_ wholly against something like needauth, I'm
> simply
> > not convinced it's good enough.  In fact, I'm still unclear on exactly
> how it
> > currently works, or perhaps rather, how it's intended to work in LyX 2.3.
>
> I already wrote you possible ways how we can cement the current situation
> more
> and IIRC you did not picked any.


Would you mind pointing me to that post?
I don't know if I simply missed it or was unable to read/consider/reply at
the time.

Meanwhile Tommaso committed patch which is
> slightly improving situation by "scary dialog". I did not hear your
> reaction to
> it either.
>

As you noticed in a later post, I did react to that.


> > > Even after all discussion I still see adding the whole needauth
> machinery
> > > as unnecessary complication of code and UI; possible future use of
> pygments
> > > still seems as made up argument for the sake of discussion rather than
> real
> > > user demand.
> >
> > Would you mind clarifying why needauth is an "unneccessary complication
> of
> > code and UI"?  (Apologies as I'm likely asking you to repeat what you've
> said
> > previously).
>
> I meant extending needauth mechanisms beyond current gnuplot/knitr usage
> for the sake of minted, because I believe minted maintainer will deliver
> us fix within couple months.
>

Ok, I understand what you mean.


> With such sight in view, I would be just fine to let minted support in
> current fashion, tell people clearly in manual that adding --shell-escape
> is dangerous, let Uwe scream at us little bit and then be done with it.
>

Ok.

Guess it comes down to the number of expected minted users. If not very
high, it's a good enough solution.
If there'd be lots of minted users, risk increase that they accidentally
leave --shell-escape in place.
/Christian

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 08/01/2017 04:54 AM, Scott Kostyshak wrote:
> On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
>> Pavel Sanda wrote:
>>> I did not hear your reaction to it either.
>> I see you just did that, sorry... P
> I believe that except for Enrico and I, everyone who participated in
> this conversation has voted. Uwe was interested in voting, but as the
> vote stands currently, his vote would not change the outcome. I don't
> have indication that any other LyX developer is interested in voting.
>
> The results are the following:
>
> 1: Guillaume (hesitantly [1]), .5 Pavel
> 2: .5 Pavel
> 3: Kornel, JMarc, Jürgen, Richard
>
> Option 3 wins the vote. My decision is to go forward with Enrico's
> latest patch for beta1.
>
> Because the vote was not a blowout, and because I did not represent the
> opinions of at least Guillaume and Pavel in the options, we will have
> another vote two weeks after beta1 is released. Each developer may
> propose one option that will be included in the vote.
>
> So to be clear: although we are going to release beta1 with minted
> support, depending on the post-beta1 vote, we may decide to remove it.

It's obviously up to you, Scott, as release manager, but I agree with
Enrico, to some extent,
that this isn't the best policy. We have spent an enormous amount of
time on this and finally
have come to some kind of resolution. It really would be best to close
that part of the debate
and spend our time figuring out how to make the needauth and
shell-escape stuff as secure as
we can make it, given the present framework. Time spent re-litigating
the questions about the
framework is going to be wasted time.

We can always decide to remove things, especially before a major
release. If there are serious
problems or concerns that arise during testing, then those can be raised
at that point. Or if one
or more of us who preferred the present course has doubts, we can raise
those. But simply
having another vote (a rare enough occurrence around here) just because
people weren't
entirely happy with how the options were presented doesn't strike me as
a good idea. There
is no indication I can see that the people who voted (3) might, let
alone would, have voted
differently had the other options been differently described. I think we
all understood well
enough what the broad issue was.

Richard

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Aug 01, 2017 at 04:54:05AM -0400, Scott Kostyshak wrote:
> 
> So to be clear: although we are going to release beta1 with minted
> support, depending on the post-beta1 vote, we may decide to remove it.

Sorry, but this is not fair. The option to remove support has been
just voted and it didn't pass. This is astonishing. Really.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Dienstag, 1. August 2017 um 11:10:09, schrieb Stephan Witt 
> Am 01.08.2017 um 10:54 schrieb Scott Kostyshak :
> > 
> > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> >> Pavel Sanda wrote:
> >>> I did not hear your reaction to it either.
> >> 
> >> I see you just did that, sorry... P
> > 
> > I believe that except for Enrico and I, everyone who participated in
> > this conversation has voted. Uwe was interested in voting, but as the
> > vote stands currently, his vote would not change the outcome. I don't
> > have indication that any other LyX developer is interested in voting.
> 
> Sorry for being late in the game - I followed the discussion closely and
> tried to make my mind on this. I patched LyX, installed the Python module
> plus pygmentize executable and tried to typeset the minted-listings example.
> 
> I had difficulties to understand the different options. Why? It worked
> on my side only after manually adding -shell-escape to the pdflatex converter.
> This I didn’t expect after the discussion.
> 
> Did I miss anything? If yes, then it would be good for the next round of 
> voting
> to give more precise hints how to test the discussed feature.

I suppose you missed the patch "shell-escape-auth-5.diff" from Enrico.
See https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg201331.html

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Tue, Aug 01, 2017 at 11:10:09AM +0200, Stephan Witt wrote:
> Am 01.08.2017 um 10:54 schrieb Scott Kostyshak :
> > 
> > On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> >> Pavel Sanda wrote:
> >>> I did not hear your reaction to it either.
> >> 
> >> I see you just did that, sorry... P
> > 
> > I believe that except for Enrico and I, everyone who participated in
> > this conversation has voted. Uwe was interested in voting, but as the
> > vote stands currently, his vote would not change the outcome. I don't
> > have indication that any other LyX developer is interested in voting.
> 
> Sorry for being late in the game - I followed the discussion closely and
> tried to make my mind on this. I patched LyX, installed the Python module
> plus pygmentize executable and tried to typeset the minted-listings example.
> 
> I had difficulties to understand the different options. Why? It worked
> on my side only after manually adding -shell-escape to the pdflatex converter.
> This I didn’t expect after the discussion.
> 
> Did I miss anything? If yes, then it would be good for the next round of 
> voting
> to give more precise hints how to test the discussed feature.

Thanks for testing the patch and for asking the question. The idea of
the patch is: adding -shell-escape automatically with minted support
could be viewed as dangerous (does the user really know what they are
doing?). So the viewer must check the box in document settings that
allows running external programs for that document. The advantage of
this is that you do not have to add -shell-escape globally to the
converter (which you might forget to remove).

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Stephan Witt]

Am 01.08.2017 um 10:54 schrieb Scott Kostyshak :
> 
> On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
>> Pavel Sanda wrote:
>>> I did not hear your reaction to it either.
>> 
>> I see you just did that, sorry... P
> 
> I believe that except for Enrico and I, everyone who participated in
> this conversation has voted. Uwe was interested in voting, but as the
> vote stands currently, his vote would not change the outcome. I don't
> have indication that any other LyX developer is interested in voting.

Sorry for being late in the game - I followed the discussion closely and
tried to make my mind on this. I patched LyX, installed the Python module
plus pygmentize executable and tried to typeset the minted-listings example.

I had difficulties to understand the different options. Why? It worked
on my side only after manually adding -shell-escape to the pdflatex converter.
This I didn’t expect after the discussion.

Did I miss anything? If yes, then it would be good for the next round of voting
to give more precise hints how to test the discussed feature.

Stephan

> 
> The results are the following:
> 
> 1: Guillaume (hesitantly [1]), .5 Pavel
> 2: .5 Pavel
> 3: Kornel, JMarc, Jürgen, Richard
> 
> Option 3 wins the vote. My decision is to go forward with Enrico's
> latest patch for beta1.
> 
> Because the vote was not a blowout, and because I did not represent the
> opinions of at least Guillaume and Pavel in the options, we will have
> another vote two weeks after beta1 is released. Each developer may
> propose one option that will be included in the vote.
> 
> So to be clear: although we are going to release beta1 with minted
> support, depending on the post-beta1 vote, we may decide to remove it.
> 
> Thank you to everyone for your time on this issue.
> 
> Scott
> 
> 
> [1] 
> https://www.mail-archive.com/search?l=mid=olnv4p%24jjn%241%40blaine.gmane.org

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Mon, Jul 31, 2017 at 11:43:04AM +0200, Enrico Forestieri wrote:

> Corrected at 44babaf6.

Thanks, that works well. The only other comment I have is to possibly
instruct the user to reconfigure after installing the module. If I
recall, that's what we say in similar contexts.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Tue, Aug 01, 2017 at 02:35:27AM +0200, Pavel Sanda wrote:
> Pavel Sanda wrote:
> > I did not hear your reaction to it either.
> 
> I see you just did that, sorry... P

I believe that except for Enrico and I, everyone who participated in
this conversation has voted. Uwe was interested in voting, but as the
vote stands currently, his vote would not change the outcome. I don't
have indication that any other LyX developer is interested in voting.

The results are the following:

1: Guillaume (hesitantly [1]), .5 Pavel
2: .5 Pavel
3: Kornel, JMarc, Jürgen, Richard

Option 3 wins the vote. My decision is to go forward with Enrico's
latest patch for beta1.

Because the vote was not a blowout, and because I did not represent the
opinions of at least Guillaume and Pavel in the options, we will have
another vote two weeks after beta1 is released. Each developer may
propose one option that will be included in the vote.

So to be clear: although we are going to release beta1 with minted
support, depending on the post-beta1 vote, we may decide to remove it.

Thank you to everyone for your time on this issue.

Scott


[1] 
https://www.mail-archive.com/search?l=mid=olnv4p%24jjn%241%40blaine.gmane.org


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Tue, Aug 01, 2017 at 12:34:56AM +0200, Christian Ridderström wrote:

> So I think a very important question is if we think needauth is
> sufficiently good,
> or can be made sufficiently good before release.

My opinion is that yes.

> IMHO it's for LyX 2.3 not ok to do a release that's unsafe just because
> it's "less" unsafe
> than before, i.e. the release has to be sufficiently good, not just less
> bad than before.

The problem is that "sufficiently good" is an opinion. But yes, I agree
that "less unsafe" is not on its own good enough.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Mon, Jul 31, 2017 at 09:07:11PM +0200, Guillaume MM wrote:

> I am sure that Scott meant to include in some way the option that I have
> been advocating constantly from the beginning, which I understand is
> probably 1. (Otherwise, I do not see what the option 1. refers to nor
> who proposed it, and I would opt for not taking part in the vote.)

Yes my intention was to represent your opinion in a simplified way. I'm
sorry to have failed that. You will have an opportunity to write your
own option for the next vote (see separate email).

> So to
> be clear, my vote is 1., and the patch gives it substance as part of a
> larger proposition providing the most secure option for beta and for
> release so far (because after minted there remains to fix needauth). It
> just incidentally happens that it does not lock one into removing minted
> by providing a better basis for what people voting 3. are trying to
> achieve.


> (In particular Scott, even if the route of 3. is voted for
> beta, this patch minus disabling of minted remains an option for 2.3.)

Yes, I agree.

> As for your counting of votes, calling the end of the poll, and deciding
> what fits or does not fit proposed options,

I should have fixed an amount of days (or some other stopping criteria)
when I opened the poll. I am taking notes of things I have done poorly,
and will try to improve things in the future.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Pavel Sanda wrote:
> I did not hear your reaction to it either.

I see you just did that, sorry... P

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Christian Ridderström wrote:
> Please note that I'm _not_ wholly against something like needauth, I'm simply
> not convinced it's good enough.  In fact, I'm still unclear on exactly how it
> currently works, or perhaps rather, how it's intended to work in LyX 2.3.

I already wrote you possible ways how we can cement the current situation more
and IIRC you did not picked any. Meanwhile Tommaso committed patch which is
slightly improving situation by "scary dialog". I did not hear your reaction to
it either.

> > Even after all discussion I still see adding the whole needauth machinery
> > as unnecessary complication of code and UI; possible future use of pygments
> > still seems as made up argument for the sake of discussion rather than real
> > user demand.
> 
> Would you mind clarifying why needauth is an "unneccessary complication of
> code and UI"?  (Apologies as I'm likely asking you to repeat what you've said
> previously).

I meant extending needauth mechanisms beyond current gnuplot/knitr usage
for the sake of minted, because I believe minted maintainer will deliver
us fix within couple months.

With such sight in view, I would be just fine to let minted support in
current fashion, tell people clearly in manual that adding --shell-escape
is dangerous, let Uwe scream at us little bit and then be done with it.

Pavel

Re: Options for resolving the minted + shell-escape issue

[Christian Ridderström]

Hi Scott,

On 31 July 2017 at 16:50, Pavel Sanda  wrote:

> Scott Kostyshak wrote:
> > I'm concerned that since this issue has left us all exhausted, there is
> > a feeling of "let's just get this over so we can move on". I encourage
> > all of us to give one more cognitive spurt and give a vote.
> >
> > From what I understand, the three options are still what I proposed
> > three weeks ago [1]:
> >
> > 1. Revert the recently added minted support.
> >
> > 2. Keep the current state of master.
> >
> > 3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
> > to lib/images. (Note that I get linker errors when I try to apply the
> > latest patch, but it might be something specific to my setup.)
> >
> > So I ask explicitly to everyone (even if you think you have already
> > voted, please give your vote again):
>

I'm just letting you know I haven't had time to look through all the recent
posts yet.

Further, and more importantly for me, the above is only for minted.
To me it doesn't make sense to not consider 'needauth' in the same context
of safety/security.

So I'm not sure I can vote as I'm not sure the alternatives make sense.

The following is likely not going to be popular, but ...

If we don't have mechanisms that keep "regular" users safe, I don't think
it matters
very much what we do with minted.

So I think a very important question is if we think needauth is
sufficiently good,
or can be made sufficiently good before release.

I do think there should, at least formally, be some kind of alternative(s)
corresponding to e.g.
removing/disabling needauth _and_ removing/disabling the automatic/silent
running of R-scripts/gnuplot,
and that it should take some "effort"/"be scary" to enable them.

IMHO it's for LyX 2.3 not ok to do a release that's unsafe just because
it's "less" unsafe
than before, i.e. the release has to be sufficiently good, not just less
bad than before.

Please note that I'm _not_ wholly against something like needauth, I'm
simply not convinced it's good enough.
In fact, I'm still unclear on exactly how it currently works, or perhaps
rather, how it's intended to work in LyX 2.3.


> Little difficult, because what I opined was not included in your list.
> To sum up I favor support of minted, which would use secure calling
> minted once it's compilation is split into separated steps as proposed
> on minted bugzilla.
> So I do not have strong opinion whether we shoould go 1 or 2 if we
> fix the issues once minted is fixed.
>
> Even after all discussion I still see adding the whole needauth machinery
> as
> unnecessary complication of code and UI; possible future use of pygments
> still seems as made up argument for the sake of discussion rather than real
> user demand.
>

Would you mind clarifying why needauth is an "unneccessary complication of
code and UI"?
(Apologies as I'm likely asking you to repeat what you've said previously).

Best regards,
/Christian


>
> So the breakdown is likely 0.5 voting points for 1 & 2.
>
> Pavel
>

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Mon, Jul 31, 2017 at 09:07:11PM +0200, Guillaume MM wrote:
> 
> I am sure that Scott meant to include in some way the option that I have
> been advocating constantly from the beginning, which I understand is
> probably 1. (Otherwise, I do not see what the option 1. refers to nor
> who proposed it, and I would opt for not taking part in the vote.) So to
> be clear, my vote is 1., and the patch gives it substance as part of a
> larger proposition providing the most secure option for beta and for
> release so far (because after minted there remains to fix needauth). It
> just incidentally happens that it does not lock one into removing minted
> by providing a better basis for what people voting 3. are trying to
> achieve. (In particular Scott, even if the route of 3. is voted for
> beta, this patch minus disabling of minted remains an option for 2.3.)

Sorry, but 1. means what it says and not what meaning you want to attach
to it. So, you vote for 1.

> As for your counting of votes, calling the end of the poll, and deciding
> what fits or does not fit proposed options, last time I have checked you
> were not release manager :)

You don't even deserve an answer here.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Guillaume MM]

Le 31/07/2017 à 13:00, Enrico Forestieri a écrit :

On Mon, Jul 31, 2017 at 12:01:39PM +0200, Enrico Forestieri wrote:


On Mon, Jul 31, 2017 at 10:57:01AM +0200, Guillaume MM wrote:


A key point to me is that, according to your other message, going with
3. locks 2.3 into implementing Pygments using minted. For this reason
and that it does not solve fundamental issues, there are more reasonable
solutions than the patch at [2] even understood as an intermediary
solution to get beta out.


I now count 4 votes for the patch at [2] and only one against it
(if I correctly interpret this post as such). I think this means that a
decision can be taken without entering in other exhausting and unfruitful
discussions that most probably would go on forever.


Just to be clear. It shouldn't and cannot work this way. There were a vote
called on some options. One cannot come up with another option when he sees
that this is not what he would like. Otherwise, another one is entitled
to come up with yet another option and so on. We have to make a decision
(which is now possible, I think) and then move from there.



I am sure that Scott meant to include in some way the option that I have
been advocating constantly from the beginning, which I understand is
probably 1. (Otherwise, I do not see what the option 1. refers to nor
who proposed it, and I would opt for not taking part in the vote.) So to
be clear, my vote is 1., and the patch gives it substance as part of a
larger proposition providing the most secure option for beta and for
release so far (because after minted there remains to fix needauth). It
just incidentally happens that it does not lock one into removing minted
by providing a better basis for what people voting 3. are trying to
achieve. (In particular Scott, even if the route of 3. is voted for
beta, this patch minus disabling of minted remains an option for 2.3.)

As for your counting of votes, calling the end of the poll, and deciding
what fits or does not fit proposed options, last time I have checked you
were not release manager :)

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Scott Kostyshak wrote:
> I'm concerned that since this issue has left us all exhausted, there is
> a feeling of "let's just get this over so we can move on". I encourage
> all of us to give one more cognitive spurt and give a vote.
> 
> From what I understand, the three options are still what I proposed
> three weeks ago [1]:
>
> 1. Revert the recently added minted support.
> 
> 2. Keep the current state of master.
>
> 3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
> to lib/images. (Note that I get linker errors when I try to apply the
> latest patch, but it might be something specific to my setup.)
>
> So I ask explicitly to everyone (even if you think you have already
> voted, please give your vote again):

Little difficult, because what I opined was not included in your list.
To sum up I favor support of minted, which would use secure calling
minted once it's compilation is split into separated steps as proposed
on minted bugzilla.
So I do not have strong opinion whether we shoould go 1 or 2 if we
fix the issues once minted is fixed.

Even after all discussion I still see adding the whole needauth machinery as
unnecessary complication of code and UI; possible future use of pygments
still seems as made up argument for the sake of discussion rather than real
user demand.

So the breakdown is likely 0.5 voting points for 1 & 2.

Pavel

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Montag, 31. Juli 2017 um 12:51:15, schrieb Enrico Forestieri 
> On Sun, Jul 30, 2017 at 02:27:02PM +0200, Kornel Benko wrote:
> > 
> > Testing the patch shell-escape-auth-5.diff, there is 1 issue from my side:
> > The session entry is totally ignored if using 'lyx -E', therefore I'd prefer
> > to save the entries in some other file (for instance "session.shellEscape"),
> > which could be loaded unconditionally.
> 
> I now actually checked and the session entry is not ignored. Maybe the
> issue you are seeing is due to the fact that the document was not
> explicitly authorized. This is indicated by a trailing ",1" instead
> of ",0". In this last case, explicit authorization is required and the
> batch command fails because this authorization is not given by default.
> 

I checked too Enrico. Exporting manually works.

# lyx2.3 -userdir 
"/BUILD/BUILDMint17/BuildLyxGitQt5.8self-gcc7.1/Testing/.lyx" -E pdf2 
minted-listings.pdf /usr2/src/lyx/lyx-git/lib/examples/minted-listings.lyx

But exporting from ctest does only if instrumenting the converters.
Here is, what I see (output got setting env var LYX_DEBUG_LATEX=1.

...
Executing /BUILD/BUILDMint17/BuildLyxGitQt5.8self-gcc7.1/bin/lyx2.3 -dbg latex 
-userdir "/BUILD/BUILDMint17/BuildLyxGitQt5.8self-gcc7.1/Testing/.lyx" -E pdf2 
minted-listings_defaultF.pdf2 
"/BUILD/BUILDMint17/BuildLyxGitQt5.8self-gcc7.1/autotests/out-home/AbC_Q0tzSx/minted-listings_pdf2_defaultF.lyx"
Setting debug level to latex
...
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded 
format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./minted-listings_pdf2_defaultF.tex
LaTeX2e <2017-04-15>
Babel <3.12> and hyphenation patterns for 84 language(s) loaded.

support/Systemcall.cpp (294): Systemcall: 'pdflatex  
"minted-listings_pdf2_defaultF.tex"' finished with exit code 1
LaTeX.cpp (652): Log file: minted-listings_pdf2_defaultF.log
LaTeX.cpp (679): Log line: This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX 
Live 2017) (preloaded format=pdflatex 2017.7.2)  31 JUL 2017 13:47
LaTeX.cpp (679): Log line: entering extended mode
LaTeX.cpp (679): Log line:  restricted \write18 enabled.
LaTeX.cpp (679): Log line:  %&-line parsing enabled.
LaTeX.cpp (679): Log line: **minted-listings_pdf2_defaultF.tex
...
LaTeX.cpp (679): Log line: ) 
(/usr9/local/texlive/2017/texmf-dist/tex/generic/oberdiek/catchfile.sty
LaTeX.cpp (679): Log line: Package: catchfile 2016/05/16 v1.7 Catch the 
contents of a file (HO)
LaTeX.cpp (679): Log line: )
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line: Package ifplatform Warning:
LaTeX.cpp (679): Log line: shell escape is disabled, so I can only detect 
\ifwindows.
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line: ) 
(/usr9/local/texlive/2017/texmf-dist/tex/generic/xstring/xstring.sty (/usr9/l
LaTeX.cpp (679): Log line: 
ocal/texlive/2017/texmf-dist/tex/generic/xstring/xstring.tex
...
LaTeX.cpp (679): Log line: runsystem(mkdir -p 
_minted-minted-listings_pdf2_defaultF)...disabled (restricte
LaTeX.cpp (679): Log line: d).
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line: ! Package minted Error: You must invoke LaTeX with 
the -shell-escape flag.
LaTeX.cpp (877): line: 18
Desc: Package minted Error: You must invoke LaTeX with the -shell-escape flag.
Text:  \setminted
   {breaklines=true}
Pass the -shell-escape flag to LaTeX. Refer to the minted.sty documentation for
 more information.

LaTeX.cpp (679): Log line: runsystem(which pygmentize && touch 
minted-listings_pdf2_defaultF.aex)...disabl
LaTeX.cpp (679): Log line: ed (restricted).
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line:
LaTeX.cpp (679): Log line: ! Package minted Error: You must have `pygmentize' 
installed to use this packag
LaTeX.cpp (877): line: 21
Desc: Package minted Error: You must have `pygmentize' installed to use this 
packag
...

So, what went wrong?

Ouch, how stupid am I!
Now I see it. The tested file-name is .../minted-listings_pdf2_defaultF.lyx and 
not .../minted-listings.lyx (which _is_ set in the session file)

The problem here is, I am unable to instrument lyx to use the newly created 
file name in the session because
there can be many parallel running exports.

Not your problem of course.

Kornel


signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Mon, Jul 31, 2017 at 12:01:39PM +0200, Enrico Forestieri wrote:

> On Mon, Jul 31, 2017 at 10:57:01AM +0200, Guillaume MM wrote:
> 
> > A key point to me is that, according to your other message, going with
> > 3. locks 2.3 into implementing Pygments using minted. For this reason
> > and that it does not solve fundamental issues, there are more reasonable
> > solutions than the patch at [2] even understood as an intermediary
> > solution to get beta out.
> 
> I now count 4 votes for the patch at [2] and only one against it
> (if I correctly interpret this post as such). I think this means that a
> decision can be taken without entering in other exhausting and unfruitful
> discussions that most probably would go on forever.

Just to be clear. It shouldn't and cannot work this way. There were a vote
called on some options. One cannot come up with another option when he sees
that this is not what he would like. Otherwise, another one is entitled
to come up with yet another option and so on. We have to make a decision
(which is now possible, I think) and then move from there.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Sun, Jul 30, 2017 at 02:27:02PM +0200, Kornel Benko wrote:
> 
> Testing the patch shell-escape-auth-5.diff, there is 1 issue from my side:
> The session entry is totally ignored if using 'lyx -E', therefore I'd prefer
> to save the entries in some other file (for instance "session.shellEscape"),
> which could be loaded unconditionally.

I now actually checked and the session entry is not ignored. Maybe the
issue you are seeing is due to the fact that the document was not
explicitly authorized. This is indicated by a trailing ",1" instead
of ",0". In this last case, explicit authorization is required and the
batch command fails because this authorization is not given by default.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Mon, Jul 31, 2017 at 10:57:01AM +0200, Guillaume MM wrote:

> A key point to me is that, according to your other message, going with
> 3. locks 2.3 into implementing Pygments using minted. For this reason
> and that it does not solve fundamental issues, there are more reasonable
> solutions than the patch at [2] even understood as an intermediary
> solution to get beta out.

I now count 4 votes for the patch at [2] and only one against it
(if I correctly interpret this post as such). I think this means that a
decision can be taken without entering in other exhausting and unfruitful
discussions that most probably would go on forever.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Guillaume MM]

Le 31/07/2017 à 10:57, Guillaume MM a écrit :


I spotted various areas of improvement:

...

* The new buffer param and the boolean argument in the session file are
superfluous and make the code much more complicated than needed.

...

I now understand better that this is by design, but it (still) does not
seem very natural to me. There are three levels, because there is
1. a switch to enable shell-escape (after which we are still given a
further dialog at compilation before actually running with shell-escape).
2. a button on this dialog to further "always allow".

The red icon on the bottom right is given when 1. is active even though
one is still safe as long as 2. has not been pressed. With e.g. Sweave
the first switch is automatic and innocuous (there are Sweave insets, so
ask for running sweave). Then with my patch the icon informs when one
has authorized to never ask anymore, so one is in an actual unsafe state.

With Enrico's patch, somebody using minted will always check the switch
1. and from this point on he always sees the red icon, so the icon
provides zero information. The contextual menu reverts the choice of
converters, which is pointless because it just makes the
document uncompilable. There is no option to see just revoke 2. or to
see the status of 2.

As a conclusion, as it is, the status bar icon in this patch is not an
implementation of visibility nor revocability. For it to be so, the icon
should be shown when 2. is active, and the context menu should switch 2.
instead of 1.

If furthermore a validation mechanism was provided for minted requiring
shell-escape, then one would just need 2 levels as with needauth.

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Sun, Jul 30, 2017 at 11:01:18PM +0200, Enrico Forestieri wrote:

> On Sat, Jul 29, 2017 at 07:51:08PM -0400, Scott Kostyshak wrote:
> > 
> > If I remove the python-pygments module (and reconfigure) and test, I get
> > the following error:
> > 
> > The driver command necessary to use the minted package
> > (pygmentize) has not been found. Make sure you have
> > the python-pygments module installed or, if the driver
> > is named differently, to add the following line to the
> > document preamble:
> > 
> > \AtBeginDocument{\renewcommand{\MintedPygmentize}{driver}}
> > 
> > where 'driver' is name of the driver command.
> > 
> > I expected to get the error, but what I was surprised by is that I get
> > the error when going to Document > Settings. I expected to get an error
> > immediately after compiling (before the LaTeX log errors). Is that
> > expected? Note that I am using a compiled Qt 5.9.2dev so if you can't
> > reproduce, it is probably a bug on my side.
> 
> No, it is not related to the Qt version. I was aware of it but further
> improvements were delayed by the exhausting discussion that followed.

Corrected at 44babaf6.

> I am contemplating the possibility of simply not allowing minted if the
> driver program is not called pygmentize, so that in this case the combo
> box is simply disabled without any warning.

This is still an option, even if I deem it to be inflexible.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Guillaume MM]

Le 29/07/2017 à 23:57, Scott Kostyshak a écrit :

On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:

On 07/20/2017 01:21 AM, Jürgen Spitzmüller wrote:

Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:

The attached patch takes into account all of these ideas. As a
disclaimer,
note that I am providing it only because I am now familiar with this
part
of the code and can quickly come up with a patch. But I am not
endorsing it.

I propose to apply this patch and return to productivity.


I would agree with that.


I'm concerned that since this issue has left us all exhausted, there is
a feeling of "let's just get this over so we can move on". I encourage
all of us to give one more cognitive spurt and give a vote.

 From what I understand, the three options are still what I proposed
three weeks ago [1]:

1. Revert the recently added minted support.

2. Keep the current state of master.

3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
to lib/images. (Note that I get linker errors when I try to apply the
latest patch, but it might be something specific to my setup.)

Hi Scott,


A key point to me is that, according to your other message, going with
3. locks 2.3 into implementing Pygments using minted. For this reason
and that it does not solve fundamental issues, there are more reasonable
solutions than the patch at [2] even understood as an intermediary
solution to get beta out.

Taking all the discussions into account, the most "usably secure"
overall solution (the criteria I have been advocating and you seem to
have made yours) is still the one I have suggested more than a month ago:

https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg200878.html

You do not reflect it in your list I think because you have been
trying to simplify the discussion (which I am not sure was really
fruitful unfortunately). Attached is a patch that implements a solution
along these lines, freely based on the patch at [2]. If your criteria is
really to get the most secure option, you will consider this:

First, improve the security of needauth by implementing the visibility
and revocability as in the patch at [2] (an icon in the status bar with
tooltip and right-click menu to unauthorize, as well as a checkbox in
the document properties). A lot of voices agreed that this was a
desirable improvement for needauth. This makes it more reasonable to
ship with new unsafe converters, and it also makes needauth a more
durable solution for unsafe converters in general. (Though there remain
issues that have been listed, for instance how to handle graphics
preview and making the authorization per-converter.)

It also revokes authorization after moving/deleting the file, which I
just realized was in fact already part of Tommaso's implementation.

Then, I did not implement shell-escape as in the patch at [2] (naively
this would have meant a second icon and a second checkbox). Several
voices including mine found the solution not satisfactory: ideally there
should be only one mechanism and one should not hard-code the addition
of shell-escape. I believe that instead it should be reflected in some
way in the converter settings. But there is no immediate need for
shell-escape, because in the second patch attached I am disabling minted.

This does not mean that there cannot be a nice solution for shell-escape
in 2.3, on the contrary. Starting from my patch, an integrated
shell-escape implementation should be easy following the idea I gave in
 if
one wants to, and there are probably many other ways. So, this patch
forms the best basis to build on for 2.3.

Moreover, the old way of adding shell-escape leads to a dialog
interrupting the compilation (based on an idea in [2]), so that
3rd-party guides will need to be updated to take needauth into account,
and old unsafe configurations are deprecated.

Given the absence of preconfigured shell-escape for now, fixable for
2.3, minted is disabled. This is done using the same proven technique
that has allowed me to propose a branch of master reading and writing
the stable file format over the past few months, and is a lightweight
solution until a satisfactory solution is found or a decision is taken.
I am not convinced with Pavel's suggestion "drop shell-escape later"
because one cannot commit to a technical choice based on a promise when
there are several plausible alternatives, but all options are left open.
Of course one could also revert immediately independently
of the final choice if one prefers. So, this patch does not lock into a
premature choice.

Lastly, I am ready to endorse my own patch because I believe it provides
the sanest basis for needauth and the further improvements it requires.




So I ask explicitly to everyone (even if you think you have already
voted, please give your vote again): who has given the latest patch a
good test, has looked at the code (the patch 

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Sonntag, 30. Juli 2017 um 23:15:53, schrieb Enrico Forestieri 

> On Sun, Jul 30, 2017 at 02:27:02PM +0200, Kornel Benko wrote:
> > 
> > Testing the patch shell-escape-auth-5.diff, there is 1 issue from my side:
> > The session entry is totally ignored if using 'lyx -E', therefore I'd prefer
> > to save the entries in some other file (for instance "session.shellEscape"),
> > which could be loaded unconditionally.
> 
> I am wondering whether it is better this way, security wise.

Well, it is easier to use this for tests. The way I test now is to add 
-shell-escape to _all_ latex converter.

> When you
> export from command line you don't get any clue that latex is allowed
> to run external programs.

For this we have to know which lyx file will need it.
Doable, but would also need extra info somewhere else.

> Perhaps a further line command option could
> be introduced, say --shell-escape, that in this case would be an explicit
> consent. This option would be effective only for batch jobs and ignored
> in normal operation.
> 

Not in my reach ATM. I try to make the tests for existing lyx.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

The latest version of Enrico's patches look good to me.

R


On 07/29/2017 05:57 PM, Scott Kostyshak wrote:
> On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:
>> On 07/20/2017 01:21 AM, Jürgen Spitzmüller wrote:
>>> Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:
 The attached patch takes into account all of these ideas. As a
 disclaimer,
 note that I am providing it only because I am now familiar with this
 part
 of the code and can quickly come up with a patch. But I am not
 endorsing it.
>>> I propose to apply this patch and return to productivity.
>> I would agree with that.
> I'm concerned that since this issue has left us all exhausted, there is
> a feeling of "let's just get this over so we can move on". I encourage
> all of us to give one more cognitive spurt and give a vote.
>
> From what I understand, the three options are still what I proposed
> three weeks ago [1]:
>
> 1. Revert the recently added minted support.
>
> 2. Keep the current state of master.
>
> 3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
> to lib/images. (Note that I get linker errors when I try to apply the
> latest patch, but it might be something specific to my setup.)
>
> So I ask explicitly to everyone (even if you think you have already
> voted, please give your vote again): who has given the latest patch a
> good test, has looked at the code (the patch is not so long), and votes
> in favor of one of the three options?
>
> Please also feel free to say that you have tested the patch but choose
> to abstain from voting. This way I will know that even if the vote is
> only 2 to 1, at least the rest of the developers are indifferent, rather
> than not having tested. But I ask that everyone tests the patch and says
> something. If you are lost because you had the fortune of not following
> the debate, let us know so we can tell you what to test, and can point
> you to some advantages/disadvantages of each option.
>
> Scott
>
>
> [1]
> https://www.mail-archive.com/search?l=mid=20170705045915.h4uyrsc27g54da3m%40steph
> [2]
> https://www.mail-archive.com/search?l=mid=20170728213142.GA7880%40GIOVE

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Sun, Jul 30, 2017 at 02:27:02PM +0200, Kornel Benko wrote:
> 
> Testing the patch shell-escape-auth-5.diff, there is 1 issue from my side:
> The session entry is totally ignored if using 'lyx -E', therefore I'd prefer
> to save the entries in some other file (for instance "session.shellEscape"),
> which could be loaded unconditionally.

I am wondering whether it is better this way, security wise. When you
export from command line you don't get any clue that latex is allowed
to run external programs. Perhaps a further line command option could
be introduced, say --shell-escape, that in this case would be an explicit
consent. This option would be effective only for batch jobs and ignored
in normal operation.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Sat, Jul 29, 2017 at 07:51:08PM -0400, Scott Kostyshak wrote:
> 
> If I remove the python-pygments module (and reconfigure) and test, I get
> the following error:
> 
> The driver command necessary to use the minted package
> (pygmentize) has not been found. Make sure you have
> the python-pygments module installed or, if the driver
> is named differently, to add the following line to the
> document preamble:
> 
> \AtBeginDocument{\renewcommand{\MintedPygmentize}{driver}}
> 
> where 'driver' is name of the driver command.
> 
> I expected to get the error, but what I was surprised by is that I get
> the error when going to Document > Settings. I expected to get an error
> immediately after compiling (before the LaTeX log errors). Is that
> expected? Note that I am using a compiled Qt 5.9.2dev so if you can't
> reproduce, it is probably a bug on my side.

No, it is not related to the Qt version. I was aware of it but further
improvements were delayed by the exhausting discussion that followed.
I am contemplating the possibility of simply not allowing minted if the
driver program is not called pygmentize, so that in this case the combo
box is simply disabled without any warning.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Freitag, 28. Juli 2017 um 23:31:42, schrieb Enrico Forestieri 

> On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:
> 
> > I am also investigating the possibility of attaching a context menu to
> > the red icon in the status bar.
> 
> This is now done in the attached patch. Hovering with the mouse over the
> red icon a warning is shown, and right clicking shows a popup menu that
> allows to disable the shell escape option for the current document.
> 

Testing the patch shell-escape-auth-5.diff, there is 1 issue from my side:
The session entry is totally ignored if using 'lyx -E', therefore I'd prefer
to save the entries in some other file (for instance "session.shellEscape"),
which could be loaded unconditionally.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am 30.07.2017 11:54 vorm. schrieb "Jean-Marc Lasgouttes" :

FWIW, I am on vacation without a computer right now. I did not test
Enrico's patch, but I agree that this is the right direction. So applying
it cannot hurt, and will allow us to move forward, not only wrt next
release, but also on this issue.


Same here (vacation, no computer and +1 Enrico's patch.

Jürgen



JMarc

Le 29 juillet 2017 23:57:52 GMT+02:00, Scott Kostyshak  a
écrit :
>I'm concerned that since this issue has left us all exhausted, there is
>a feeling of "let's just get this over so we can move on". I encourage
>all of us to give one more cognitive spurt and give a vote.

Re: Options for resolving the minted + shell-escape issue

[Jean-Marc Lasgouttes]

FWIW, I am on vacation without a computer right now. I did not test Enrico's 
patch, but I agree that this is the right direction. So applying it cannot 
hurt, and will allow us to move forward, not only wrt next release, but also on 
this issue. 

JMarc

Le 29 juillet 2017 23:57:52 GMT+02:00, Scott Kostyshak  a 
écrit :
>I'm concerned that since this issue has left us all exhausted, there is
>a feeling of "let's just get this over so we can move on". I encourage
>all of us to give one more cognitive spurt and give a vote.

 

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Sat, Jul 29, 2017 at 07:51:08PM -0400, Scott Kostyshak wrote:
> On Sat, Jul 29, 2017 at 05:48:46PM -0400, Scott Kostyshak wrote:
> > On Fri, Jul 28, 2017 at 11:31:42PM +0200, Enrico Forestieri wrote:
> > > On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:
> > > 
> > > > I am also investigating the possibility of attaching a context menu to
> > > > the red icon in the status bar.
> > > 
> > > This is now done in the attached patch. Hovering with the mouse over the
> > > red icon a warning is shown, and right clicking shows a popup menu that
> > > allows to disable the shell escape option for the current document.
> > 
> > I'm trying to test the patch with respect to d3b2cb09, because I could
> > not compile current master for some reason.
> 
> I was able to apply the patch at 770ab9a8 and successfully compile it.
> The improvements look good. I've only noticed one issue so far:
> 
> If I remove the python-pygments module (and reconfigure) and test, I get
> the following error:
> 
> The driver command necessary to use the minted package
> (pygmentize) has not been found. Make sure you have
> the python-pygments module installed or, if the driver
> is named differently, to add the following line to the
> document preamble:
> 
> \AtBeginDocument{\renewcommand{\MintedPygmentize}{driver}}
> 
> where 'driver' is name of the driver command.
> 
> I expected to get the error, but what I was surprised by is that I get
> the error when going to Document > Settings. I expected to get an error
> immediately after compiling (before the LaTeX log errors). Is that
> expected? Note that I am using a compiled Qt 5.9.2dev so if you can't
> reproduce, it is probably a bug on my side.

I should clarify that the above comment has nothing to do with the
patch. It is about the previously implemented minted support.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Sat, Jul 29, 2017 at 05:48:46PM -0400, Scott Kostyshak wrote:
> On Fri, Jul 28, 2017 at 11:31:42PM +0200, Enrico Forestieri wrote:
> > On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:
> > 
> > > I am also investigating the possibility of attaching a context menu to
> > > the red icon in the status bar.
> > 
> > This is now done in the attached patch. Hovering with the mouse over the
> > red icon a warning is shown, and right clicking shows a popup menu that
> > allows to disable the shell escape option for the current document.
> 
> I'm trying to test the patch with respect to d3b2cb09, because I could
> not compile current master for some reason.

I was able to apply the patch at 770ab9a8 and successfully compile it.
The improvements look good. I've only noticed one issue so far:

If I remove the python-pygments module (and reconfigure) and test, I get
the following error:

The driver command necessary to use the minted package
(pygmentize) has not been found. Make sure you have
the python-pygments module installed or, if the driver
is named differently, to add the following line to the
document preamble:

\AtBeginDocument{\renewcommand{\MintedPygmentize}{driver}}

where 'driver' is name of the driver command.

I expected to get the error, but what I was surprised by is that I get
the error when going to Document > Settings. I expected to get an error
immediately after compiling (before the LaTeX log errors). Is that
expected? Note that I am using a compiled Qt 5.9.2dev so if you can't
reproduce, it is probably a bug on my side.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Samstag, 29. Juli 2017 um 17:57:52, schrieb Scott Kostyshak 

> On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:
> > On 07/20/2017 01:21 AM, Jürgen Spitzmüller wrote:
> > > Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:
> > >> The attached patch takes into account all of these ideas. As a
> > >> disclaimer,
> > >> note that I am providing it only because I am now familiar with this
> > >> part
> > >> of the code and can quickly come up with a patch. But I am not
> > >> endorsing it.
> > > I propose to apply this patch and return to productivity.
> > 
> > I would agree with that.
> 
> I'm concerned that since this issue has left us all exhausted, there is
> a feeling of "let's just get this over so we can move on". I encourage
> all of us to give one more cognitive spurt and give a vote.
> 
> From what I understand, the three options are still what I proposed
> three weeks ago [1]:
> 
> 1. Revert the recently added minted support.
> 
> 2. Keep the current state of master.
> 
> 3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
> to lib/images. (Note that I get linker errors when I try to apply the
> latest patch, but it might be something specific to my setup.)

I opt for this one. There is work to be done for tests, but this looks doable.

> So I ask explicitly to everyone (even if you think you have already
> voted, please give your vote again): who has given the latest patch a
> good test, has looked at the code (the patch is not so long), and votes
> in favor of one of the three options?
> 
> Please also feel free to say that you have tested the patch but choose
> to abstain from voting. This way I will know that even if the vote is
> only 2 to 1, at least the rest of the developers are indifferent, rather
> than not having tested. But I ask that everyone tests the patch and says
> something. If you are lost because you had the fortune of not following
> the debate, let us know so we can tell you what to test, and can point
> you to some advantages/disadvantages of each option.
> 
> Scott
> 
> 
> [1]
> https://www.mail-archive.com/search?l=mid=20170705045915.h4uyrsc27g54da3m%40steph
> [2]
> https://www.mail-archive.com/search?l=mid=20170728213142.GA7880%40GIOVE

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Kornel Benko]

Am Samstag, 29. Juli 2017 um 17:48:46, schrieb Scott Kostyshak 

> On Fri, Jul 28, 2017 at 11:31:42PM +0200, Enrico Forestieri wrote:
> > On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:
> > 
> > > I am also investigating the possibility of attaching a context menu to
> > > the red icon in the status bar.
> > 
> > This is now done in the attached patch. Hovering with the mouse over the
> > red icon a warning is shown, and right clicking shows a popup menu that
> > allows to disable the shell escape option for the current document.
> 
> I'm trying to test the patch with respect to d3b2cb09, because I could
> not compile current master for some reason.
> 
> I get linker errors. I attach the errors from both CMake and autotools.
> If you think my problems are due to using an older commit or something
> else specific to my system, please don't spend time on it. I'll figure
> out soon why I cannot compile current master on CMake. Maybe that will
> somehow fix the issue.
> 
> Scott

I had no errors, neither compile nor link with cmake build.

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:
> On 07/20/2017 01:21 AM, Jürgen Spitzmüller wrote:
> > Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:
> >> The attached patch takes into account all of these ideas. As a
> >> disclaimer,
> >> note that I am providing it only because I am now familiar with this
> >> part
> >> of the code and can quickly come up with a patch. But I am not
> >> endorsing it.
> > I propose to apply this patch and return to productivity.
> 
> I would agree with that.

I'm concerned that since this issue has left us all exhausted, there is
a feeling of "let's just get this over so we can move on". I encourage
all of us to give one more cognitive spurt and give a vote.

From what I understand, the three options are still what I proposed
three weeks ago [1]:

1. Revert the recently added minted support.

2. Keep the current state of master.

3. Apply the patch at [2]. Don't forget to copy emblem-shellescape.svgz
to lib/images. (Note that I get linker errors when I try to apply the
latest patch, but it might be something specific to my setup.)

So I ask explicitly to everyone (even if you think you have already
voted, please give your vote again): who has given the latest patch a
good test, has looked at the code (the patch is not so long), and votes
in favor of one of the three options?

Please also feel free to say that you have tested the patch but choose
to abstain from voting. This way I will know that even if the vote is
only 2 to 1, at least the rest of the developers are indifferent, rather
than not having tested. But I ask that everyone tests the patch and says
something. If you are lost because you had the fortune of not following
the debate, let us know so we can tell you what to test, and can point
you to some advantages/disadvantages of each option.

Scott


[1]
https://www.mail-archive.com/search?l=mid=20170705045915.h4uyrsc27g54da3m%40steph
[2]
https://www.mail-archive.com/search?l=mid=20170728213142.GA7880%40GIOVE


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Sat, Jul 29, 2017 at 02:03:52AM +0200, Tommaso Cucinotta wrote:

> 3) I liked to propose addition of support for running Gnuplot scripts as 
> well, with the corresponding converter tagged as a 'needauth' one
> [ this last feature cannot be tagged as a security enhancement,

(clarifying, in case anyone was wondering why Tommaso said the above):
because we did not previously ship Gnuplot.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Fri, Jul 28, 2017 at 11:31:42PM +0200, Enrico Forestieri wrote:
> On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:
> 
> > I am also investigating the possibility of attaching a context menu to
> > the red icon in the status bar.
> 
> This is now done in the attached patch. Hovering with the mouse over the
> red icon a warning is shown, and right clicking shows a popup menu that
> allows to disable the shell escape option for the current document.

I'm trying to test the patch with respect to d3b2cb09, because I could
not compile current master for some reason.

I get linker errors. I attach the errors from both CMake and autotools.
If you think my problems are due to using an older commit or something
else specific to my system, please don't spend time on it. I'll figure
out soon why I cannot compile current master on CMake. Maybe that will
somehow fix the issue.

Scott
/home/scott/lyxbuilds/master/repo/src/Buffer.cpp:984: undefined reference to 
`lyx::ShellEscapeSection::find(std::__cxx11::basic_string const&) const'
liblyxcore.a(Converter.o): In function 
`lyx::Converters::checkAuth(lyx::Converter const&, 
std::__cxx11::basic_string 
const&, bool)':
/home/scott/lyxbuilds/master/repo/src/Converter.cpp:360: undefined reference to 
`lyx::ShellEscapeSection::findAuth(std::__cxx11::basic_string const&) const'
/home/scott/lyxbuilds/master/repo/src/Converter.cpp:360: undefined reference to 
`lyx::AuthFilesSection::find(std::__cxx11::basic_string const&) const'
/home/scott/lyxbuilds/master/repo/src/Converter.cpp:367: undefined reference to 
`lyx::ShellEscapeSection::insert(std::__cxx11::basic_string const&, bool)'
/home/scott/lyxbuilds/master/repo/src/Converter.cpp:369: undefined reference to 
`lyx::AuthFilesSection::insert(std::__cxx11::basic_string const&)'
liblyxcore.a(LyX.o): In function 
`lyx::ShellEscapeSection::~ShellEscapeSection()':
/home/scott/lyxbuilds/master/repo/src/Session.h:347: undefined reference to 
`vtable for lyx::ShellEscapeSection'
/home/scott/lyxbuilds/master/repo/src/Session.h:347: undefined reference to 
`vtable for lyx::ShellEscapeSection'
/home/scott/lyxbuilds/master/repo/src/Session.h:347: undefined reference to 
`vtable for lyx::ShellEscapeSection'
liblyxcore.a(Session.o): In function 
`lyx::ShellEscapeSection::ShellEscapeSection()':
/home/scott/lyxbuilds/master/repo/src/Session.h:351: undefined reference to 
`vtable for lyx::ShellEscapeSection'
liblyxcore.a(Session.o): In function 
`lyx::ShellEscapeSection::~ShellEscapeSection()':
/home/scott/lyxbuilds/master/repo/src/Session.h:347: undefined reference to 
`vtable for lyx::ShellEscapeSection'
frontends/qt4/liblyxqt4.a(GuiView.o): In function 
`lyx::frontend::GuiView::disableShellEscape()':
/home/scott/lyxbuilds/master/repo/src/frontends/qt4/GuiView.cpp:661: undefined 
reference to `lyx::ShellEscapeSection::remove(std::__cxx11::basic_string const&)'
frontends/qt4/liblyxqt4.a(GuiDocument.o): In function 
`lyx::frontend::GuiDocument::shellescapeChanged()':
/home/scott/lyxbuilds/master/repo/src/frontends/qt4/GuiDocument.cpp:1550: 
undefined reference to 
`lyx::ShellEscapeSection::remove(std::__cxx11::basic_string const&)'
/home/scott/lyxbuilds/master/repo/src/frontends/qt4/GuiDocument.cpp:1551: 
undefined reference to 
`lyx::ShellEscapeSection::find(std::__cxx11::basic_string const&) const'
/home/scott/lyxbuilds/master/repo/src/frontends/qt4/GuiDocument.cpp:1552: 
undefined reference to 
`lyx::ShellEscapeSection::insert(std::__cxx11::basic_string const&, bool)'
collect2: error: ld returned 1 exit status
Makefile:2125: recipe for target 'lyx' failed
make[4]: *** [lyx] Error 1
make[4]: Leaving directory '/home/scott/lyxbuilds/master/repo/src'
Makefile:2467: recipe for target 'all-recursive' failed
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory '/home/scott/lyxbuilds/master/repo/src'
Makefile:1703: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/home/scott/lyxbuilds/master/repo/src'
Makefile:575: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/scott/lyxbuilds/master/repo'
Makefile:470: recipe for target 'all' failed
make: *** [all] Error 2

CMakeFiles/lyx.dir/Session.cpp.o: In function 
`lyx::ShellEscapeSection::ShellEscapeSection()':
/home/scott/lyxbuilds/master/repo/src/Session.h:351: undefined reference to 
`vtable for lyx::ShellEscapeSection'
CMakeFiles/lyx.dir/Session.cpp.o: In function 
`lyx::ShellEscapeSection::~ShellEscapeSection()':

Re: Options for resolving the minted + shell-escape issue

[Tommaso Cucinotta]

On 17/07/2017 15:18, Enrico Forestieri wrote:

I don't understand if the 'needauth' is new in LyX 2.3.0 or already existed.


This is new to 2.3.0 and is instead deemed a security improvement. Yes,
security is improved by allowing running dangerous converters.


I have to step in for a clarification:

1) without the needauth mechanism, LyX would run arbitrary R code snippets 
embedded within a .lyx code whenever you tried to compile your doc, without 
asking anything, without telling the user, etc, such R code could even grab 
your ~/.ssh/ folder and upload it to a remote site, or just rm -rf $HOME/ 
(delete your entire home folder)

2) needauth was added as a safety measure that tags those potentially dangerous 
converters (the one executing R scripts) as 'needauth' (i.e., needs user's 
authorization before being run), so that, whenever LyX tries to run that 
converter, it will ask the user first

Therefore, it should be quite clear that needauth is a security improvement 
(compared with LyX supporting R/sweave/knitr without needauth).

3) I liked to propose addition of support for running Gnuplot scripts as well, 
with the corresponding converter tagged as a 'needauth' one
[ this last feature cannot be tagged as a security enhancement, but it's not 
more dangerous than what we already had with R scripting ]


The text indicates to me that it's possible for a document to store some
kind of setting that allows a converter (here external program) to be run
in an automated manner without my manual intervention or consent.


it is possible to use preferences settings in THREE possible combinations:

A) forbid any use of any 'needauth' converters altogether (will be the default 
for all users), that means users trying to compile a doc embedding a R script 
would see a dialog telling them that it was not possible to convert that part 
of the document because it is unsafe and the preferences options forbid it

  \use_converter_needauth_forbidden true

B) allow for using 'needauth' converters in the normal/designed mode, i.e., only after 
gaining user's authorization through a confirmation dialog that pops up on-demand 
per-converter invocation and per-document (but it's not asked again once the converted 
image is cached, so a second compilation of the same doc would NOT ask anything, because 
LyX would use images from its cache in ~/.lyx/cache/*): users can check a "don't ask 
me again for this document" if they trust the document, and there's quite a few 
converters therein, so any further 'needauth' converter in the doc is run without further 
auth question

  \use_converter_needauth_forbidden false
  \use_converter_needauth true

C) allow for using 'needauth' converters in the old unsafe way, as per status 
quo before 'needauth' was merged in: any converter is just run, including 
'needauth' ones, without asking anything; after commit [8a4fcd3d/lyxgit], when 
a user tries to get to this configuration from the preferences, he/she gets an 
additional SECURITY WARNING dialog explaining how unsafe/dangerous an action 
they're doing -- at this point, if users go on and approve the config options 
change, then they're assumed to know what they're doing

  \use_converter_needauth_forbidden false
  \use_converter_needauth false

There's an additional patch worked out in the TT below, which I like to think 
of as the final solution:

D) execute any converter within a protected/safe sandbox, this can easily be 
accomplished on Ubuntu/Linux using AppArmor, but it's not easy to merge such a 
change unless there's a portable corss-OS way -- albeit, for Linux users, this 
would be THE right way to go (with an initial transient until the AppArmor 
rules are correctly tunes, which I cannot claim is a trivial task). This would 
require no needauth, because the converter could be restricted to write only 
within a temporary folder, and be forced into having no network access, then 
only the created image file would be copied back from there, and anything else 
trashed away.

I hope the above clarifies.


but
consider the following example:
I create a document with some embedded code to be run by converters. It's
my document, I trust it. Then I e-mail it to a colleague or perhaps my
customer for review. Time goes by and eventually I get the document back
from review, but the review took longer than expected and my next deadline
was yesterday, so I'm in a hurry and build "my" document.


this is an excellent scenario to add to the list of additional enhancements, 
thanks; just TT-ified it:

  http://www.lyx.org/trac/ticket/10735

The enhancement that was already discussed back long ago, was an expiry 
timestamp
of the user's authorization, so that, when user says "don't ask me again", this
will be remembered only for X days, weeks, months, then it will expire, and if
the same doc is opened back 1y later, one would be warned again.

  http://www.lyx.org/trac/ticket/10730

Another discussed one was about hashing the content 

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Jul 25, 2017 at 09:16:15PM +0200, Enrico Forestieri wrote:

> I am also investigating the possibility of attaching a context menu to
> the red icon in the status bar.

This is now done in the attached patch. Hovering with the mouse over the
red icon a warning is shown, and right clicking shows a popup menu that
allows to disable the shell escape option for the current document.

-- 
Enrico
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index a9cc619060..d7e827fc2a 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -55,6 +55,7 @@
 #include "ParagraphParameters.h"
 #include "ParIterator.h"
 #include "PDFOptions.h"
+#include "Session.h"
 #include "SpellChecker.h"
 #include "sgml.h"
 #include "texstream.h"
@@ -980,6 +981,8 @@ int Buffer::readHeader(Lexer & lex)
errorList.push_back(ErrorItem(_("Document header error"), s));
}
 
+   params().shell_escape = 
theSession().shellescapeFiles().find(absFileName());
+
params().makeDocumentClass();
 
return unknown_tokens;
diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 22e84778fe..c16c308c21 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -459,6 +459,7 @@ BufferParams::BufferParams()
html_css_as_file = false;
display_pixel_ratio = 1.0;
 
+   shell_escape = false;
output_sync = false;
use_refstyle = true;
use_minted = false;
diff --git a/src/BufferParams.h b/src/BufferParams.h
index c20601e2ac..bc5c10d194 100644
--- a/src/BufferParams.h
+++ b/src/BufferParams.h
@@ -535,6 +535,8 @@ public:
std::string html_latex_end;
///
bool html_css_as_file;
+   /// allow the LaTeX backend to run external programs
+   bool shell_escape;
/// generate output usable for reverse/forward search
bool output_sync;
/// custom LaTeX macro from user instead our own
diff --git a/src/Converter.cpp b/src/Converter.cpp
index 489f5df439..c9bce7f2db 100644
--- a/src/Converter.cpp
+++ b/src/Converter.cpp
@@ -279,20 +279,52 @@ OutputParams::FLAVOR 
Converters::getFlavor(Graph::EdgePath const & path,
 }
 
 
-bool Converters::checkAuth(Converter const & conv, string const & doc_fname)
+bool Converters::checkAuth(Converter const & conv, string const & doc_fname,
+  bool use_shell_escape)
 {
-   if (!conv.need_auth())
+   string conv_command = conv.command();
+   bool const has_shell_escape = contains(conv_command, "-shell-escape")
+   || contains(conv_command, "-enable-write18");
+   if (conv.latex() && has_shell_escape && !use_shell_escape) {
+   docstring const shellescape_warning =
+ bformat(_("The following LaTeX backend has been "
+   "configured to allow execution of external programs "
+   "for any document:"
+   "%1$s"
+   "This is a dangerous configuration. Please, "
+   "consider using the support offered by LyX for "
+   "allowing this privilege only to documents that "
+   "actually need it, instead."),
+   from_utf8(conv_command));
+   frontend::Alert::error(_("Security Warning"),
+   shellescape_warning , false);
+   } else if (!conv.latex())
+   use_shell_escape = false;
+   if (!conv.need_auth() && !use_shell_escape)
return true;
-   const docstring security_warning = bformat(
- _("The requested operation requires the use of a converter 
from "
-   "%2$s to %3$s:"
+   size_t const token_pos = conv_command.find("$$");
+   bool const has_token = token_pos != string::npos;
+   string const command = use_shell_escape && !has_shell_escape
+   ? (has_token ? conv_command.insert(token_pos, "-shell-escape ")
+: conv_command.append(" -shell-escape"))
+   : conv_command;
+   docstring const security_warning = (use_shell_escape
+   ? bformat(_("The following LaTeX backend has been requested "
+   "to allow execution of external programs:"
+   "%1$s"
+   "The external programs can execute arbitrary commands on "
+   "your system, including dangerous ones, if instructed to do "
+   "so by a maliciously crafted LyX document."),
+ from_utf8(command))
+   : bformat(_("The requested operation requires the use of a "
+   "converter from %2$s to %3$s:"
"%1$s"
-   "This external program can execute arbitrary commands on 
your "
-   "system, including dangerous ones, if instructed to do so by a "
-   "maliciously crafted .lyx document."),
- from_utf8(conv.command()), from_utf8(conv.from()),
- from_utf8(conv.to()));
-   if 

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Jul 25, 2017 at 11:11:36AM +0200, Pavel Sanda wrote:

> Scott Kostyshak wrote:
> > It did feel strange that the "OK" and "Apply" buttons were not enabled
> > when checking the "Allow running external programs" checkbox, but I got
> > used to it quickly.
> 
> Likely not feature but just forgotten connect with signal of box being 
> checked.

No, this was done by design. The status of this checkbox is not saved in
the document, so the document should not be marked as dirty.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Mon, Jul 24, 2017 at 11:05:42PM -0400, Scott Kostyshak wrote:
> 
> I think the following is unexpected:
> 
> Change a different document setting, e.g., "Synchronize with output".
> The buttons "OK" and "Apply" are enabled. Then click on "Allow running
> external programs". Notice that the "OK" and "Apply" buttons are
> disabled, because the settings were applied. I expected only the "Allow
> running external programs" setting to be instantly applied, and that the
> "Synchronize with output" would only be applied when I click on "OK" or
> "Apply".

Yes, I was aware of this glitch, which is addressed by the attached patch.

> It did feel strange that the "OK" and "Apply" buttons were not enabled
> when checking the "Allow running external programs" checkbox, but I got
> used to it quickly.

This is by design. The checkbox status is not saved in the document but
in the session file, so the document should not be marked as dirty (as
it would be after "OK" or "Apply").

> I wonder if it would be more clear if we append to the current tooltip
> "(this setting is changed applied)".

The attached patch appends "(this setting is always applied immediately)".

> Regarding the message that's displayed when someone manually adds
> -shell-escape to a converter, I think we should point the user to more
> information so that they can learn how to use the built-in LyX feature.
> 
> Where is the appropriate place in our documentation for this feature?
> 
> Similarly, we should point the user to the relevant information for the
> minted example files.

Yes, many improvements are possible with a more little time at hand.
I am also investigating the possibility of attaching a context menu to
the red icon in the status bar.

-- 
Enrico
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index a9cc619060..d7e827fc2a 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -55,6 +55,7 @@
 #include "ParagraphParameters.h"
 #include "ParIterator.h"
 #include "PDFOptions.h"
+#include "Session.h"
 #include "SpellChecker.h"
 #include "sgml.h"
 #include "texstream.h"
@@ -980,6 +981,8 @@ int Buffer::readHeader(Lexer & lex)
errorList.push_back(ErrorItem(_("Document header error"), s));
}
 
+   params().shell_escape = 
theSession().shellescapeFiles().find(absFileName());
+
params().makeDocumentClass();
 
return unknown_tokens;
diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index 22e84778fe..c16c308c21 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -459,6 +459,7 @@ BufferParams::BufferParams()
html_css_as_file = false;
display_pixel_ratio = 1.0;
 
+   shell_escape = false;
output_sync = false;
use_refstyle = true;
use_minted = false;
diff --git a/src/BufferParams.h b/src/BufferParams.h
index c20601e2ac..bc5c10d194 100644
--- a/src/BufferParams.h
+++ b/src/BufferParams.h
@@ -535,6 +535,8 @@ public:
std::string html_latex_end;
///
bool html_css_as_file;
+   /// allow the LaTeX backend to run external programs
+   bool shell_escape;
/// generate output usable for reverse/forward search
bool output_sync;
/// custom LaTeX macro from user instead our own
diff --git a/src/Converter.cpp b/src/Converter.cpp
index 489f5df439..c9bce7f2db 100644
--- a/src/Converter.cpp
+++ b/src/Converter.cpp
@@ -279,20 +279,52 @@ OutputParams::FLAVOR 
Converters::getFlavor(Graph::EdgePath const & path,
 }
 
 
-bool Converters::checkAuth(Converter const & conv, string const & doc_fname)
+bool Converters::checkAuth(Converter const & conv, string const & doc_fname,
+  bool use_shell_escape)
 {
-   if (!conv.need_auth())
+   string conv_command = conv.command();
+   bool const has_shell_escape = contains(conv_command, "-shell-escape")
+   || contains(conv_command, "-enable-write18");
+   if (conv.latex() && has_shell_escape && !use_shell_escape) {
+   docstring const shellescape_warning =
+ bformat(_("The following LaTeX backend has been "
+   "configured to allow execution of external programs "
+   "for any document:"
+   "%1$s"
+   "This is a dangerous configuration. Please, "
+   "consider using the support offered by LyX for "
+   "allowing this privilege only to documents that "
+   "actually need it, instead."),
+   from_utf8(conv_command));
+   frontend::Alert::error(_("Security Warning"),
+   shellescape_warning , false);
+   } else if (!conv.latex())
+   use_shell_escape = false;
+   if (!conv.need_auth() && !use_shell_escape)
return true;
-   const docstring security_warning = bformat(
- _("The requested operation requires the use of a converter 

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Scott Kostyshak wrote:
> It did feel strange that the "OK" and "Apply" buttons were not enabled
> when checking the "Allow running external programs" checkbox, but I got
> used to it quickly.

Likely not feature but just forgotten connect with signal of box being checked.
P

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Mon, Jul 24, 2017 at 05:49:33PM -0400, Scott Kostyshak wrote:
> On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:
> 
> > > I propose to apply this patch and return to productivity.
> > 
> > I would agree with that.
> 
> It seems this patch has some support. I will do some testing and give my
> thoughts soon.

I think the following is unexpected:

Change a different document setting, e.g., "Synchronize with output".
The buttons "OK" and "Apply" are enabled. Then click on "Allow running
external programs". Notice that the "OK" and "Apply" buttons are
disabled, because the settings were applied. I expected only the "Allow
running external programs" setting to be instantly applied, and that the
"Synchronize with output" would only be applied when I click on "OK" or
"Apply".

It did feel strange that the "OK" and "Apply" buttons were not enabled
when checking the "Allow running external programs" checkbox, but I got
used to it quickly.

I wonder if it would be more clear if we append to the current tooltip
"(this setting is changed applied)".

Regarding the message that's displayed when someone manually adds
-shell-escape to a converter, I think we should point the user to more
information so that they can learn how to use the built-in LyX feature.

Where is the appropriate place in our documentation for this feature?

Similarly, we should point the user to the relevant information for the
minted example files.

Scott


signature.asc
Description: PGP signature

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Thu, Jul 20, 2017 at 03:30:14PM -0400, Richard Heck wrote:

> > I propose to apply this patch and return to productivity.
> 
> I would agree with that.

It seems this patch has some support. I will do some testing and give my
thoughts soon. I'm still traveling but will be back and start having
more time hopefully by Thursday.

Scott

Re: Options for resolving the minted + shell-escape issue

[Scott Kostyshak]

On Mon, Jul 17, 2017 at 12:49:05AM +0200, Christian Ridderström wrote:

Hi Christian,

I did not respond to your other questions since I've been traveling, and
others have done so in the mean time. Let me know if some of your
questions have gone unanswered.

> As an aside, I've used org-mode documents in Emacs to invoke MATLAB on
> snippets from within the document and it's very nice, except the really
> annoying part where you for each snippet have to approve that it's run.
> Each time. A big bug in org-mode is that it's not properly showing the
> snippet that's about to be executed before you approve it to be run...
> Perhaps I'm a masochist...

I would want that feature also.

> I'm sorry for not being of more help, but hopefully my comments will
> trigger someone else to contribute something more helpful.

You have been a great help! Thank you for bravely jumping in! I know it
is not the most fun, but as you've mentioned it is an important issue.

Scott

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 07/20/2017 01:21 AM, Jürgen Spitzmüller wrote:
> Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:
>> The attached patch takes into account all of these ideas. As a
>> disclaimer,
>> note that I am providing it only because I am now familiar with this
>> part
>> of the code and can quickly come up with a patch. But I am not
>> endorsing it.
> I propose to apply this patch and return to productivity.

I would agree with that.

Richard

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am Mittwoch, den 19.07.2017, 16:37 +0200 schrieb Enrico Forestieri:
> The attached patch takes into account all of these ideas. As a
> disclaimer,
> note that I am providing it only because I am now familiar with this
> part
> of the code and can quickly come up with a patch. But I am not
> endorsing it.

I propose to apply this patch and return to productivity.

Jürgen

signature.asc
Description: This is a digitally signed message part

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 07/19/2017 02:04 PM, Enrico Forestieri wrote:
> On Wed, Jul 19, 2017 at 10:58:44AM -0400, Richard Heck wrote:
>> Thanks for this, Enrico. Let me just ask one question about it: Is the
>> mechanism here per-document
>> or per-document and also per-converter?
> This only addresses particular converters, i.e., the latex backends.
>
>> That is, suppose there was a
>> document that contained R
>> code and minted code.
> Please, note that this is not specifically aimed at minted. It only
> allows the latex backend to execute external commands. In case there
> is R code, it doesn't need this permission and would execute whether
> or not one allows shell-escape.
>
>> Would one be warned about both or would one only
>> be asked once?
> If there is a needauth converter involved in a latex run, one would
> be warned about it independently of whether shell-escape is allowed
> or not. So, one would be warned first to allow at all the latex run
> in case shell-escape is allowed, and then would be warned that there
> is also a dangerous converter to be run. Both warnings can be
> independently suppressed. So, even if you allowed the dangerous
> converter, you will be warned about shell-escape, and vice versa.
>
>> If the
>> the latter, how hard would it be to change that?
> I hope I answered your question.

Thanks for the clarifications. I've found all of this somewhat
confusing, since I don't use
any of these mechanisms myself.

Richard

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Wed, Jul 19, 2017 at 10:58:44AM -0400, Richard Heck wrote:
> 
> Thanks for this, Enrico. Let me just ask one question about it: Is the
> mechanism here per-document
> or per-document and also per-converter?

This only addresses particular converters, i.e., the latex backends.

> That is, suppose there was a
> document that contained R
> code and minted code.

Please, note that this is not specifically aimed at minted. It only
allows the latex backend to execute external commands. In case there
is R code, it doesn't need this permission and would execute whether
or not one allows shell-escape.

> Would one be warned about both or would one only
> be asked once?

If there is a needauth converter involved in a latex run, one would
be warned about it independently of whether shell-escape is allowed
or not. So, one would be warned first to allow at all the latex run
in case shell-escape is allowed, and then would be warned that there
is also a dangerous converter to be run. Both warnings can be
independently suppressed. So, even if you allowed the dangerous
converter, you will be warned about shell-escape, and vice versa.

> If the
> the latter, how hard would it be to change that?

I hope I answered your question.

> Oh, maybe another thought: Could clicking the 'warning' icon easily be
> used to disable execution?
> Just a UI thought there.

I don't know how much easily. But this is simply because of my ignorance
about Qt. It could be very easy to do that.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 07/19/2017 10:37 AM, Enrico Forestieri wrote:
> On Tue, Jul 18, 2017 at 07:26:23PM -0400, Richard Heck wrote:
>> On 07/18/2017 09:56 AM, Jürgen Spitzmüller wrote:
>>> Am Dienstag, den 18.07.2017, 15:39 +0200 schrieb Jean-Marc Lasgouttes:
 Whi, not, maybe along with the names of the converters (features) 
 Sweave/gnuplot/minted present in current document and accepted by the
 user.
>>> I would add a verbose tooltip when hovering the icon, something like
>>>
>>> '''
>>> NOTE: Shell escape access granted.
>>>
>>> For this document, access to the -shell-escape feature has been granted
>>> for the following converters: ...
>>>
>>> Note that this is a potential security risk. Use only if you trust the
>>> source of this document. Please refer to sec. xx of the User Guide for
>>> details.
>>>
>>> To withdraw shell escape access, press this icon.
>>> '''
>> This seems a reasonable solution to me. It is not perfect, but nothing is.
>>
>> As I see it, the issue is that there are actually a wide variety of
>> reasons that users might want to
>> enable -shell-escape for various converters. As LyX currently functions,
>> the only way to do this
>> is to add this to the converter itself. This is dangerous from our point
>> of view NOT so much (or
>> only) because it is intrinsically dangerous, but rather because it it is
>> the kind of thing that is too
>> easy to "do and forget". Or, to put it differently: It is a serious
>> hassle to enable -shell-escape as
>> things are, and that invites people to do it and leave it. And that
>> really is a security risk.
> The attached patch takes into account all of these ideas. As a disclaimer,
> note that I am providing it only because I am now familiar with this part
> of the code and can quickly come up with a patch. But I am not endorsing it.
>
> The user can choose to allow execution of external programs for a given
> document through the document settings. However, this is a property that
> only holds on the computer used to edit the document. There is no way
> to send out a document with the shell-escape thing activated.
>
> Once activated, the user is prompted for confirmation each time a
> latex backend is used, unless he decides to always allow execution for
> a particular document. A document marked as requiring shell escape
> privileges is characterized by a red icon on the status bar.
>
> The shell-escape privilege can be revoked through the document settings.
> Given the peculiarity that this cannot be a mere document property but
> rather a property tied to both document and computer, the check box
> works  differently from all other check boxes. Indeed, checking or
> unchecking it should not make dirty the document. So, the privilege
> is given or revoked instantly when checking or unchecking, without the
> need of confirming the change.
>
> The patch also nags the user when -shell-escape is added to a latex
> backend, suggesting to use the support provided by LyX instead of
> allowing this privilege to any document. This is all we can do in
> this case to try to increase security, because we should't change
> users' choices.
>
> When a document with shell-escape privileges is moved to a new location
> or removed, it loses the privilege. So, if a new file with same name is
> later placed there, it doesn't inherit the privilege. 

Thanks for this, Enrico. Let me just ask one question about it: Is the
mechanism here per-document
or per-document and also per-converter? That is, suppose there was a
document that contained R
code and minted code. Would one be warned about both or would one only
be asked once? If the
the latter, how hard would it be to change that?

Oh, maybe another thought: Could clicking the 'warning' icon easily be
used to disable execution?
Just a UI thought there.

Richard

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Jul 18, 2017 at 07:26:23PM -0400, Richard Heck wrote:
> On 07/18/2017 09:56 AM, Jürgen Spitzmüller wrote:
> > Am Dienstag, den 18.07.2017, 15:39 +0200 schrieb Jean-Marc Lasgouttes:
> >> Whi, not, maybe along with the names of the converters (features) 
> >> Sweave/gnuplot/minted present in current document and accepted by the
> >> user.
> > I would add a verbose tooltip when hovering the icon, something like
> >
> > '''
> > NOTE: Shell escape access granted.
> >
> > For this document, access to the -shell-escape feature has been granted
> > for the following converters: ...
> >
> > Note that this is a potential security risk. Use only if you trust the
> > source of this document. Please refer to sec. xx of the User Guide for
> > details.
> >
> > To withdraw shell escape access, press this icon.
> > '''
> 
> This seems a reasonable solution to me. It is not perfect, but nothing is.
> 
> As I see it, the issue is that there are actually a wide variety of
> reasons that users might want to
> enable -shell-escape for various converters. As LyX currently functions,
> the only way to do this
> is to add this to the converter itself. This is dangerous from our point
> of view NOT so much (or
> only) because it is intrinsically dangerous, but rather because it it is
> the kind of thing that is too
> easy to "do and forget". Or, to put it differently: It is a serious
> hassle to enable -shell-escape as
> things are, and that invites people to do it and leave it. And that
> really is a security risk.

The attached patch takes into account all of these ideas. As a disclaimer,
note that I am providing it only because I am now familiar with this part
of the code and can quickly come up with a patch. But I am not endorsing it.

The user can choose to allow execution of external programs for a given
document through the document settings. However, this is a property that
only holds on the computer used to edit the document. There is no way
to send out a document with the shell-escape thing activated.

Once activated, the user is prompted for confirmation each time a
latex backend is used, unless he decides to always allow execution for
a particular document. A document marked as requiring shell escape
privileges is characterized by a red icon on the status bar.

The shell-escape privilege can be revoked through the document settings.
Given the peculiarity that this cannot be a mere document property but
rather a property tied to both document and computer, the check box
works  differently from all other check boxes. Indeed, checking or
unchecking it should not make dirty the document. So, the privilege
is given or revoked instantly when checking or unchecking, without the
need of confirming the change.

The patch also nags the user when -shell-escape is added to a latex
backend, suggesting to use the support provided by LyX instead of
allowing this privilege to any document. This is all we can do in
this case to try to increase security, because we should't change
users' choices.

When a document with shell-escape privileges is moved to a new location
or removed, it loses the privilege. So, if a new file with same name is
later placed there, it doesn't inherit the privilege. 

To try the patch, the emblem-shellescape.svgz icon should be copied to
lib/images.

-- 
Enrico
diff --git a/src/Buffer.cpp b/src/Buffer.cpp
index df7efa8f0c..3f3f706251 100644
--- a/src/Buffer.cpp
+++ b/src/Buffer.cpp
@@ -55,6 +55,7 @@
 #include "ParagraphParameters.h"
 #include "ParIterator.h"
 #include "PDFOptions.h"
+#include "Session.h"
 #include "SpellChecker.h"
 #include "sgml.h"
 #include "texstream.h"
@@ -980,6 +981,8 @@ int Buffer::readHeader(Lexer & lex)
errorList.push_back(ErrorItem(_("Document header error"), s));
}
 
+   params().shell_escape = 
theSession().shellescapeFiles().find(absFileName());
+
params().makeDocumentClass();
 
return unknown_tokens;
diff --git a/src/BufferParams.cpp b/src/BufferParams.cpp
index f4890eaeec..d69c9f1ecf 100644
--- a/src/BufferParams.cpp
+++ b/src/BufferParams.cpp
@@ -38,6 +38,7 @@
 #include "Lexer.h"
 #include "LyXRC.h"
 #include "OutputParams.h"
+#include "Session.h"
 #include "Spacing.h"
 #include "texstream.h"
 #include "TexRow.h"
@@ -459,6 +460,7 @@ BufferParams::BufferParams()
html_css_as_file = false;
display_pixel_ratio = 1.0;
 
+   shell_escape = false;
output_sync = false;
use_refstyle = true;
use_minted = false;
@@ -1441,6 +1443,21 @@ void BufferParams::writeFile(ostream & os, Buffer const 
* buf) const
 os << "\\html_latex_end " << html_latex_end << '\n';
 
os << pimpl_->authorlist;
+
+   if (!shell_escape) {
+   std::set & shellescape_files =
+   theSession().shellescapeFiles().shellescapeFiles();
+   char const * ext[2] = { ",0", ",1" };
+   for (int i = 0; i < 2; ++i) {
+   

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

>
> But I'd make one more suggestion: Every time a user opens a
> document for which this
> sort of  thing will be enabled, we pop a warning before we do anything.
> I.e., we do NOT just run
> gnuplot in the background, but we say something like what Jürgen had
> above, with buttons offering
> either to proceed or not. Doing this once per document per session does
> not seem too much to ask.
> (It would streamline things a bit, too, if we could 'inherit' this
> setting for child documents. So you
> would not have to keep clicking through if there were a lot of children.)
>

I agree this is not too much of a burden UI-wise and enhances awareness.

Jürgen



>
> Richard
>
>

Re: Options for resolving the minted + shell-escape issue

[Richard Heck]

On 07/18/2017 09:56 AM, Jürgen Spitzmüller wrote:
> Am Dienstag, den 18.07.2017, 15:39 +0200 schrieb Jean-Marc Lasgouttes:
>> Whi, not, maybe along with the names of the converters (features) 
>> Sweave/gnuplot/minted present in current document and accepted by the
>> user.
> I would add a verbose tooltip when hovering the icon, something like
>
> '''
> NOTE: Shell escape access granted.
>
> For this document, access to the -shell-escape feature has been granted
> for the following converters: ...
>
> Note that this is a potential security risk. Use only if you trust the
> source of this document. Please refer to sec. xx of the User Guide for
> details.
>
> To withdraw shell escape access, press this icon.
> '''

This seems a reasonable solution to me. It is not perfect, but nothing is.

As I see it, the issue is that there are actually a wide variety of
reasons that users might want to
enable -shell-escape for various converters. As LyX currently functions,
the only way to do this
is to add this to the converter itself. This is dangerous from our point
of view NOT so much (or
only) because it is intrinsically dangerous, but rather because it it is
the kind of thing that is too
easy to "do and forget". Or, to put it differently: It is a serious
hassle to enable -shell-escape as
things are, and that invites people to do it and leave it. And that
really is a security risk.

The needauth mechanism provides some protection, but it seems to
introduce its own risks.

The current proposal is very much addressed to that problem, and I think
something like it should
be workable. But I'd make one more suggestion: Every time a user opens a
document for which this
sort of  thing will be enabled, we pop a warning before we do anything.
I.e., we do NOT just run
gnuplot in the background, but we say something like what Jürgen had
above, with buttons offering
either to proceed or not. Doing this once per document per session does
not seem too much to ask.
(It would streamline things a bit, too, if we could 'inherit' this
setting for child documents. So you
would not have to keep clicking through if there were a lot of children.)

Richard

Re: Options for resolving the minted + shell-escape issue

[Enrico Forestieri]

On Tue, Jul 18, 2017 at 05:03:52PM +0200, Pavel Sanda wrote:
> 
> After reading all the thread now, would you call the position below
> hypocritical?
> 
> 1. I do not like needauth mechanism much, but I don't see better way how to
>allow advanced users to work with knitr/gnuplot without too much hassle.

Note that users can define their own dangerous converters and cannot be
stopped from doing that. The issue is that these converters are offered
by LyX itself. People can copy the converters from 2.2 without too much
hassle. The hypocrisy I was referring to is wanting to justify the
presence of such dangerous converters and at the same time asking to
revert a feature which is not insecure by itself.

> 2. We want minted support. New implementation does not work out of the box,
>and if we want it working & secure we need extend needauth more broadly
>into the code.

I have to disagree. It works out of the box and the user has only to know
what to do for using it. This is not a problem nowadays, because you can
use google to find web pages that explain in detail and with images what
has to be done. With the support offered by LyX it is simply not necessary
anymore using ERT. So, I don't see why this can be a problem.

>If there was no other way, it would be indeed unfair to use needauth
>unilateraly (that's your 'hypocrisy' argument if I get it right), but there
>seems to be pretty good chance that minted maintainer will solve the issue
>within couple months, which gives two other possible ways for ppl uneasy
>with needauth:
>a) return back to 2.2 behaviour and wait until maintainer of minted
>   let us call pygments directly and introduce it in the next cycle.
>b) have minted & needauth, but drop it once we can (and I would propose
>   the same thing for knitr/gnuplot if it ever becomes possible).
> 
> I can't read other people minds to speculate about their inner intensions,
> but to feel uneasiness about needauth-machinery creepism into the code
> seems valid reasoning from my perspective.

Now I don't care anymore. Do whatever you want.

-- 
Enrico

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Pavel Sanda wrote:
>b) have minted & needauth, but drop it once we can (and I would propose

.. by "it" I mean needauth ..

> 
> Pavel

Re: Options for resolving the minted + shell-escape issue

[Pavel Sanda]

Enrico Forestieri wrote:
> On Tue, Jul 18, 2017 at 01:28:38PM +0200, Pavel Sanda wrote:
> > Enrico Forestieri wrote:
> > > On Tue, Jul 18, 2017 at 12:57:24PM +0200, Kornel Benko wrote:
> > > > you have good and valid arguments. But don't you see how insulting some 
> > > > of
> > > > your mails are?
> > > 
> > > No, I actually don't. And I apologize if it may seem so.
> > 
> > It unfortunately seems so :(
> 
> Please, pay attention to not confusing hypocrisy and frankness.

After reading all the thread now, would you call the position below 
hypocritical?

1. I do not like needauth mechanism much, but I don't see better way how to
   allow advanced users to work with knitr/gnuplot without too much hassle.
   
2. We want minted support. New implementation does not work out of the box,
   and if we want it working & secure we need extend needauth more broadly
   into the code.
   
   If there was no other way, it would be indeed unfair to use needauth
   unilateraly (that's your 'hypocrisy' argument if I get it right), but there
   seems to be pretty good chance that minted maintainer will solve the issue
   within couple months, which gives two other possible ways for ppl uneasy
   with needauth:
   a) return back to 2.2 behaviour and wait until maintainer of minted
  let us call pygments directly and introduce it in the next cycle.
   b) have minted & needauth, but drop it once we can (and I would propose
  the same thing for knitr/gnuplot if it ever becomes possible).

I can't read other people minds to speculate about their inner intensions,
but to feel uneasiness about needauth-machinery creepism into the code
seems valid reasoning from my perspective.

Pavel

Re: Options for resolving the minted + shell-escape issue

[Jürgen Spitzmüller]

Am Dienstag, den 18.07.2017, 15:39 +0200 schrieb Jean-Marc Lasgouttes:
> Whi, not, maybe along with the names of the converters (features) 
> Sweave/gnuplot/minted present in current document and accepted by the
> user.

I would add a verbose tooltip when hovering the icon, something like

'''
NOTE: Shell escape access granted.

For this document, access to the -shell-escape feature has been granted
for the following converters: ...

Note that this is a potential security risk. Use only if you trust the
source of this document. Please refer to sec. xx of the User Guide for
details.

To withdraw shell escape access, press this icon.
'''

Jürgen

signature.asc
Description: This is a digitally signed message part