Python 3 incompatibilities still exist right?

[Scott Kostyshak]

I cleaned RELEASE-NOTES for the master branch. This note is still
relevant right?

- LyX needs to be run under Python 2 and will not work properly on systems
  where Python 3 is the default binary. See bug #7030 to know how to fix
  this properly, since simple sheebang conversion in *.py files will not
  be enough.


I've seen some references here and there (e.g. Benjamin's work on
features/python3), but Python 3 incompatibility is still in master
right?

Scott

Re: Clear RELEASE-NOTES on master?

[Scott Kostyshak]

On Tue, Nov 18, 2014 at 3:05 AM, Jean-Marc Lasgouttes
lasgout...@lyx.org wrote:
 Le 18/11/14 02:14, Scott Kostyshak a écrit :

 The RELEASE-NOTES on master is for 2.1.0. Should it be cleaned so we
 can add 2.2.0 stuff or is this done somewhere else that I forgot? I
 can do the work but I wanted to check in first.


 Sure.

Done at ef71f667.

Scott

Re: Help with patch for word selection mode on double click

[Scott Kostyshak]

On Tue, Oct 28, 2014 at 7:03 AM, Scott Kostyshak skost...@lyx.org wrote:
 I have a patch that enables word selection mode on double click. It
 seems to work well. The patch is here:

 http://www.lyx.org/trac/attachment/ticket/7890/0001-Fix-9160-and-7890.patch

 The only major issue is that when dragging to the left, you cannot
 highlight the first word. Does anyone have an idea for where to fix
 this?

 The following patch might give a clue as to where the problem is. When
 applied, you can select the first word. It is not meant to be taken as
 a correct patch:

 http://www.lyx.org/trac/attachment/ticket/7890/0002-HACK-just-to-show-where-the-problem-is.patch

I'm still looking for help on this in case anyone is motivated.

Scott

Re: Python 3 incompatibilities still exist right?

[stefano franchi]

On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak skost...@lyx.org wrote:

 I cleaned RELEASE-NOTES for the master branch. This note is still
 relevant right?

 - LyX needs to be run under Python 2 and will not work properly on systems
   where Python 3 is the default binary. See bug #7030 to know how to fix
   this properly, since simple sheebang conversion in *.py files will not
   be enough.


 I've seen some references here and there (e.g. Benjamin's work on
 features/python3), but Python 3 incompatibility is still in master
 right?


Actually, I thought this problem was solved a couple of years or so ago
(about Lyx? 2.0.2)
I used to have problems on my ArchLinux box (with python 3 as a default)
but that is no longer
the case.

Stefano



-- 
__
Stefano Franchi

stefano.fran...@gmail.com stef...@tamu.edu
http://stefano.cleinias.org

Re: Python 3 incompatibilities still exist right?

[Alex Vergara Gil]

The incompatibilities still exist at least in Windows sistems, since
LyX installer bundles a python distribution itself and creates a
conflict when python3 is the default. Moreover you cannot use python3
within LyX in Windows, because you will beak the system, LyX will not
reconfigure anymore!!
Just my experiences
LyX 2.1.2, Win7x64, Anaconda 2.1(python 3.4)

Re: Python 3 incompatibilities still exist right?

[Richard Heck]

On 11/20/2014 09:55 AM, stefano franchi wrote:
On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak skost...@lyx.org 
mailto:skost...@lyx.org wrote:


I cleaned RELEASE-NOTES for the master branch. This note is still
relevant right?

- LyX needs to be run under Python 2 and will not work properly on
systems
  where Python 3 is the default binary. See bug #7030 to know how
to fix
  this properly, since simple sheebang conversion in *.py files
will not
  be enough.


I've seen some references here and there (e.g. Benjamin's work on
features/python3), but Python 3 incompatibility is still in master
right?


Actually, I thought this problem was solved a couple of years or so 
ago (about Lyx? 2.0.2)
I used to have problems on my ArchLinux box (with python 3 as a 
default) but that is no longer

the case.


There are two different issues here: (i) Will LyX work when Python 3 is 
the default? (ii) Will
LyX work with Python 3? The answer to the former is Yes, as long as 
Python 2 is available:
We now look for it explicitly. The answer to the latter, I believe, is 
No: The LyX scripts will
not run under Python 3, though some work has been done on that, as Scott 
mentioned.


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Richard Heck wrote:

 I'm not well informed enough about implementation details to have an
 opinion about whether this
 diagnosis is correct. But I am somewhat puzzled that we are only seeing
 this issue now. Of course,
 there may have been other random crashes that people haven't reported.

This is an important point, and I asked myself the same question as well.  I 
gues the most plausible explanation is what James wrote in the gcc bug 
report: In this particular case, there is only a very, very small window
of time in which the error can occur. It might also be influenced 
considerably by compiler version and/or settings. Which linux distro do you 
use? Both the bug reporter and I am on debian.

 But I have used LyX with
 threaded export enabled, on Linux, for a very long time, with large
 documents with lots of children,
 and I've never seen anything like this crash. I thus remain curious
 about the details of the document
 itself. Is there something special about it that is triggering this?

It uses xelatex and biber, apart from that I don't know, but I did not 
really look at the contents. As I wrote in trac: Do we have a tool to 
anonymize documents currently? If not I'll write an lfun, because we 
definitely need a test case which can be tested by others, and I am not 
going to anonymize the document by hand.

 Obviously, we need some substantial discussion before making any
 decision about what to do. I'm
 going to cc a bunch of people to try to get their attention.

Definitely.


Georg

Re: Thread problems with std::basic_string and GNU libstdc++

[Abdelrazak Younes]

Hi Georg,

Warning: I haven't read LyX source code for a long long time :-)

On 19/11/2014 22:43, Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ gives
only a relatively weak thread-safety guarantee about std::basic string. It
is not completely thread-safe in the POSIX sense. The reason for this lies
in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already acknowledged
that they will get rid of it, but since this is an ABI incompatible change
they want to wait until they have several such changes to do all in one go.
For details please see the excellent gcc bug report by James Kanze (who is a
C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document language.

This looks all rather theoretical, but in practice it means that it is quite
easy to produce crashes if you use LyX built with #define EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child setups
seem to be much more vulnerable). The thing which seems to be most affected
is Language::babel_, which is copied a lot, but problems could arise from
all std::string and docstring variables.


I am not questioning your reasoning but I think STL thread safety 
limitation has limited impact in our case. AFAIR, the Buffer is cloned 
in in the current and main thread so we have no race condition here. 
After that the cloned buffer shares nothing with the original Buffer and 
can thus be used safely in another thread.





What do we do now? I can think of several alternatives:

1) Do nothing, because my findings are incorrect. I don't believe that of
course, but I would appreciate it if somebody would try to find flaws in my
reasoning.


I believe that there could be many reasons that could cause a crash 
before incriminating the stl string, etc:

* Gui elements (message passing?)
* Graphics preview elements



2) Disable EXPORT_in_THREAD if GNU libstdc++ is found. This is an easy and
safe solution, with the disadvantage that it removes a feature (but I think
that this is much better than risking crashes).

3) Derive an own class lyx::basic_string from std::basic_string which
circumvents copy-on-wrtite in the copy-constructor and assignment operator
by creating the copy from c_str(). Then use it

namespace lyx {
typedef basic_stringchar_type docstring;
typedef basic_stringchar string;
}

and replace all occurances of std::string with lyx::string. This would of
course need a lot of testing, but I believe it could work.

4) Do not use std::basic_string at all, but an own implementation which
could be stolen from STLport or clang (if the licenses permit that and are
compatible to the GPL, I did not check that). However, I believe that this
would create lots of interface problems with other libraries.


Does anybody see any other solution? I think for the stable branch we do not
really have any choice and need to implement 2).


Why not compile with c++11 option? I guess basic_string won't do 
copy-on-write if compiled this way?





The other question is: What can we learn for the future? AFAIK
EXPORT_in_THREAD was developed on windows, and it was probably well tested
on that platform. However, one basic lesson to learn is that such important
infrastructure changes need thorough testing on all supported platforms, and
I don't think this did happen in this case. Despite qt hides many
differences for us there are still a lot of platform specific issues left.


Well, IIRC I developed buffer cloning and initial threaded export using 
Linux :-)

Maybe Peter Kuemmel continued the work under Windows but I don't think so...

Thanks Georg, that was nice to discuss this.

Abdel.

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 02:35 PM, Georg Baum wrote:

Richard Heck wrote:


I'm not well informed enough about implementation details to have an
opinion about whether this
diagnosis is correct. But I am somewhat puzzled that we are only seeing
this issue now. Of course,
there may have been other random crashes that people haven't reported.

This is an important point, and I asked myself the same question as well.  I
gues the most plausible explanation is what James wrote in the gcc bug
report: In this particular case, there is only a very, very small window
of time in which the error can occur. It might also be influenced
considerably by compiler version and/or settings. Which linux distro do you
use? Both the bug reporter and I am on debian.


I'm on Fedora 20, which has gcc 4.8.3.


But I have used LyX with
threaded export enabled, on Linux, for a very long time, with large
documents with lots of children,
and I've never seen anything like this crash. I thus remain curious
about the details of the document
itself. Is there something special about it that is triggering this?

It uses xelatex and biber, apart from that I don't know, but I did not
really look at the contents. As I wrote in trac: Do we have a tool to
anonymize documents currently? If not I'll write an lfun, because we
definitely need a test case which can be tested by others, and I am not
going to anonymize the document by hand.


Scott mentioned something there.

Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 02:39 PM, Abdelrazak Younes wrote:

Hi Georg,

Warning: I haven't read LyX source code for a long long time :-)

On 19/11/2014 22:43, Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ 
gives
only a relatively weak thread-safety guarantee about std::basic 
string. It
is not completely thread-safe in the POSIX sense. The reason for this 
lies

in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already 
acknowledged
that they will get rid of it, but since this is an ABI incompatible 
change
they want to wait until they have several such changes to do all in 
one go.
For details please see the excellent gcc bug report by James Kanze 
(who is a

C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document 
language.


This looks all rather theoretical, but in practice it means that it 
is quite
easy to produce crashes if you use LyX built with #define 
EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child 
setups
seem to be much more vulnerable). The thing which seems to be most 
affected
is Language::babel_, which is copied a lot, but problems could arise 
from

all std::string and docstring variables.


I am not questioning your reasoning but I think STL thread safety 
limitation has limited impact in our case. AFAIR, the Buffer is cloned 
in in the current and main thread so we have no race condition here. 
After that the cloned buffer shares nothing with the original Buffer 
and can thus be used safely in another thread.


Yes, that is correct as to how the cloning happens. I'm no expert as to 
the rest.


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Abdelrazak Younes wrote:

 Hi Georg,
 
 Warning: I haven't read LyX source code for a long long time :-)

This is no problem, I think the main point can be understood quite well 
without code details.

 I am not questioning your reasoning but I think STL thread safety
 limitation has limited impact in our case. AFAIR, the Buffer is cloned
 in in the current and main thread so we have no race condition here.
 After that the cloned buffer shares nothing with the original Buffer and
 can thus be used safely in another thread.

Yes, this is true if you look at the involved data structures from the 
outside. If you look at the internal std::basic string members in gdb after 
the crash then you see that two instances which where both originally 
created from the document language Language::babel_ use the same pointer to 
the string data internally. This is of course no surprise if you know that 
copy-on-write is used, but it also means that the two buffers are _not_ 
completely independant from each other. BTW, I am pretty sure that the 
cloning itself works very reliably.

 I believe that there could be many reasons that could cause a crash
 before incriminating the stl string, etc:
 * Gui elements (message passing?)
 * Graphics preview elements

I don't think so in this case. The crash happens because internal 
std::basic_string members are corrupt, and it is almost always a string 
created from Language::babel_. Furthermore, I was not able to reproduce the 
crash after changing

std::string const  babel() const { return babel_; }

to

std::string const babel() const { return babel_.c_str(); }

in Language.h (which circumvents copy-on-write).

I also did tests with valgrind. memcheck reported an unrelated error in 
LaTeX log file parsing which I fixed, and helgrind pointed at 
std::basic_string (see log in trac). I am not 100% sure if the helgrind log 
does not contain false positives, but it does at least not contain any hints 
to other causes of the problem.

 Why not compile with c++11 option? I guess basic_string won't do
 copy-on-write if compiled this way?

Because the GNU libstdc++ is not conformant in this aspect: It uses copy-on-
write even with C++11.

 Well, IIRC I developed buffer cloning and initial threaded export using
 Linux :-)
 Maybe Peter Kuemmel continued the work under Windows but I don't think
 so...

Sorry, then my memory was wrong. I thought that Vincent developed this.

BTW, I would love to be proven wrong, because I find this issue rather 
scary, but up to now I did not find any convincing argument, so please try 
harder;-)


Georg

Re: Python 3 incompatibilities still exist right?

[stefano franchi]

On Thu, Nov 20, 2014 at 11:36 AM, Richard Heck rgh...@lyx.org wrote:

  On 11/20/2014 09:55 AM, stefano franchi wrote:

 On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak skost...@lyx.org wrote:

 I cleaned RELEASE-NOTES for the master branch. This note is still
 relevant right?

 - LyX needs to be run under Python 2 and will not work properly on systems
   where Python 3 is the default binary. See bug #7030 to know how to fix
   this properly, since simple sheebang conversion in *.py files will not
   be enough.


 I've seen some references here and there (e.g. Benjamin's work on
 features/python3), but Python 3 incompatibility is still in master
 right?


  Actually, I thought this problem was solved a couple of years or so ago
 (about Lyx? 2.0.2)
 I used to have problems on my ArchLinux box (with python 3 as a default)
 but that is no longer
  the case.


 There are two different issues here: (i) Will LyX work when Python 3 is
 the default? (ii) Will
 LyX work with Python 3? The answer to the former is Yes, as long as
 Python 2 is available:
 We now look for it explicitly. The answer to the latter, I believe, is
 No: The LyX scripts will
 not run under Python 3, though some work has been done on that, as Scott
 mentioned.


That's my experience on Linux. However, Alex seemed to suggest (i) is not
completely true on Windows. If I understood his point correctly, LyX will
work when python 3 is the default by installing its own python2
interpreter, which proceeds to break things (which things, I am not sure.
Alex did not say).

So perhaps there are three cases:

(i) Will LyX work when Python 3 is the default, but Python 2 is available?

(ii) Will LyX work with Python 3?

(iii) Will LyX work when Python 3 is the default and LyX install its own
version of Python?

And he answers are yes, no, kind of?


S.


-- 
__
Stefano Franchi

stefano.fran...@gmail.com stef...@tamu.edu
http://stefano.cleinias.org

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Richard Heck wrote:

 I'm on Fedora 20, which has gcc 4.8.3.

I have gcc 4.7.2.

 Scott mentioned something there.

I see. I installed pLyX, but I fear my way of thinking is very different 
from the one in the pLyX system. At least I did not manage to get it to do 
what I want.


Georg

Re: Thread problems with std::basic_string and GNU libstdc++

[Enrico Forestieri]

On Thu, Nov 20, 2014 at 09:04:39PM +0100, Georg Baum wrote:
 Abdelrazak Younes wrote:
  I believe that there could be many reasons that could cause a crash
  before incriminating the stl string, etc:
  * Gui elements (message passing?)
  * Graphics preview elements
 
 I don't think so in this case. The crash happens because internal 
 std::basic_string members are corrupt, and it is almost always a string 
 created from Language::babel_. Furthermore, I was not able to reproduce the 
 crash after changing
 
 std::string const  babel() const { return babel_; }
 
 to
 
 std::string const babel() const { return babel_.c_str(); }
 
 in Language.h (which circumvents copy-on-write).

Cannot a deep copy be forced after cloning? What about recursing over
all non-const strings and doing something like str = str.c_str() ?

-- 
Enrico

Re: Thread problems with std::basic_string and GNU libstdc++

[Cyrille Artho]

Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ gives
only a relatively weak thread-safety guarantee about std::basic string. It
is not completely thread-safe in the POSIX sense. The reason for this lies
in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already acknowledged
that they will get rid of it, but since this is an ABI incompatible change
they want to wait until they have several such changes to do all in one go.
For details please see the excellent gcc bug report by James Kanze (who is a
C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document language.

This looks all rather theoretical, but in practice it means that it is quite
easy to produce crashes if you use LyX built with #define EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child setups
seem to be much more vulnerable). The thing which seems to be most affected
is Language::babel_, which is copied a lot, but problems could arise from
all std::string and docstring variables.

What do we do now? I can think of several alternatives:

1) Do nothing, because my findings are incorrect. I don't believe that of
course, but I would appreciate it if somebody would try to find flaws in my
reasoning.

2) Disable EXPORT_in_THREAD if GNU libstdc++ is found. This is an easy and
safe solution, with the disadvantage that it removes a feature (but I think
that this is much better than risking crashes).

3) Derive an own class lyx::basic_string from std::basic_string which
circumvents copy-on-wrtite in the copy-constructor and assignment operator
by creating the copy from c_str(). Then use it

namespace lyx {
typedef basic_stringchar_type docstring;
typedef basic_stringchar string;
}

and replace all occurances of std::string with lyx::string. This would of
course need a lot of testing, but I believe it could work.

4) Do not use std::basic_string at all, but an own implementation which
could be stolen from STLport or clang (if the licenses permit that and are
compatible to the GPL, I did not check that). However, I believe that this
would create lots of interface problems with other libraries.


Does anybody see any other solution? I think for the stable branch we do not
really have any choice and need to implement 2).


The other question is: What can we learn for the future? AFAIK
EXPORT_in_THREAD was developed on windows, and it was probably well tested
on that platform. However, one basic lesson to learn is that such important
infrastructure changes need thorough testing on all supported platforms, and
I don't think this did happen in this case. Despite qt hides many
differences for us there are still a lot of platform specific issues left.


Cheers,
Georg



Hi Georg,
Thank you for bringing this up. You are almost certainly right, so I agree 
with not using option (1) :-)
Option (2) is the easiest fix, but we could also introduce an extra lock to 
implement option (3). This would wrap all operations by an extra locking step.
If the buffer that gets corrupted in libstdc++ is shared between different 
basic_string instances, then we would have to use a single global lock, 
which makes (3) behave like (2). So (3) only makes sense if we can 
introduce an extra lock field in the derived class, assuming that the 
internal buffers in libstdc++ each operate only on one basic_string 
instance. Of course, we should confirm this first before assuming anything.


To those who did not think they observed the crashes: Such crashes may be 
rare (1 in 100,000 uses), so unless you are looking for them and force them 
to occur with tool support, you are not going to notice this in everyday 
use. A random crash every couple of weeks will likely not be noticed as a 
specific problem.

--
Regards,
Cyrille Artho - http://artho.com/
The universe is like a safe to which there is a combination --
but the combination is locked up in the safe.
-- Peter DeVries

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 07:07 PM, Cyrille Artho wrote:
To those who did not think they observed the crashes: Such crashes may 
be rare (1 in 100,000 uses), so unless you are looking for them and 
force them to occur with tool support, you are not going to notice 
this in everyday use. A random crash every couple of weeks will likely 
not be noticed as a specific problem.


I agree with this, of course. But then: Why does this particular file 
crash *reliably*? when so many others do not?


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Cyrille Artho]

Richard Heck wrote:

On 11/20/2014 07:07 PM, Cyrille Artho wrote:

To those who did not think they observed the crashes: Such crashes may
be rare (1 in 100,000 uses), so unless you are looking for them and
force them to occur with tool support, you are not going to notice
this in everyday use. A random crash every couple of weeks will likely
not be noticed as a specific problem.


I agree with this, of course. But then: Why does this particular file
crash *reliably*? when so many others do not?

Richard

The frequency of a crash cannot be stated exactly. In some cases crashes 
may be rare but in others they may occur (say) half of the time you use 
certain multi-threaded code. This depends on how many variable accesses are 
involved in the code that competes for the same memory location, among 
other things.


In any case I think it's no problem to sacrifice a bit of performance to 
prevent possible crashes, even if the crashes are rare.

--
Regards,
Cyrille Artho - http://artho.com/
We are all like soldiers,
crouching behind the fortifications we have raised.
-- Steven Erikson, Midnight Tides

Python 3 incompatibilities still exist right?

[Scott Kostyshak]

I cleaned RELEASE-NOTES for the master branch. This note is still
relevant right?

- LyX needs to be run under Python 2 and will not work properly on systems
  where Python 3 is the default binary. See bug #7030 to know how to fix
  this properly, since simple sheebang conversion in *.py files will not
  be enough.


I've seen some references here and there (e.g. Benjamin's work on
features/python3), but Python 3 incompatibility is still in master
right?

Scott

Re: Clear RELEASE-NOTES on master?

[Scott Kostyshak]

On Tue, Nov 18, 2014 at 3:05 AM, Jean-Marc Lasgouttes
 wrote:
> Le 18/11/14 02:14, Scott Kostyshak a écrit :
>>
>> The RELEASE-NOTES on master is for 2.1.0. Should it be cleaned so we
>> can add 2.2.0 stuff or is this done somewhere else that I forgot? I
>> can do the work but I wanted to check in first.
>
>
> Sure.

Done at ef71f667.

Scott

Re: Help with patch for "word selection mode" on double click

[Scott Kostyshak]

On Tue, Oct 28, 2014 at 7:03 AM, Scott Kostyshak  wrote:
> I have a patch that enables "word selection mode" on double click. It
> seems to work well. The patch is here:
>
> http://www.lyx.org/trac/attachment/ticket/7890/0001-Fix-9160-and-7890.patch
>
> The only major issue is that when dragging to the left, you cannot
> highlight the first word. Does anyone have an idea for where to fix
> this?
>
> The following patch might give a clue as to where the problem is. When
> applied, you can select the first word. It is not meant to be taken as
> a correct patch:
>
> http://www.lyx.org/trac/attachment/ticket/7890/0002-HACK-just-to-show-where-the-problem-is.patch

I'm still looking for help on this in case anyone is motivated.

Scott

Re: Python 3 incompatibilities still exist right?

[stefano franchi]

On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak  wrote:

> I cleaned RELEASE-NOTES for the master branch. This note is still
> relevant right?
>
> - LyX needs to be run under Python 2 and will not work properly on systems
>   where Python 3 is the default binary. See bug #7030 to know how to fix
>   this properly, since simple sheebang conversion in *.py files will not
>   be enough.
>
>
> I've seen some references here and there (e.g. Benjamin's work on
> features/python3), but Python 3 incompatibility is still in master
> right?
>

Actually, I thought this problem was solved a couple of years or so ago
(about Lyx? 2.0.2)
I used to have problems on my ArchLinux box (with python 3 as a default)
but that is no longer
the case.

Stefano



-- 
__
Stefano Franchi

stefano.fran...@gmail.com 
http://stefano.cleinias.org

Re: Python 3 incompatibilities still exist right?

[Alex Vergara Gil]

The incompatibilities still exist at least in Windows sistems, since
LyX installer bundles a python distribution itself and creates a
conflict when python3 is the default. Moreover you cannot use python3
within LyX in Windows, because you will beak the system, LyX will not
reconfigure anymore!!
Just my experiences
LyX 2.1.2, Win7x64, Anaconda 2.1(python 3.4)

Re: Python 3 incompatibilities still exist right?

[Richard Heck]

On 11/20/2014 09:55 AM, stefano franchi wrote:
On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak > wrote:


I cleaned RELEASE-NOTES for the master branch. This note is still
relevant right?

- LyX needs to be run under Python 2 and will not work properly on
systems
  where Python 3 is the default binary. See bug #7030 to know how
to fix
  this properly, since simple sheebang conversion in *.py files
will not
  be enough.


I've seen some references here and there (e.g. Benjamin's work on
features/python3), but Python 3 incompatibility is still in master
right?


Actually, I thought this problem was solved a couple of years or so 
ago (about Lyx? 2.0.2)
I used to have problems on my ArchLinux box (with python 3 as a 
default) but that is no longer

the case.


There are two different issues here: (i) Will LyX work when Python 3 is 
the default? (ii) Will
LyX work with Python 3? The answer to the former is "Yes", as long as 
Python 2 is available:
We now look for it explicitly. The answer to the latter, I believe, is 
"No": The LyX scripts will
not run under Python 3, though some work has been done on that, as Scott 
mentioned.


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Richard Heck wrote:

> I'm not well informed enough about implementation details to have an
> opinion about whether this
> diagnosis is correct. But I am somewhat puzzled that we are only seeing
> this issue now. Of course,
> there may have been other random crashes that people haven't reported.

This is an important point, and I asked myself the same question as well.  I 
gues the most plausible explanation is what James wrote in the gcc bug 
report: "In this particular case, there is only a very, very small window
of time in which the error can occur." It might also be influenced 
considerably by compiler version and/or settings. Which linux distro do you 
use? Both the bug reporter and I am on debian.

> But I have used LyX with
> threaded export enabled, on Linux, for a very long time, with large
> documents with lots of children,
> and I've never seen anything like this crash. I thus remain curious
> about the details of the document
> itself. Is there something special about it that is triggering this?

It uses xelatex and biber, apart from that I don't know, but I did not 
really look at the contents. As I wrote in trac: Do we have a tool to 
anonymize documents currently? If not I'll write an lfun, because we 
definitely need a test case which can be tested by others, and I am not 
going to anonymize the document by hand.

> Obviously, we need some substantial discussion before making any
> decision about what to do. I'm
> going to cc a bunch of people to try to get their attention.

Definitely.


Georg

Re: Thread problems with std::basic_string and GNU libstdc++

[Abdelrazak Younes]

Hi Georg,

Warning: I haven't read LyX source code for a long long time :-)

On 19/11/2014 22:43, Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ gives
only a relatively weak thread-safety guarantee about std::basic string. It
is not completely thread-safe in the POSIX sense. The reason for this lies
in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already acknowledged
that they will get rid of it, but since this is an ABI incompatible change
they want to wait until they have several such changes to do all in one go.
For details please see the excellent gcc bug report by James Kanze (who is a
C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document language.

This looks all rather theoretical, but in practice it means that it is quite
easy to produce crashes if you use LyX built with #define EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child setups
seem to be much more vulnerable). The thing which seems to be most affected
is Language::babel_, which is copied a lot, but problems could arise from
all std::string and docstring variables.


I am not questioning your reasoning but I think STL thread safety 
limitation has limited impact in our case. AFAIR, the Buffer is cloned 
in in the current and main thread so we have no race condition here. 
After that the cloned buffer shares nothing with the original Buffer and 
can thus be used safely in another thread.





What do we do now? I can think of several alternatives:

1) Do nothing, because my findings are incorrect. I don't believe that of
course, but I would appreciate it if somebody would try to find flaws in my
reasoning.


I believe that there could be many reasons that could cause a crash 
before incriminating the stl string, etc:

* Gui elements (message passing?)
* Graphics preview elements



2) Disable EXPORT_in_THREAD if GNU libstdc++ is found. This is an easy and
safe solution, with the disadvantage that it removes a feature (but I think
that this is much better than risking crashes).

3) Derive an own class lyx::basic_string from std::basic_string which
circumvents copy-on-wrtite in the copy-constructor and assignment operator
by creating the copy from c_str(). Then use it

namespace lyx {
typedef basic_string docstring;
typedef basic_string string;
}

and replace all occurances of std::string with lyx::string. This would of
course need a lot of testing, but I believe it could work.

4) Do not use std::basic_string at all, but an own implementation which
could be stolen from STLport or clang (if the licenses permit that and are
compatible to the GPL, I did not check that). However, I believe that this
would create lots of interface problems with other libraries.


Does anybody see any other solution? I think for the stable branch we do not
really have any choice and need to implement 2).


Why not compile with c++11 option? I guess basic_string won't do 
copy-on-write if compiled this way?





The other question is: What can we learn for the future? AFAIK
EXPORT_in_THREAD was developed on windows, and it was probably well tested
on that platform. However, one basic lesson to learn is that such important
infrastructure changes need thorough testing on all supported platforms, and
I don't think this did happen in this case. Despite qt hides many
differences for us there are still a lot of platform specific issues left.


Well, IIRC I developed buffer cloning and initial threaded export using 
Linux :-)

Maybe Peter Kuemmel continued the work under Windows but I don't think so...

Thanks Georg, that was nice to discuss this.

Abdel.

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 02:35 PM, Georg Baum wrote:

Richard Heck wrote:


I'm not well informed enough about implementation details to have an
opinion about whether this
diagnosis is correct. But I am somewhat puzzled that we are only seeing
this issue now. Of course,
there may have been other random crashes that people haven't reported.

This is an important point, and I asked myself the same question as well.  I
gues the most plausible explanation is what James wrote in the gcc bug
report: "In this particular case, there is only a very, very small window
of time in which the error can occur." It might also be influenced
considerably by compiler version and/or settings. Which linux distro do you
use? Both the bug reporter and I am on debian.


I'm on Fedora 20, which has gcc 4.8.3.


But I have used LyX with
threaded export enabled, on Linux, for a very long time, with large
documents with lots of children,
and I've never seen anything like this crash. I thus remain curious
about the details of the document
itself. Is there something special about it that is triggering this?

It uses xelatex and biber, apart from that I don't know, but I did not
really look at the contents. As I wrote in trac: Do we have a tool to
anonymize documents currently? If not I'll write an lfun, because we
definitely need a test case which can be tested by others, and I am not
going to anonymize the document by hand.


Scott mentioned something there.

Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 02:39 PM, Abdelrazak Younes wrote:

Hi Georg,

Warning: I haven't read LyX source code for a long long time :-)

On 19/11/2014 22:43, Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ 
gives
only a relatively weak thread-safety guarantee about std::basic 
string. It
is not completely thread-safe in the POSIX sense. The reason for this 
lies

in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already 
acknowledged
that they will get rid of it, but since this is an ABI incompatible 
change
they want to wait until they have several such changes to do all in 
one go.
For details please see the excellent gcc bug report by James Kanze 
(who is a

C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document 
language.


This looks all rather theoretical, but in practice it means that it 
is quite
easy to produce crashes if you use LyX built with #define 
EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child 
setups
seem to be much more vulnerable). The thing which seems to be most 
affected
is Language::babel_, which is copied a lot, but problems could arise 
from

all std::string and docstring variables.


I am not questioning your reasoning but I think STL thread safety 
limitation has limited impact in our case. AFAIR, the Buffer is cloned 
in in the current and main thread so we have no race condition here. 
After that the cloned buffer shares nothing with the original Buffer 
and can thus be used safely in another thread.


Yes, that is correct as to how the cloning happens. I'm no expert as to 
the rest.


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Abdelrazak Younes wrote:

> Hi Georg,
> 
> Warning: I haven't read LyX source code for a long long time :-)

This is no problem, I think the main point can be understood quite well 
without code details.

> I am not questioning your reasoning but I think STL thread safety
> limitation has limited impact in our case. AFAIR, the Buffer is cloned
> in in the current and main thread so we have no race condition here.
> After that the cloned buffer shares nothing with the original Buffer and
> can thus be used safely in another thread.

Yes, this is true if you look at the involved data structures from the 
outside. If you look at the internal std::basic string members in gdb after 
the crash then you see that two instances which where both originally 
created from the document language Language::babel_ use the same pointer to 
the string data internally. This is of course no surprise if you know that 
copy-on-write is used, but it also means that the two buffers are _not_ 
completely independant from each other. BTW, I am pretty sure that the 
cloning itself works very reliably.

> I believe that there could be many reasons that could cause a crash
> before incriminating the stl string, etc:
> * Gui elements (message passing?)
> * Graphics preview elements

I don't think so in this case. The crash happens because internal 
std::basic_string members are corrupt, and it is almost always a string 
created from Language::babel_. Furthermore, I was not able to reproduce the 
crash after changing

std::string const & babel() const { return babel_; }

to

std::string const babel() const { return babel_.c_str(); }

in Language.h (which circumvents copy-on-write).

I also did tests with valgrind. memcheck reported an unrelated error in 
LaTeX log file parsing which I fixed, and helgrind pointed at 
std::basic_string (see log in trac). I am not 100% sure if the helgrind log 
does not contain false positives, but it does at least not contain any hints 
to other causes of the problem.

> Why not compile with c++11 option? I guess basic_string won't do
> copy-on-write if compiled this way?

Because the GNU libstdc++ is not conformant in this aspect: It uses copy-on-
write even with C++11.

> Well, IIRC I developed buffer cloning and initial threaded export using
> Linux :-)
> Maybe Peter Kuemmel continued the work under Windows but I don't think
> so...

Sorry, then my memory was wrong. I thought that Vincent developed this.

BTW, I would love to be proven wrong, because I find this issue rather 
scary, but up to now I did not find any convincing argument, so please try 
harder;-)


Georg

Re: Python 3 incompatibilities still exist right?

[stefano franchi]

On Thu, Nov 20, 2014 at 11:36 AM, Richard Heck  wrote:

>  On 11/20/2014 09:55 AM, stefano franchi wrote:
>
> On Thu, Nov 20, 2014 at 2:09 AM, Scott Kostyshak  wrote:
>
>> I cleaned RELEASE-NOTES for the master branch. This note is still
>> relevant right?
>>
>> - LyX needs to be run under Python 2 and will not work properly on systems
>>   where Python 3 is the default binary. See bug #7030 to know how to fix
>>   this properly, since simple sheebang conversion in *.py files will not
>>   be enough.
>>
>>
>> I've seen some references here and there (e.g. Benjamin's work on
>> features/python3), but Python 3 incompatibility is still in master
>> right?
>>
>
>  Actually, I thought this problem was solved a couple of years or so ago
> (about Lyx? 2.0.2)
> I used to have problems on my ArchLinux box (with python 3 as a default)
> but that is no longer
>  the case.
>
>
> There are two different issues here: (i) Will LyX work when Python 3 is
> the default? (ii) Will
> LyX work with Python 3? The answer to the former is "Yes", as long as
> Python 2 is available:
> We now look for it explicitly. The answer to the latter, I believe, is
> "No": The LyX scripts will
> not run under Python 3, though some work has been done on that, as Scott
> mentioned.
>
>
That's my experience on Linux. However, Alex seemed to suggest (i) is not
completely true on Windows. If I understood his point correctly, LyX will
work when python 3 is the default by installing its own python2
interpreter, which proceeds to break things (which things, I am not sure.
Alex did not say).

So perhaps there are three cases:

(i) Will LyX work when Python 3 is the default, but Python 2 is available?

(ii) Will LyX work with Python 3?

(iii) Will LyX work when Python 3 is the default and LyX install its own
version of Python?

And he answers are yes, no, kind of?


S.


-- 
__
Stefano Franchi

stefano.fran...@gmail.com 
http://stefano.cleinias.org

Re: Thread problems with std::basic_string and GNU libstdc++

[Georg Baum]

Richard Heck wrote:

> I'm on Fedora 20, which has gcc 4.8.3.

I have gcc 4.7.2.

> Scott mentioned something there.

I see. I installed pLyX, but I fear my way of thinking is very different 
from the one in the pLyX system. At least I did not manage to get it to do 
what I want.


Georg

Re: Thread problems with std::basic_string and GNU libstdc++

[Enrico Forestieri]

On Thu, Nov 20, 2014 at 09:04:39PM +0100, Georg Baum wrote:
> Abdelrazak Younes wrote:
> > I believe that there could be many reasons that could cause a crash
> > before incriminating the stl string, etc:
> > * Gui elements (message passing?)
> > * Graphics preview elements
> 
> I don't think so in this case. The crash happens because internal 
> std::basic_string members are corrupt, and it is almost always a string 
> created from Language::babel_. Furthermore, I was not able to reproduce the 
> crash after changing
> 
> std::string const & babel() const { return babel_; }
> 
> to
> 
> std::string const babel() const { return babel_.c_str(); }
> 
> in Language.h (which circumvents copy-on-write).

Cannot a deep copy be forced after cloning? What about recursing over
all non-const strings and doing something like str = str.c_str() ?

-- 
Enrico

Re: Thread problems with std::basic_string and GNU libstdc++

[Cyrille Artho]

Georg Baum wrote:

Hi,

while investigating http://www.lyx.org/trac/ticket/9336 I found out a
fundamental problem with our multithreaded export: The GNU libstdc++ gives
only a relatively weak thread-safety guarantee about std::basic string. It
is not completely thread-safe in the POSIX sense. The reason for this lies
in the employed copy-on-write technique, which is not used in other STL
implementations such as the one in clang or MSVC. It is even explicitly
forbidden in the new C++11 standard, and the gcc folks already acknowledged
that they will get rid of it, but since this is an ABI incompatible change
they want to wait until they have several such changes to do all in one go.
For details please see the excellent gcc bug report by James Kanze (who is a
C++ expert) at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334,
especially https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334#c21, which
describes almost exactly the situation we have with the document language.

This looks all rather theoretical, but in practice it means that it is quite
easy to produce crashes if you use LyX built with #define EXPORT_in_THREAD 1
and gcc, and start an export, depending on the document (master-child setups
seem to be much more vulnerable). The thing which seems to be most affected
is Language::babel_, which is copied a lot, but problems could arise from
all std::string and docstring variables.

What do we do now? I can think of several alternatives:

1) Do nothing, because my findings are incorrect. I don't believe that of
course, but I would appreciate it if somebody would try to find flaws in my
reasoning.

2) Disable EXPORT_in_THREAD if GNU libstdc++ is found. This is an easy and
safe solution, with the disadvantage that it removes a feature (but I think
that this is much better than risking crashes).

3) Derive an own class lyx::basic_string from std::basic_string which
circumvents copy-on-wrtite in the copy-constructor and assignment operator
by creating the copy from c_str(). Then use it

namespace lyx {
typedef basic_string docstring;
typedef basic_string string;
}

and replace all occurances of std::string with lyx::string. This would of
course need a lot of testing, but I believe it could work.

4) Do not use std::basic_string at all, but an own implementation which
could be stolen from STLport or clang (if the licenses permit that and are
compatible to the GPL, I did not check that). However, I believe that this
would create lots of interface problems with other libraries.


Does anybody see any other solution? I think for the stable branch we do not
really have any choice and need to implement 2).


The other question is: What can we learn for the future? AFAIK
EXPORT_in_THREAD was developed on windows, and it was probably well tested
on that platform. However, one basic lesson to learn is that such important
infrastructure changes need thorough testing on all supported platforms, and
I don't think this did happen in this case. Despite qt hides many
differences for us there are still a lot of platform specific issues left.


Cheers,
Georg



Hi Georg,
Thank you for bringing this up. You are almost certainly right, so I agree 
with not using option (1) :-)
Option (2) is the easiest fix, but we could also introduce an extra lock to 
implement option (3). This would wrap all operations by an extra locking step.
If the buffer that gets corrupted in libstdc++ is shared between different 
basic_string instances, then we would have to use a single global lock, 
which makes (3) behave like (2). So (3) only makes sense if we can 
introduce an extra lock field in the derived class, assuming that the 
internal buffers in libstdc++ each operate only on one basic_string 
instance. Of course, we should confirm this first before assuming anything.


To those who did not think they observed the crashes: Such crashes may be 
rare (1 in 100,000 uses), so unless you are looking for them and force them 
to occur with tool support, you are not going to notice this in everyday 
use. A random crash every couple of weeks will likely not be noticed as a 
specific problem.

--
Regards,
Cyrille Artho - http://artho.com/
The universe is like a safe to which there is a combination --
but the combination is locked up in the safe.
-- Peter DeVries

Re: Thread problems with std::basic_string and GNU libstdc++

[Richard Heck]

On 11/20/2014 07:07 PM, Cyrille Artho wrote:
To those who did not think they observed the crashes: Such crashes may 
be rare (1 in 100,000 uses), so unless you are looking for them and 
force them to occur with tool support, you are not going to notice 
this in everyday use. A random crash every couple of weeks will likely 
not be noticed as a specific problem.


I agree with this, of course. But then: Why does this particular file 
crash *reliably*? when so many others do not?


Richard

Re: Thread problems with std::basic_string and GNU libstdc++

[Cyrille Artho]

Richard Heck wrote:

On 11/20/2014 07:07 PM, Cyrille Artho wrote:

To those who did not think they observed the crashes: Such crashes may
be rare (1 in 100,000 uses), so unless you are looking for them and
force them to occur with tool support, you are not going to notice
this in everyday use. A random crash every couple of weeks will likely
not be noticed as a specific problem.


I agree with this, of course. But then: Why does this particular file
crash *reliably*? when so many others do not?

Richard

The frequency of a crash cannot be stated exactly. In some cases crashes 
may be rare but in others they may occur (say) half of the time you use 
certain multi-threaded code. This depends on how many variable accesses are 
involved in the code that competes for the same memory location, among 
other things.


In any case I think it's no problem to sacrifice a bit of performance to 
prevent possible crashes, even if the crashes are rare.

--
Regards,
Cyrille Artho - http://artho.com/
We are all like soldiers,
crouching behind the fortifications we have raised.
-- Steven Erikson, "Midnight Tides"