Re: Python error
On 5/18/20 1:39 PM, Enrico Forestieri wrote: > On Sat, Mar 14, 2020 at 05:27:19PM +0100, Enrico Forestieri wrote: > >> On Sat, Mar 14, 2020 at 04:36:41PM +0100, Jürgen Spitzmüller wrote: >>> Am Samstag, den 14.03.2020, 16:22 +0100 schrieb Enrico Forestieri: Please, try the attached patch. >>> This fixes the problem for me. >> I have extended the patch to some other obvious cases and committed it. >> >> Riki, I think this (bd6d09fc) should also go to stable. > As well as 92c4bb46. OK! Riki -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
On Sat, Mar 14, 2020 at 05:27:19PM +0100, Enrico Forestieri wrote: > On Sat, Mar 14, 2020 at 04:36:41PM +0100, Jürgen Spitzmüller wrote: > > Am Samstag, den 14.03.2020, 16:22 +0100 schrieb Enrico Forestieri: > > > Please, try the attached patch. > > > > This fixes the problem for me. > > I have extended the patch to some other obvious cases and committed it. > > Riki, I think this (bd6d09fc) should also go to stable. As well as 92c4bb46. -- Enrico -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
On Sat, Mar 14, 2020 at 04:36:41PM +0100, Jürgen Spitzmüller wrote: > Am Samstag, den 14.03.2020, 16:22 +0100 schrieb Enrico Forestieri: > > Please, try the attached patch. > > This fixes the problem for me. I have extended the patch to some other obvious cases and committed it. Riki, I think this (bd6d09fc) should also go to stable. -- Enrico -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
On 2020-03-14, Jürgen Spitzmüller wrote: > [-- Type: text/plain, Encoding: quoted-printable --] > Am Samstag, den 14.03.2020, 12:29 + schrieb Guenter Milde: >> I'll have a look if you tell me how to reproduce... > MWE attached. The cuplrit here seem to be log messages such as the > following (in latin9, as the document is set to that): > Package csquotes Info: Allocating character ». Thanks. Enricos patch seems good to me. Günter -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
Am Samstag, den 14.03.2020, 16:22 +0100 schrieb Enrico Forestieri: > Please, try the attached patch. This fixes the problem for me. Thanks, Enrico. Jürgen signature.asc Description: This is a digitally signed message part -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
On Fri, Mar 13, 2020 at 10:06:48AM +0100, Jürgen Spitzmüller wrote:
> I get the following python traceback in master:
>
> Traceback (most recent call last):
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview2bitmap.py",
> line 594, in
> sys.exit(main(sys.argv)[0])
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview2bitmap.py",
> line 472, in main
> latex_status, latex_stdout = run_latex(latex, latex_file, bibtex)
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview_tools.py",
> line 297, in run_latex
> if string_in_file("Warning: Citation", log_file):
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview_tools.py",
> line 317, in string_in_file
> for line in f.readlines():
> File "/usr/lib64/python3.8/codecs.py", line 322, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position
> 4180: invalid start byte
Please, try the attached patch.
--
Enrico
diff --git a/lib/scripts/legacy_lyxpreview2ppm.py
b/lib/scripts/legacy_lyxpreview2ppm.py
index a5eb05a129..6e3c349f35 100644
--- a/lib/scripts/legacy_lyxpreview2ppm.py
+++ b/lib/scripts/legacy_lyxpreview2ppm.py
@@ -154,10 +154,10 @@ def legacy_extract_metrics_info(log_file):
return results
def extract_resolution(log_file, dpi):
-fontsize_re = re.compile("Preview: Fontsize")
-magnification_re = re.compile("Preview: Magnification")
-extract_decimal_re = re.compile("([0-9\.]+)")
-extract_integer_re = re.compile("([0-9]+)")
+fontsize_re = re.compile(b"Preview: Fontsize")
+magnification_re = re.compile(b"Preview: Magnification")
+extract_decimal_re = re.compile(b"([0-9\.]+)")
+extract_integer_re = re.compile(b"([0-9]+)")
found_fontsize = 0
found_magnification = 0
@@ -167,7 +167,7 @@ def extract_resolution(log_file, dpi):
fontsize = 10.0
try:
-for line in open(log_file, 'r').readlines():
+for line in open(log_file, 'rb').readlines():
if found_fontsize and found_magnification:
break
diff --git a/lib/scripts/lyxpreview_tools.py b/lib/scripts/lyxpreview_tools.py
index 93964083a5..51917b8cdf 100644
--- a/lib/scripts/lyxpreview_tools.py
+++ b/lib/scripts/lyxpreview_tools.py
@@ -313,9 +313,9 @@ def run_tex(tex, tex_file):
def string_in_file(string, infile):
if not os.path.isfile(infile):
return False
-f = open(infile, 'r')
+f = open(infile, 'rb')
for line in f.readlines():
-if string in line:
+if string.encode() in line:
f.close()
return True
f.close()
--
lyx-devel mailing list
lyx-devel@lists.lyx.org
http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
Am Samstag, den 14.03.2020, 12:29 + schrieb Guenter Milde: > I'll have a look if you tell me how to reproduce... MWE attached. The cuplrit here seem to be log messages such as the following (in latin9, as the document is set to that): Package csquotes Info: Allocating character ». Thanks, Jürgen laatin9log.lyx Description: application/lyx signature.asc Description: This is a digitally signed message part -- lyx-devel mailing list lyx-devel@lists.lyx.org http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error
On 2020-03-13, Jürgen Spitzmüller wrote:
> I get the following python traceback in master:
> Traceback (most recent call last):
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview2bitmap.py",
> line 594, in
> sys.exit(main(sys.argv)[0])
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview2bitmap.py",
> line 472, in main
> latex_status, latex_stdout = run_latex(latex, latex_file, bibtex)
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview_tools.py",
> line 297, in run_latex
> if string_in_file("Warning: Citation", log_file):
> File "/home/juergen/lyx/lyx-devel/lib/scripts/lyxpreview_tools.py",
> line 317, in string_in_file
> for line in f.readlines():
> File "/usr/lib64/python3.8/codecs.py", line 322, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position
> 4180: invalid start byte
I'll have a look if you tell me how to reproduce...
Günter
--
lyx-devel mailing list
lyx-devel@lists.lyx.org
http://lists.lyx.org/mailman/listinfo/lyx-devel
Re: Python error (postats.py)
Am Montag, den 15.07.2019, 11:29 + schrieb Guenter Milde: > I am following the list via the gmane news interface and cannot send > or read > attachemnts in my newsreader. Not good. > You can simply save the complete post and pass it to `git apply`, > though. > Or I could just commit to master. Please do. I don't want to fiddle with copy and paste. Jürgen > > Günter > signature.asc Description: This is a digitally signed message part
Re: Python error (postats.py)
On 2019-07-15, Jürgen Spitzmüller wrote: > [-- Type: text/plain, Encoding: quoted-printable --] > Am Sonntag, den 14.07.2019, 21:28 + schrieb Guenter Milde: >> It did not. Is it required? > I believe so. >> The patch below makes it run with both, 2.7 and 3.5 in a simple >> test: > Could you post a real patch (as attachment)? I am following the list via the gmane news interface and cannot send or read attachemnts in my newsreader. You can simply save the complete post and pass it to `git apply`, though. Or I could just commit to master. Günter
Re: Python error (postats.py)
Am Sonntag, den 14.07.2019, 21:28 + schrieb Guenter Milde: > It did not. Is it required? I believe so. > The patch below makes it run with both, 2.7 and 3.5 in a simple > test: Could you post a real patch (as attachment)? Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error (configure.py)
On 2019-07-14, Jean-Marc Lasgouttes wrote:
> Le 14/07/2019 à 12:52, Kornel Benko a écrit :
>> Am Sonntag, 14. Juli 2019, 12:09:41 CEST schrieb Jean-Marc Lasgouttes:
>>> If I start lyx with a non-existing user directory, LyX fails.
>>> JMarc
>> I don't see it. Instead I see a new dialog stating that there are no latex
>> classes found,
>> and the option to reconfigure again. Checking this, lyx starts and
>> everything is OK.
> Yes, but before the dialog, the stuff I pasted is written to the console.
I can reproduce.
And I think it is fixed with the patch below.
However, I don't know whether it runs under 2.7: tested with
python configure.py
but this does not create the directory ~/.lyx2.4, so I am not sure.
Günter
diff --git a/lib/configure.py b/lib/configure.py
index 0b3c2a761c..ac3b15f0a7 100644
--- a/lib/configure.py
+++ b/lib/configure.py
@@ -1,4 +1,4 @@
-#! /usr/bin/python3
+#! /usr/bin/python
# -*- coding: utf-8 -*-
#
# file configure.py
@@ -11,6 +11,11 @@
from __future__ import print_function
import glob, logging, os, errno, re, shutil, subprocess, sys, stat
+if sys.version_info[0] < 3:
+import codecs
+open = codecs.open
+
+
# set up logging
logging.basicConfig(level = logging.DEBUG,
format = '%(levelname)s: %(message)s', # ignore application name
@@ -46,7 +51,7 @@ def addToRC(lines):
''' utility function: shortcut for appending lines to outfile
add newline at the end of lines.
'''
-if lines.strip() != '':
+if lines.strip():
writeToFile(outfile, lines + '\n', append = True)
logger.debug('Add to RC:\n' + lines + '\n\n')
@@ -540,7 +545,7 @@ def checkLatex(dtl_tools):
path, PPLATEX = checkProg('a DVI postprocessing program', ['pplatex $$i'])
#-
path, PLATEX = checkProg('pLaTeX, the Japanese LaTeX', ['platex $$i'])
-if PLATEX != '':
+if PLATEX:
# check if PLATEX is pLaTeX2e
writeToFile('chklatex.ltx', r'\nonstopmode\makeatletter\@@end')
# run platex on chklatex.ltx and check result
@@ -561,7 +566,7 @@ def checkLatex(dtl_tools):
else:
addToRC(r'\converter latex dvi"%s"
"latex,hyperref-driver=dvips"' % PPLATEX)
# no latex
-if LATEX != '':
+if LATEX:
# Check if latex is usable
writeToFile('chklatex.ltx', r'''
\nonstopmode
@@ -586,9 +591,9 @@ def checkLuatex():
''' Check if luatex is there '''
path, LUATEX = checkProg('LuaTeX', ['lualatex $$i'])
path, DVILUATEX = checkProg('LuaTeX (DVI)', ['dvilualatex $$i'])
-if LUATEX != '':
+if LUATEX:
addToRC(r'\converter luatex pdf5 "%s"
"latex=lualatex"' % LUATEX)
-if DVILUATEX != '':
+if DVILUATEX:
addToRC(r'\converter dviluatex dvi3"%s"
"latex=dvilualatex"' % DVILUATEX)
@@ -1109,7 +1114,7 @@ def checkConverterEntries():
'''])
path, lilypond = checkProg('a LilyPond -> EPS/PDF/PNG converter',
['lilypond'])
-if (lilypond != ''):
+if (lilypond):
version_string = cmdOutput("lilypond --version")
match = re.match('GNU LilyPond (\S+)', version_string)
if match:
@@ -1132,7 +1137,7 @@ def checkConverterEntries():
logger.info('+ found LilyPond, but could not extract version
number.')
#
path, lilypond_book = checkProg('a LilyPond book (LaTeX) -> LaTeX
converter', ['lilypond-book'])
-if (lilypond_book != ''):
+if (lilypond_book):
version_string = cmdOutput("lilypond-book --version")
match = re.match('(\S+)$', version_string)
if match:
@@ -1214,7 +1219,7 @@ def checkDocBook():
r'''\converter docbookdvi""""
\converter docbookhtml """"'''])
#
-if DOCBOOK != '':
+if DOCBOOK:
return ('yes', 'true', '\\def\\hasdocbook{yes}')
else:
return ('no', 'false', '')
@@ -1256,6 +1261,13 @@ def checkOtherEntries():
\copierprogram"python -tt $$s/scripts/ext_copy.py $$i $$o"
''')
+def _checkForClassExtension(x):
+'''if the extension for a latex class is not
+provided, add .cls to the classname'''
+if not '.' in x:
+return x.strip() + '.cls'
+else:
+return x.strip()
def processLayoutFile(file, bool_docbook):
''' process layout file and get a line of result
@@ -1284,46 +1296,39 @@ def processLayoutFile(file, bool_docbook):
"scrbook" "scrbook" "book (koma-script)" "false" "scrbook.cls" "Books"
"svjog" "svjour" "article (Springer - svjour/jog)" "false"
"svjour.cls,svjog.clo" ""
'''
-def checkForClassExtension(x):
-'''if the extension for a latex class is not
- provided, add .cls to the classname'''
-if not b'.' in x:
-return x.strip() + b'.cls'
-else:
-return x.strip()
classname = file.split(os.sep)[-1].split('.')[0]
#
Re: Python error (postats.py)
On 2019-07-14, Jürgen Spitzmüller wrote:
> [-- Type: text/plain, Encoding: quoted-printable --]
> Am Sonntag, den 14.07.2019, 08:17 + schrieb Guenter Milde:
>> This file is for Python2.
>> The patch below made it work with Python3.5 here.
> Does it also still work with python 2.7?
It did not. Is it required?
The patch below makes it run with both, 2.7 and 3.5 in a simple
test:
python3 postats.py "2.4.0dev" ar.po de.po
python postats.py "2.4.0dev" ar.po de.po
Günter
diff --git a/po/postats.py b/po/postats.py
index 8c5f058642..52690967da 100755
--- a/po/postats.py
+++ b/po/postats.py
@@ -37,6 +37,8 @@ ommitted = ('en.po')
import os
import sys
+import codecs
+from subprocess import Popen, PIPE
# Reset the locale
import locale
@@ -69,7 +71,7 @@ def read_pofile(pofile):
""" Read the header of the pofile and return it as a dictionary"""
header = {}
read_header = False
-for line in open(pofile):
+for line in codecs.open(pofile, encoding='utf8'):
line = line[:-1]
if line[:5] == 'msgid':
if read_header:
@@ -112,12 +114,13 @@ def run_msgfmt(pofile):
prop["email"] = prop["email"].replace(".", " ! ")
translator = header['Last-Translator'].split('<')[0].strip()
try:
-prop["translator"] =
translator.decode(charset).encode('ascii','xmlcharrefreplace')
+prop["translator"] = translator.encode('ascii','xmlcharrefreplace')
except LookupError:
prop["translator"] = translator
-p_in, p_out = os.popen4("msgfmt --statistics -o %s %s" % (gmofile, pofile))
-extract_number(p_out.readline(),
+P = Popen("msgfmt --statistics -o %s %s" % (gmofile, pofile),
+ shell=True, stdin=PIPE, stdout=PIPE, close_fds=True)
+extract_number(P.stdout.readline().decode(),
('translated', 'fuzzy', 'untranslated'),
prop)
return """
Re: Python error (configure.py)
Le 14/07/2019 à 12:52, Kornel Benko a écrit : Am Sonntag, 14. Juli 2019, 12:09:41 CEST schrieb Jean-Marc Lasgouttes: If I start lyx with a non-existing user directory, LyX fails. JMarc I don't see it. Instead I see a new dialog stating that there are no latex classes found, and the option to reconfigure again. Checking this, lyx starts and everything is OK. Yes, but before the dialog, the stuff I pasted is written to the console. HLarc
Re: Python error (configure.py)
Am Sonntag, 14. Juli 2019, 12:09:41 CEST schrieb Jean-Marc Lasgouttes: > If I start lyx with a non-existing user directory, LyX fails. > > JMarc > I don't see it. Instead I see a new dialog stating that there are no latex classes found, and the option to reconfigure again. Checking this, lyx starts and everything is OK. Kornel signature.asc Description: This is a digitally signed message part.
Re: Python error (postats.py)
Am Sonntag, den 14.07.2019, 08:17 + schrieb Guenter Milde: > This file is for Python2. > > The patch below made it work with Python3.5 here. Does it also still work with python 2.7? Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error (postats.py)
On 2019-07-13, Jürgen Spitzmüller wrote:
> [-- Type: text/plain, Encoding: quoted-printable --]
> make_i18n.inc (in po/) fails for me with this error:
> /usr/bin/python3 ./postats.py "2.4.0dev" ar.po bg.po cs.po de.po el.po
> en.po es.po eu.po fi.po fr.po he.po hu.po ia.po id.po it.po ja.po nb.po
> nl.po nn.po pl.po pt_BR.po pt_PT.po ru.po sk.po sv.po tr.po uk.po
> zh_CN.po zh_TW.po >i18n.inc
> Traceback (most recent call last):
> File "./postats.py", line 144, in
> )?>""" % (sys.argv[1], branch_tag, ",".join([run_msgfmt(po) for po
> in sys.argv[2:] if po not in ommitted])))
> File "./postats.py", line 144, in
> )?>""" % (sys.argv[1], branch_tag, ",".join([run_msgfmt(po) for po
> in sys.argv[2:] if po not in ommitted])))
> File "./postats.py", line 103, in run_msgfmt
> header = read_pofile(pofile)
> File "./postats.py", line 72, in read_pofile
> for line in open(pofile):
> File "/usr/lib64/python3.7/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position
> 872: ordinal not in range(128)
> make: *** [Makefile:820: i18n.inc] Error 1
> Pythonists, any idea?
This file is for Python2.
The patch below made it work with Python3.5 here.
Günter
diff --git a/po/postats.py b/po/postats.py
index 8c5f058642..392c830683 100755
--- a/po/postats.py
+++ b/po/postats.py
@@ -37,6 +37,7 @@ ommitted = ('en.po')
import os
import sys
+from subprocess import Popen, PIPE
# Reset the locale
import locale
@@ -69,7 +70,7 @@ def read_pofile(pofile):
""" Read the header of the pofile and return it as a dictionary"""
header = {}
read_header = False
-for line in open(pofile):
+for line in open(pofile, encoding='utf8'):
line = line[:-1]
if line[:5] == 'msgid':
if read_header:
@@ -112,12 +113,13 @@ def run_msgfmt(pofile):
prop["email"] = prop["email"].replace(".", " ! ")
translator = header['Last-Translator'].split('<')[0].strip()
try:
-prop["translator"] =
translator.decode(charset).encode('ascii','xmlcharrefreplace')
+prop["translator"] = translator.encode('ascii','xmlcharrefreplace')
except LookupError:
prop["translator"] = translator
-p_in, p_out = os.popen4("msgfmt --statistics -o %s %s" % (gmofile, pofile))
-extract_number(p_out.readline(),
+P = Popen("msgfmt --statistics -o %s %s" % (gmofile, pofile),
+ shell=True, stdin=PIPE, stdout=PIPE, close_fds=True)
+extract_number(P.stdout.readline().decode(),
('translated', 'fuzzy', 'untranslated'),
prop)
return """
Re: Python error with master on windows
On Friday, 7 June 2019 15.17.53 WEST Jean-Marc Lasgouttes wrote: > Go ahead. > > JMarc Done. -- José Abílio
Re: Python error with master on windows
Le 07/06/2019 à 14:17, José Abílio Matos a écrit : On Wednesday, 5 June 2019 23.24.29 WEST José Abílio Matos wrote: Thank you. This is only the first step (it controls the build time). Attached is a patch that does the same for run time. I will commit this. Tell me if you have problems with this code. Go ahead. JMarc
Re: Python error with master on windows
On Wednesday, 5 June 2019 23.24.29 WEST José Abílio Matos wrote: > Thank you. This is only the first step (it controls the build time). > > Attached is a patch that does the same for run time. I will commit this. Tell me if you have problems with this code. -- José Abílio
Re: Python error with master on windows
On Thursday, 16 May 2019 17.37.06 WEST Jean-Marc Lasgouttes wrote:
> Done.
>
> JMarc
Thank you. This is only the first step (it controls the build time).
Attached is a patch that does the same for run time.
What do you think?
It search first for python3 and only later for python 2.
PS: There is a final stage that is to change configure.py to replace the call
to python by the python executable used to run configure.py.
--
José Abílio
diff --git a/src/support/os.cpp b/src/support/os.cpp
index 8eea49370a..936e759b8b 100644
--- a/src/support/os.cpp
+++ b/src/support/os.cpp
@@ -11,9 +11,11 @@
#include
+#include "support/convert.h"
#include "support/debug.h"
#include "support/filetools.h"
#include "support/qstring_helpers.h"
+#include "support/regex.h"
#include
@@ -40,18 +42,25 @@ namespace os {
static string const python23(string const & binary, bool verbose = false)
{
+ const string version_info = " -c 'from __future__ import print_function;import sys; print(sys.version_info[:2], end=\"\")'";
+ static regex const python_reg("\\((\\d*), (\\d*)\\)");
if (verbose)
lyxerr << "Examining " << binary << "\n";
// Check whether this is a python 2 or 3 binary.
- cmd_ret const out = runCommand(binary + " -V 2>&1");
- if (out.first < 0 ||
- (!prefixIs(out.second, "Python 2") &&
- !prefixIs(out.second, "Python 3")))
+ cmd_ret const out = runCommand(binary + version_info);
+
+ smatch sm;
+ if (out.first < 0 || !regex_match(out.second, sm, python_reg))
+ return string();
+
+ int major = convert(sm.str(1));
+ int minor = convert(sm.str(2));
+ if((major == 2 and minor < 7) or (major == 3 and minor < 4))
return string();
if (verbose)
- lyxerr << "Found " << out.second << "\n";
+ lyxerr << "Found Python " << out.second << "\n";
return binary;
}
@@ -64,26 +73,25 @@ int timeout_min()
string const python(bool reset)
{
- // Check whether the first python in PATH is the right one.
- static string command = python23("python -tt");
+ // Check whether python3 in PATH is the right one.
+ static string command = python23("python3");
// FIXME THREAD
if (reset) {
- command = python23("python -tt");
+ command = python23("python3");
}
+ // python3 does not exists, let us try python3.x
if (command.empty()) {
// It was not, so check whether we can find it elsewhere in
// PATH, maybe with some suffix appended.
vector const path = getEnvPath("PATH");
- vector::const_iterator it = path.begin();
- vector::const_iterator const end = path.end();
- lyxerr << "Looking for python v2.x or 3.x ...\n";
- for (; it != end; ++it) {
- QString const dir = toqstr(*it);
+ lyxerr << "Looking for python 3.x ...\n";
+ for (auto bin: path) {
+ QString const dir = toqstr(bin);
string const localdir = dir.toLocal8Bit().constData();
QDir qdir(dir);
qdir.setFilter(QDir::Files | QDir::Executable);
- QStringList list = qdir.entryList(QStringList("python*"));
+ QStringList list = qdir.entryList(QStringList("python3*"));
for (int i = 0; i < list.size() && command.empty(); ++i) {
string const binary = addName(localdir,
list.at(i).toLocal8Bit().constData());
@@ -91,16 +99,42 @@ string const python(bool reset)
}
}
- // Default to "python" if no usable binary was found.
- if (command.empty()) {
- lyxerr << "Warning: No python v2.x or 3.x binary found.\n";
- command = "python";
+ }
+ // python 3 not found let us look for python 2
+ if (command.empty())
+ command = python23("python2");
+
+ // python2 does not exists, let us try python2.x
+ if (command.empty()) {
+ // It was not, so check whether we can find it elsewhere in
+ // PATH, maybe with some suffix appended.
+ vector const path = getEnvPath("PATH");
+ lyxerr << "Looking for python 3.x ...\n";
+ for (auto bin: path) {
+ QString const dir = toqstr(bin);
+ string const localdir = dir.toLocal8Bit().constData();
+ QDir qdir(dir);
+ qdir.setFilter(QDir::Files | QDir::Executable);
+ QStringList list = qdir.entryList(QStringList("python2*"));
+ for (int i = 0; i < list.size() && command.empty(); ++i) {
+string const binary = addName(localdir,
+ list.at(i).toLocal8Bit().constData());
+command = python23(binary, true);
+ }
}
- // Add the -tt switch so that mixed tab/whitespace
- // indentation is an error
- command += " -tt";
}
+
+ // Default to "python" if no usable binary was found.
+ // If this happens all hope is lost that there is a sane system
+ if (command.empty()) {
+ lyxerr << "Warning: No python v2.x or 3.x binary found.\n";
+ command = "python";
+ }
+
+ // Add the -tt switch so that mixed tab/whitespace
+ // indentation is an error
+ command += " -tt";
return command;
}
Re: Python error with master on windows
Le 15/05/2019 à 21:41, José Abílio Matos a écrit : On Wednesday, 15 May 2019 16.19.49 WEST Jean-Marc Lasgouttes wrote: OK, so I change the list to [python3 python2], right? Done. JMarc
Re: Python error with master on windows
Jean-Marc Lasgouttes > > The sentence makes sense to me, although not in the intended way. > Couldn't you say the same for any bug? ;) Jürgen > JMarc >
Re: Python error with master on windows
On Wednesday, 15 May 2019 16.19.49 WEST Jean-Marc Lasgouttes wrote: > OK, so I change the list to [python3 python2], right? > > JMarc Yes. -- José Abílio
Re: Python error with master on windows
Le 15/05/2019 à 16:27, José Abílio Matos a écrit : Currently, we have a baroque list which is: [python python2 python3 python3.3 python3.2 python3.1 python3.0 python2.7 dnl python2.6 python2.5 python2.4 python2.3 python2.2 python2.1 python2.0]) We should get rid of those and only test python2 and python3, not even just python. OK, so I change the list to [python3 python2], right? JMarc
Re: Python error with master on windows
On Wednesday, 15 May 2019 13.45.55 WEST Jean-Marc Lasgouttes wrote: > Le 15/05/2019 à 14:29, José Abílio Matos a écrit : > > On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: > >> On my self-build master LyX on windows 10, I get the error below when > >> reconfiguring. Python is 3.6. > >> > >> What is that? > >> > >> JMarc > > > > Based on this I would like to suggest before 2.4 to modify the current > > python detection logic. > > You mean that I was a bit too brave when deciding to install only python3? Nope. :-) As I have told you I had some of the fedora packages (from the equivalent of a ppa) running with python 3 only. Some of the python3 problems found later did not happen for me because I did not execute that code (like some converters and equivalent). > > Instead of using the first version we find, I would suggest to try first > > to > > detect python 3 and only then if it is not detected to search for python > > 2. > > Currently, we have a baroque list which is: > > [python python2 python3 python3.3 python3.2 python3.1 python3.0 > python2.7 dnl > python2.6 python2.5 python2.4 python2.3 python2.2 python2.1 python2.0]) > > Does it really correspond to some relevant ordering? What would you > suggest instead? Move "python2" just before python2.7? And put "poython" > at the end? Not only that but it is plainly wrong. We support python 2 but just for python 2.7. Our code to be compatible with python 3 requires python 2.7 at least. > I guess we can get rid of most of the pythonX.y values, or do they make > sense in any way? We should get rid of those and only test python2 and python3, not even just python. That comes from https://www.python.org/dev/peps/pep-0394/ (Note that this pep is originally from 2011 and so it should be implemented in all the systems that we support) First the convention: python2 will refer to some version of Python 2.x. python3 will refer to some version of Python 3.x. Notice that at the begin it recommends: * Unix-like software distributions (including systems like Mac OS X and Cygwin) should install the python2 command into the default path whenever a version of the Python 2 interpreter is installed, and the same for python3 and the Python 3 interpreter. * When invoked, python2 should run some version of the Python 2 interpreter, and python3 should run some version of the Python 3 interpreter. On my system I have: $ ls -l /usr/bin/python lrwxrwxrwx. 1 root root 7 Apr 30 16:55 /usr/bin/python -> python2 $ ls -l /usr/bin/python2 lrwxrwxrwx. 1 root root 9 Apr 30 16:55 /usr/bin/python2 -> python2.7 $ ls -l /usr/bin/python3 lrwxrwxrwx. 1 root root 9 May 11 01:41 /usr/bin/python3 -> python3.7 So all the systems that we support should have one of python2 or python3. > JMarc > > > The advantage of this approach is that it works for places that only have > > python 2 installed. > > But how is this related to the problem I had? I suspect that Jürgen has python3 installed. If when developing by default python3 was active Jürgen would catch this issue immediately and so no problem for you. > JMarc -- José Abílio
Re: Python error with master on windows
On Wednesday, 15 May 2019 14.01.23 WEST jungok wrote: > > 1) The first step is to change both cmake and autotols detection. > > Cmake already searches for python3 first. That is nice. Thank you. :-) -- José Abílio
Re: Python error with master on windows
Am Mittwoch, 15. Mai 2019, 13:29:10 CEST schrieb José Abílio Matos: > On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: > > On my self-build master LyX on windows 10, I get the error below when > > reconfiguring. Python is 3.6. > > > > What is that? > > > > JMarc > > Based on this I would like to suggest before 2.4 to modify the current > python detection logic. > > Instead of using the first version we find, I would suggest to try first to > detect python 3 and only then if it is not detected to search for python 2. > > The search logic can also be changed from searching for, e.g. on linux, > instead of searching for (/usr/bin/)python we should search for either > (/usr/ bin/)python3 or (/usr/bin/)python2 (they do exist if one of the > versions is installed). > > 1) The first step is to change both cmake and autotols detection. Cmake already searches for python3 first. > 2) There is a second stage where we need to change configure.py (replacing > the calls to python to python2 or python2 depending on the version of > python called to run it). > > I can easily do 2) and I would like help to do 1). > > The advantage of this approach is that it works for places that only have > python 2 installed. > > After lyx 2.4 is released I suggest to remove python 2 support from the code > base. > > What do you think? +1 > Best regards, Kornel
Re: Python error with master on windows
Le 15/05/2019 à 14:29, José Abílio Matos a écrit : On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: On my self-build master LyX on windows 10, I get the error below when reconfiguring. Python is 3.6. What is that? JMarc Based on this I would like to suggest before 2.4 to modify the current python detection logic. You mean that I was a bit too brave when deciding to install only python3? Instead of using the first version we find, I would suggest to try first to detect python 3 and only then if it is not detected to search for python 2. Currently, we have a baroque list which is: [python python2 python3 python3.3 python3.2 python3.1 python3.0 python2.7 dnl python2.6 python2.5 python2.4 python2.3 python2.2 python2.1 python2.0]) Does it really correspond to some relevant ordering? What would you suggest instead? Move "python2" just before python2.7? And put "poython" at the end? I guess we can get rid of most of the pythonX.y values, or do they make sense in any way? JMarc The advantage of this approach is that it works for places that only have python 2 installed. But how is this related to the problem I had? JMarc
Re: Python error with master on windows
On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: > On my self-build master LyX on windows 10, I get the error below when > reconfiguring. Python is 3.6. > > What is that? > > JMarc Based on this I would like to suggest before 2.4 to modify the current python detection logic. Instead of using the first version we find, I would suggest to try first to detect python 3 and only then if it is not detected to search for python 2. The search logic can also be changed from searching for, e.g. on linux, instead of searching for (/usr/bin/)python we should search for either (/usr/ bin/)python3 or (/usr/bin/)python2 (they do exist if one of the versions is installed). 1) The first step is to change both cmake and autotols detection. 2) There is a second stage where we need to change configure.py (replacing the calls to python to python2 or python2 depending on the version of python called to run it). I can easily do 2) and I would like help to do 1). The advantage of this approach is that it works for places that only have python 2 installed. After lyx 2.4 is released I suggest to remove python 2 support from the code base. What do you think? Best regards, -- José Abílio
Re: Python error with master on windows
On Wednesday, 15 May 2019 07.06.54 WEST Jürgen Spitzmüller wrote: > Then point the finder towards me. > > Thanks for fixing > Jürgen I think that you misunderstood me (my fault). :-) I was anthropomorphising the code because then when we say that the code gives some fight it is easier to relate. :-) -- José Abílio
Re: Python error with master on windows
Le 15/05/2019 à 08:11, Jürgen Spitzmüller a écrit : Am Mittwoch, den 15.05.2019, 08:06 +0200 schrieb Jürgen Spitzmüller: Then point the finder towards me. *Sigh* Even my accusatives are buggy these days. The sentence makes sense to me, although not in the intended way. JMarc
Re: Python error with master on windows
Am Mittwoch, den 15.05.2019, 08:06 +0200 schrieb Jürgen Spitzmüller: > Then point the finder towards me. *Sigh* Even my accusatives are buggy these days. Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error with master on windows
Am Dienstag, den 14.05.2019, 22:15 +0100 schrieb José Abílio Matos: > On a second look this is the culprit. One of those variables is a > string while > it should be bytes. I will try to find who (it should be *what* but > it is > funnier if I make this personal ;-) ) Then point the finder towards me. Thanks for fixing Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error with master on windows
On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: > On my self-build master LyX on windows 10, I get the error below when > reconfiguring. Python is 3.6. > > What is that? > > JMarc The culprit was the _local_ variable. The following patch should fix it. diff --git a/lib/configure.py b/lib/configure.py index 9247957948..0b3c2a761c 100644 --- a/lib/configure.py +++ b/lib/configure.py @@ -1587,9 +1587,9 @@ def processModuleFile(file, filename, bool_docbook): cm.write(line + '\n') cm.close() -local = "true" +local = b"true" if (file.startswith(srcdir)): -local = "false" +local = b"false" return (b'"%s" "%s" "%s" "%s" "%s" "%s" "%s" "%s"\n' % (modname, filename, desc, pkgs, req, excl, catgy, local)) -- José Abílio
Re: Python error with master on windows
On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote: > % (modname, filename, desc, pkgs, req, excl, catgy, local)) On a second look this is the culprit. One of those variables is a string while it should be bytes. I will try to find who (it should be *what* but it is funnier if I make this personal ;-) ) is the guilty part. -- José Abílio
Re: Python error with master on windows
On Tuesday, 14 May 2019 19.17.56 WEST Jean-Marc Lasgouttes wrote:
> On my self-build master LyX on windows 10, I get the error below when
> reconfiguring. Python is 3.6.
>
> What is that?
>
> JMarc
>
> +checking list of modules...
> Z:/Jean-Marc/lyx/lib\layouts\algorithm2e.module
> Traceback (most recent call last):
>File "Z:/Jean-Marc/lyx/lib/configure.py", line 1875, in
> checkModulesConfig()
>File "Z:/Jean-Marc/lyx/lib/configure.py", line 1495, in
> checkModulesConfig
> retval = processModuleFile(file, filename.encode('ascii'),
> bool_docbook)
>File "Z:/Jean-Marc/lyx/lib/configure.py", line 1594, in processModuleFile
> % (modname, filename, desc, pkgs, req, excl, catgy, local))
> TypeError: %b requires a bytes-like object, or an object that implements
> __bytes__, not 'str'
> support\Systemcall.cpp (291): Systemcall: 'python -tt
> "Z:/Jean-Marc/lyx/lib/configure.py"
> --binary-dir="Z:/Jean-Marc/build/bin/Debug/"' finished with exit code 1
I looks like the file was opened in binary mode (thus the reference to a
bytes-like object). For python 2 there are no differences, but for python 3
they are different types.
I thought that Georg had change all occurrences of that.
I will look into it.
Regards,
--
José Abílio
Re: Python error
On Thursday, 13 April 2017 15.52.57 WEST Enrico Forestieri wrote: > Thanks for the pointer. I also see there: > > Changed in version 3.6: Unrecognized escape sequences produce a > DeprecationWarning. In some future version of Python they will > be a SyntaxError. > > which is what I would have expected, indeed. FWIW I agree with you. :-) -- José Abílio
Re: Python error
On Thu, Apr 13, 2017 at 11:36:56AM +, Guenter Milde wrote: > > Ther difference in backslash handling is due to the "r" praefix, not > depending on binary or not: > > rb'\n' <-> r'\n' >b'\n' <-> '\n' Yes, I was aware of the meaning of the r prefix. > In Python, the backslash can be used for some replacements like > \n - newline, \t tab, \\ backslash ... > > In non-defined combinations (like \s) the backslash is kept and > "\s" == "\\s". This is what surprises me. I would expect an error in this case. -- Enrico
Re: Python error
On Thu, Apr 13, 2017 at 02:45:07PM +0100, José Abílio Matos wrote: > On Thursday, 13 April 2017 10.46.18 WEST Enrico Forestieri wrote: > > The rule I infer from the above is that in b'' objects the backslash > > is not special unless followed by another backslash, in which case > > is quotes the backslash. This seems nonsense to me. > > FWIW a couple of weeks ago I was bitten by this. :-) > > The explanation can be found here: > https://docs.python.org/3/reference/lexical_analysis.html#index-19 > > "Unlike Standard C, all unrecognized escape sequences are left in the string > unchanged, i.e., the backslash is left in the result. (This behavior is > useful > when debugging: if an escape sequence is mistyped, the resulting output is > more easily recognized as broken.) It is also important to note that the > escape sequences only recognized in string literals fall into the category of > unrecognized escapes for bytes literals." Thanks for the pointer. I also see there: Changed in version 3.6: Unrecognized escape sequences produce a DeprecationWarning. In some future version of Python they will be a SyntaxError. which is what I would have expected, indeed. -- Enrico
Re: Python error
On Thursday, 13 April 2017 10.46.18 WEST Enrico Forestieri wrote: > The rule I infer from the above is that in b'' objects the backslash > is not special unless followed by another backslash, in which case > is quotes the backslash. This seems nonsense to me. FWIW a couple of weeks ago I was bitten by this. :-) The explanation can be found here: https://docs.python.org/3/reference/lexical_analysis.html#index-19 "Unlike Standard C, all unrecognized escape sequences are left in the string unchanged, i.e., the backslash is left in the result. (This behavior is useful when debugging: if an escape sequence is mistyped, the resulting output is more easily recognized as broken.) It is also important to note that the escape sequences only recognized in string literals fall into the category of unrecognized escapes for bytes literals." -- José Abílio
Re: Python error
On 2017-04-13, Enrico Forestieri wrote: > On Thu, Apr 13, 2017 at 09:48:00AM +0100, José Abílio Matos wrote: >> On Wednesday, 12 April 2017 19.40.32 WEST Enrico Forestieri wrote: >> > I have some difficulty understanding how backslashes are treated >> > in bytes-like objects, though. >> Do you have an examples? >> I am not aware of any differences regarding backslashes in strings or bytes. ... > So, b'\s' and r'\s' are equivalent, but b'\\s' and r'\\s' are not. > Instead, b's' is equivalent to r'\\s'. Ther difference in backslash handling is due to the "r" praefix, not depending on binary or not: rb'\n' <-> r'\n' b'\n' <-> '\n' > The rule I infer from the above is that in b'' objects the backslash > is not special unless followed by another backslash, in which case > is quotes the backslash. This seems nonsense to me. In Python, the backslash can be used for some replacements like \n - newline, \t tab, \\ backslash ... In non-defined combinations (like \s) the backslash is kept and "\s" == "\\s". Günter
Re: Python error
On Thu, Apr 13, 2017 at 09:48:00AM +0100, José Abílio Matos wrote: > On Wednesday, 12 April 2017 19.40.32 WEST Enrico Forestieri wrote: > > I have some difficulty understanding how backslashes are treated > > in bytes-like objects, though. > > Do you have an examples? > > I am not aware of any differences regarding backslashes in strings or bytes. See below: >>> b'\s' '\\s' >>> b'\\s' '\\s' >>> r'\s' '\\s' >>> r'\\s' 's' >>> len(b'\s') 2 >>> len(b'\\s') 2 >>> len(r'\s') 2 >>> len(r'\\s') 3 >>> b's' 's' >>> len(b's') 3 So, b'\s' and r'\s' are equivalent, but b'\\s' and r'\\s' are not. Instead, b's' is equivalent to r'\\s'. The rule I infer from the above is that in b'' objects the backslash is not special unless followed by another backslash, in which case is quotes the backslash. This seems nonsense to me. -- Enrico
Re: Python error
On Wednesday, 12 April 2017 19.25.29 WEST Enrico Forestieri wrote: > Maybe I beat him Yes, you did. :-) > Please, try the attached. It works for me with both python 2 and 3. I like this code more than the previous since the previous section that was windows specific could fail if we had a file with a different encoding. That is now treated equally for all platforms, as it should. Thank you. :-) -- José Abílio
Re: Python error
On Wednesday, 12 April 2017 19.40.32 WEST Enrico Forestieri wrote: > I have some difficulty understanding how backslashes are treated > in bytes-like objects, though. Do you have an examples? I am not aware of any differences regarding backslashes in strings or bytes. -- José Abílio
Re: Python error
Am Mittwoch, den 12.04.2017, 20:25 +0200 schrieb Enrico Forestieri: > On Wed, Apr 12, 2017 at 07:28:49PM +0200, Enrico Forestieri wrote: > > On Wed, Apr 12, 2017 at 06:09:00PM +0200, Jürgen Spitzmüller wrote: > > > > > > FWIW the patch fixes my case (although reconfiguring seems > > > significantly slower). > > > > Let's see whether José comes up with something, then. > > Maybe I beat him ;) > > Please, try the attached. It works for me with both python 2 and 3. Works for me as well (Python 2.7.13). Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error
On Wed, Apr 12, 2017 at 05:03:24PM +0100, José Abílio Matos wrote: > On Wednesday, 12 April 2017 16.49.54 WEST Enrico Forestieri wrote: > > And how would you perform regex matches when dealing with bytes-like > > objects in python3? Don't you get a TypeError? > > http://stackoverflow.com/questions/5618988/regular-expression-parsing-a-binary-file > > tldr; defining the regex expressions using bytes. Of course ;) I have some difficulty understanding how backslashes are treated in bytes-like objects, though. -- Enrico
Re: Python error
On Wed, Apr 12, 2017 at 07:28:49PM +0200, Enrico Forestieri wrote:
> On Wed, Apr 12, 2017 at 06:09:00PM +0200, Jürgen Spitzmüller wrote:
> >
> > FWIW the patch fixes my case (although reconfiguring seems
> > significantly slower).
>
> Let's see whether José comes up with something, then.
Maybe I beat him ;)
Please, try the attached. It works for me with both python 2 and 3.
--
Enrico
diff --git a/lib/configure.py b/lib/configure.py
index bdd825f..b16f744 100644
--- a/lib/configure.py
+++ b/lib/configure.py
@@ -9,7 +9,7 @@
# Full author contact details are available in file CREDITS.
from __future__ import print_function
-import glob, logging, os, re, shutil, subprocess, sys, stat, io
+import glob, logging, os, re, shutil, subprocess, sys, stat
# set up logging
logging.basicConfig(level = logging.DEBUG,
@@ -1337,9 +1337,9 @@ def checkLatexConfig(check_config, bool_docbook):
# Construct the list of classes to test for.
# build the list of available layout files and convert it to commands
# for chkconfig.ltx
-declare =
re.compile(r'^\s*#\s*\\Declare(LaTeX|DocBook)Class\s*(\[([^,]*)(,.*)*\])*\s*{(.*)}\s*$')
-category = re.compile(r'^\s*#\s*\\DeclareCategory{(.*)}\s*$')
-empty = re.compile(r'^\s*$')
+declare =
re.compile(b'^\s*#\s*Declare(LaTeX|DocBook)Class\s*(\[([^,]*)(,.*)*\])*\s*{(.*)}\s*$')
+category = re.compile(b'^\s*#\s*DeclareCategory{(.*)}\s*$')
+empty = re.compile(b'^\s*$')
testclasses = list()
for file in (glob.glob( os.path.join('layouts', '*.layout') )
+ glob.glob( os.path.join(srcdir, 'layouts', '*.layout' ) ) ):
@@ -1347,36 +1347,34 @@ def checkLatexConfig(check_config, bool_docbook):
if not os.path.isfile(file):
continue
classname = file.split(os.sep)[-1].split('.')[0]
-decline = ""
-catline = ""
-if os.name == 'nt':
-enco = sys.getfilesystemencoding()
-else:
-enco="utf8"
-for line in io.open(file, encoding=enco).readlines():
-if not empty.match(line) and line[0] != '#':
-if decline == "":
+decline = b""
+catline = b""
+for line in open(file, 'rb').readlines():
+if not empty.match(line) and line[0] != b'#'[0]:
+if decline == b"":
logger.warning("Failed to find valid \Declare line "
"for layout file `%s'.\n\t=> Skipping this file!" %
file)
nodeclaration = True
# A class, but no category declaration. Just break.
break
if declare.search(line) != None:
-decline = "\\TestDocClass{%s}{%s}" % (classname,
line[1:].strip())
+decline = b"\\TestDocClass{%s}{%s}" \
+ % (classname.encode('ascii'), line[1:].strip())
testclasses.append(decline)
elif category.search(line) != None:
-catline = ("\\DeclareCategory{%s}{%s}"
- % (classname, category.search(line).groups()[0]))
+catline = (b"\\DeclareCategory{%s}{%s}"
+ % (classname.encode('ascii'),
+ category.search(line).groups()[0]))
testclasses.append(catline)
-if catline == "" or decline == "":
+if catline == b"" or decline == b"":
continue
break
if nodeclaration:
continue
testclasses.sort()
-cl = io.open('chklayouts.tex', 'w', encoding=enco)
+cl = open('chklayouts.tex', 'wb')
for line in testclasses:
-cl.write(line + '\n')
+cl.write(line)
cl.close()
#
# we have chklayouts.tex, then process it
Re: Python error
On Wed, Apr 12, 2017 at 06:09:00PM +0200, Jürgen Spitzmüller wrote: > > FWIW the patch fixes my case (although reconfiguring seems > significantly slower). Let's see whether José comes up with something, then. -- Enrico
Re: Python error
Am Mittwoch, den 12.04.2017, 16:42 +0200 schrieb Enrico Forestieri: > If we agree on this, with the attached patch no re-encoding is > necessary. > Please, try it. The python chardet module is required. FWIW the patch fixes my case (although reconfiguring seems significantly slower). Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error
On Wednesday, 12 April 2017 16.49.54 WEST Enrico Forestieri wrote: > And how would you perform regex matches when dealing with bytes-like > objects in python3? Don't you get a TypeError? http://stackoverflow.com/questions/5618988/regular-expression-parsing-a-binary-file tldr; defining the regex expressions using bytes. Note that my only purpose is to reproduce more or less what is already happening with python2. -- José Abílio
Re: Python error
On Wed, Apr 12, 2017 at 04:21:33PM +0100, José Abílio Matos wrote: > > The other option would be to open the file in binary mode and then we have to > make sure to use the b"" strings when there are file manipulations. And how would you perform regex matches when dealing with bytes-like objects in python3? Don't you get a TypeError? -- Enrico
Re: Python error
On Wednesday, 12 April 2017 15.42.42 WEST Enrico Forestieri wrote: > On Wed, Apr 12, 2017 at 03:36:10PM +0200, Jürgen Spitzmüller wrote: > > I understand that there is no perfect method. However, if we can catch > > a significant set of cases without too much effort, we should probably > > do that. > > If we agree on this, with the attached patch no re-encoding is necessary. > Please, try it. The python chardet module is required. The other option would be to open the file in binary mode and then we have to make sure to use the b"" strings when there are file manipulations. This is an option that tries to minimize the errors. Something that we had no time to research is, what happens to the files that have different encoding, are they dealt appropriately by our current tools? -- José Abílio
Re: Python error
On Wed, Apr 12, 2017 at 03:36:10PM +0200, Jürgen Spitzmüller wrote:
>
> I understand that there is no perfect method. However, if we can catch
> a significant set of cases without too much effort, we should probably
> do that.
If we agree on this, with the attached patch no re-encoding is necessary.
Please, try it. The python chardet module is required.
--
Enrico
diff --git a/lib/configure.py b/lib/configure.py
index bdd825f..b16c695 100644
--- a/lib/configure.py
+++ b/lib/configure.py
@@ -9,7 +9,7 @@
# Full author contact details are available in file CREDITS.
from __future__ import print_function
-import glob, logging, os, re, shutil, subprocess, sys, stat, io
+import glob, logging, os, re, shutil, subprocess, sys, stat, io, chardet
# set up logging
logging.basicConfig(level = logging.DEBUG,
@@ -1352,7 +1352,7 @@ def checkLatexConfig(check_config, bool_docbook):
if os.name == 'nt':
enco = sys.getfilesystemencoding()
else:
-enco="utf8"
+enco = chardet.detect(open(file, 'rb').read())['encoding']
for line in io.open(file, encoding=enco).readlines():
if not empty.match(line) and line[0] != '#':
if decline == "":
@@ -1374,6 +1374,8 @@ def checkLatexConfig(check_config, bool_docbook):
if nodeclaration:
continue
testclasses.sort()
+if os.name != 'nt':
+enco = "utf-8"
cl = io.open('chklayouts.tex', 'w', encoding=enco)
for line in testclasses:
cl.write(line + '\n')
Re: Python error
On Wed, Apr 12, 2017 at 03:36:10PM +0200, Jürgen Spitzmüller wrote: > > Could you tell me again why it is now necessary to open the files with > a specific encoding, while we did not do that until now? (I admit that > I skipped most of those "Python 3" threads) Because of this commit: http://www.lyx.org/trac/changeset/50e21b71/lyxgit meant to fix python3 support on native Windows. -- Enrico
Re: Python error
Am Mittwoch, den 12.04.2017, 15:22 +0200 schrieb Enrico Forestieri: > I find that the chardet library is the least reliable. I get better > results with python-magic. However, all methods are unreliable to > some extent. Try the attached patch to see in how many encodings > you can read a file. Try it also on binary files ;) I understand that there is no perfect method. However, if we can catch a significant set of cases without too much effort, we should probably do that. Could you tell me again why it is now necessary to open the files with a specific encoding, while we did not do that until now? (I admit that I skipped most of those "Python 3" threads) Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error
On Wed, Apr 12, 2017 at 03:22:58PM +0200, Enrico Forestieri wrote: > some extent. Try the attached patch to see in how many encodings ^ Sorry, I meant script, of course. -- Enrico
Re: Python error
On Wed, Apr 12, 2017 at 02:28:48PM +0200, Jürgen Spitzmüller wrote:
>
> There seem to be some ways to detect the file encoding in python:
> http://stackoverflow.com/questions/436220/determine-the-encoding-of-tex
> t-in-python
I find that the chardet library is the least reliable. I get better
results with python-magic. However, all methods are unreliable to
some extent. Try the attached patch to see in how many encodings
you can read a file. Try it also on binary files ;)
--
Enrico
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import print_function
import os,sys,io
from getopt import getopt
def error(message):
sys.stderr.write(message + '\n')
sys.exit(1)
def usage(prog_name):
return "Usage: %s [-v] " % os.path.basename(prog_name)
encodings = ["ascii", "utf-8", "utf-16", "utf-32", "iso-8859-1", "iso-8859-2",
"iso-8859-3", "iso-8859-4", "iso-8859-5", "iso-8859-6", "iso-8859-7",
"iso-8859-8", "iso-8859-9", "iso-8859-10", "iso-8859-13", "iso-8859-14",
"iso-8859-15", "iso-8859-16", "big5", "big5hkscs", "cp037", "cp424", "cp437",
"cp500", "cp720", "cp737", "cp775", "cp850", "cp852", "cp855", "cp856",
"cp857", "cp858", "cp860", "cp861", "cp862", "cp863", "cp864", "cp865",
"cp866", "cp869", "cp874", "cp875", "cp932", "cp949", "cp950", "cp1006",
"cp1026", "cp1140", "cp1250", "cp1251", "cp1252", "cp1253", "cp1254",
"cp1255", "cp1256", "cp1257", "cp1258", "euc_jp", "euc_jis_2004",
"euc_jisx0213", "euc_kr", "gb2312", "gbk", "gb18030", "hz", "iso2022_jp",
"iso2022_jp_1", "iso2022_jp_2", "iso2022_jp_2004", "iso2022_jp_3",
"iso2022_jp_ext", "iso2022_kr", "johab", "koi8_r", "koi8_u", "mac_cyrillic",
"mac_greek", "mac_iceland", "mac_latin2", "mac_roman", "mac_turkish",
"ptcp154", "shift_jis", "shift_jis_2004", "shift_jisx0213", "utf_32_be",
"utf_32_le", "utf_16_be", "utf_16_le", "utf_7", "utf_8_sig"]
verbose = False
try:
(options, args) = getopt(sys.argv[1:], "hv")
except:
error(usage(sys.argv[0]))
if len(args) == 0:
error(usage(sys.argv[0]))
for (opt, param) in options:
if opt == "-h":
print(usage(sys.argv[0]))
if opt == "-v":
verbose = True
for file in args:
if not os.path.isdir(file):
first = True
length = 0
for e in encodings:
try:
fh = io.open(file, 'r', encoding=e)
fh.readlines()
fh.close()
except:
if verbose:
print('%s: got unicode error with %s' % (file, e))
else:
if verbose:
print('%s: success with encoding: %s' % (file, e))
else:
if first:
print('%s: %s' % (file, e), end='')
length += len(file)+len(e)+2
first = False
else:
if length+len(e)+2 > 79:
print(',\n\t%s' % e, end='')
length = len(e)+8
else:
print(', %s' % e, end='')
length += len(e)+2
sys.stdout.flush()
if not first:
print('')
else:
print('%s: directory' % file)
Re: Python error
Am Mittwoch, den 12.04.2017, 14:13 +0200 schrieb Enrico Forestieri: > I highlighted this issue here: > http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg199359.html > I don't know what will happen. I see. So the patch you have posted (which has been applied by JMarc) helps in JMarc's own case, but not in mine. There seem to be some ways to detect the file encoding in python: http://stackoverflow.com/questions/436220/determine-the-encoding-of-tex t-in-python Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error
On Wed, Apr 12, 2017 at 01:38:57PM +0200, Jürgen Spitzmüller wrote: > Am Mittwoch, den 12.04.2017, 13:05 +0200 schrieb Enrico Forestieri: > > > I think that this deserves a comment > > in release notes. > > At least. If there's a way to do the recoding via layout2layout, I > would very much prefer that. This will irritate affected users. I highlighted this issue here: http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg199359.html I don't know what will happen. -- Enrico
Re: Python error
Am Mittwoch, den 12.04.2017, 13:05 +0200 schrieb Enrico Forestieri: > Check that all your layout files are utf8 encoded: > > file ~/.lyx/layout/* Indeed, some weren't. > If not, use iconv to convert them. Did that (using recode). That helped. > I think that this deserves a comment > in release notes. At least. If there's a way to do the recoding via layout2layout, I would very much prefer that. This will irritate affected users. Thanks, Jürgen signature.asc Description: This is a digitally signed message part
Re: Python error
On Wed, Apr 12, 2017 at 12:24:20PM +0200, Jürgen Spitzmüller wrote: > I am currently getting the following error while doing reconfigure: > > checking LaTeX configuration... auto > Traceback (most recent call last): > File "/home/juergen/lyx/lyx-devel2/lib/configure.py", line 1825, in > > ret = checkLatexConfig(lyx_check_config and LATEX != '', > bool_docbook) > File "/home/juergen/lyx/lyx-devel2/lib/configure.py", line 1356, in > checkLatexConfig > for line in io.open(file, encoding=enco).readlines(): > File "/usr/lib64/python2.7/codecs.py", line 314, in decode > (result, consumed) = self._buffer_decode(data, self.errors, final) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position > 294: invalid continuation byte > Systemcall.cpp (295): Systemcall: 'python -tt "/home/juergen/lyx/lyx- > devel2/lib/configure.py" --with-version-suffix=-dev --binary- > dir="/home/juergen/lyx/lyx-devel2/src/"' finished with exit code 1 Check that all your layout files are utf8 encoded: file ~/.lyx/layout/* If not, use iconv to convert them. I think that this deserves a comment in release notes. -- Enrico
