Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
Hi Miklos, 2011/9/10 Miklos Vajna vmik...@frugalware.org: I'm attaching newer versions of both patches: - the translations one is fixed wrt python (see 82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same problem) - the core one should be fixed wrt the above two issues Now it works perfectly. Your script does its job well. Unfortunately I still have not found the bug in fast_merge.pl which cause misterious omission even together with your script. I would like to push everything when the whole toolchain works reliably. Cheers, Andras ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Wed, Sep 14, 2011 at 08:15:16PM +0200, Andras Timar tima...@gmail.com wrote: Hi Miklos, 2011/9/10 Miklos Vajna vmik...@frugalware.org: I'm attaching newer versions of both patches: - the translations one is fixed wrt python (see 82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same problem) - the core one should be fixed wrt the above two issues Now it works perfectly. Your script does its job well. Unfortunately I still have not found the bug in fast_merge.pl which cause misterious omission even together with your script. I would like to push everything when the whole toolchain works reliably. Actually that's not an omission: The accessibility strings end up at translations/unxlngx6.pro/misc/sdf/avmedia/source/framework/localize.sdf here. pgpDmrzuD8mQl.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Thu, Sep 15, 2011 at 12:44:01AM +0200, Miklos Vajna vmik...@frugalware.org
wrote:
Actually that's not an omission:
The accessibility strings end up at
translations/unxlngx6.pro/misc/sdf/avmedia/source/framework/localize.sdf
here.
The attached patch fixes the issue here.
From 3381db95cc026f404094daabbed4a8d84685a142 Mon Sep 17 00:00:00 2001
From: Miklos Vajna vmik...@frugalware.org
Date: Thu, 15 Sep 2011 01:13:05 +0200
Subject: [PATCH] fast_merge: fix mis-merge of first module's strings
The problem was that in write_lines() when $first_run was true, the
add_to_buffer() call already set the current module's name to the second
module.
A fix for this is to use let make_paths() take a parameter, and in case
of the first run, pass the really first module name, not the current
one.
---
l10ntools/scripts/fast_merge.pl | 10 ++
1 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/l10ntools/scripts/fast_merge.pl b/l10ntools/scripts/fast_merge.pl
index 5dc63cf..73b824e 100644
--- a/l10ntools/scripts/fast_merge.pl
+++ b/l10ntools/scripts/fast_merge.pl
@@ -91,7 +91,7 @@ while( hasLines() )
}
if( $#current+1 ne 0 )
{
-( $path , $localize_file ) = make_paths();
+( $path , $localize_file ) = make_paths($current[ 0 ]-module);
add_to_buffer();
write_buffer( $path , $localize_file );
}
@@ -240,7 +240,8 @@ sub hasLines
sub make_paths
{
-my $localizeFile = $merge_dir.\\.$current[ 0 ]-module.\\.$current[ 0
]-file;
+my $module = shift ;
+my $localizeFile = $merge_dir.\\.$module.\\.$current[ 0 ]-file;
my $path = getDir( $localizeFile );
$path =~ s/\\/\//g;
@@ -251,8 +252,9 @@ sub make_paths
sub write_lines
{
if( $first_run ){
+my $module = $current[ 0 ]-module;
add_to_buffer();
-my( $path , $localize_file ) = make_paths();
+my( $path , $localize_file ) = make_paths($module);
$last_path = $path;
$last_localize_file = $localize_file;
mkpath $path;
@@ -262,7 +264,7 @@ sub write_lines
else
{
return , if ( $#current+1 eq 0 );
-my( $path , $localize_file ) = make_paths();
+my( $path , $localize_file ) = make_paths($current[ 0 ]-module);
if( $path eq $last_path )
{
add_to_buffer();
--
1.7.6
pgpUyiOQ7sznm.pgp
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
Hi Miklos, 2011/9/8 Miklos Vajna vmik...@frugalware.org: I'm attaching an updated patch that fixes po2lo on Windows, where it did not translate anything due to path translating (/ vs \) issues. First of all, many thanks for your work. Full build time in translations module with all languages took 82 minutes, with your patch it took only 19 minutes on a Windows box. There are 2 minor issues. 1. When the English string contains \n and translation does not contain \n, the script outputs the English string instead of the translation. See for example this line: basctl source\basicide\basidesh.src0 string RID_STR_SOURCETOBIG 15504 sd The source text is too large and can be neither compiled nor saved.\nDelete some of the comments or transfer some methods into another module. 2. Your script produces sdf lines even when the corresponding po file does not exist at all. For example many languages do not have help translations at all, but your script copies English heIp lines to localized sdf files in this case. Best regards, Andras ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Fri, Sep 09, 2011 at 10:40:54AM +0200, Andras Timar tima...@gmail.com
wrote:
There are 2 minor issues.
1. When the English string contains \n and translation does not
contain \n, the script outputs the English string instead of the
translation. See for example this line:
basctlsource\basicide\basidesh.src0 string
RID_STR_SOURCETOBIG 15504 sd The
source text is too large and can be neither compiled nor
saved.\nDelete some of the comments or transfer some methods into
another module.
2. Your script produces sdf lines even when the corresponding po file
does not exist at all. For example many languages do not have help
translations at all, but your script copies English heIp lines to
localized sdf files in this case.
I'm attaching newer versions of both patches:
- the translations one is fixed wrt python (see
82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same
problem)
- the core one should be fixed wrt the above two issues
[ The newline issue was a bug in the po parser, handling multiline
msgid/msgstr entires, the second problem is simply fixed by checking
if the relevant po file exists or not. ]
This time I tested it with not only hu, but af and sd locales as
well.
Miklos
From 60173c8ed5180067773b7df503300f16752235c2 Mon Sep 17 00:00:00 2001
From: Miklos Vajna vmik...@frugalware.org
Date: Wed, 7 Sep 2011 23:39:15 +0200
Subject: [PATCH] Add po2lo tool
---
solenv/bin/po2lo | 205 ++
1 files changed, 205 insertions(+), 0 deletions(-)
create mode 100755 solenv/bin/po2lo
diff --git a/solenv/bin/po2lo b/solenv/bin/po2lo
new file mode 100755
index 000..0f81ebc
--- /dev/null
+++ b/solenv/bin/po2lo
@@ -0,0 +1,205 @@
+#!/usr/bin/env python
+# Version: MPL 1.1 / GPLv3+ / LGPLv3+
+#
+# The contents of this file are subject to the Mozilla Public License Version
+# 1.1 (the License); you may not use this file except in compliance with
+# the License or as specified alternatively below. You may obtain a copy of
+# the License at http://www.mozilla.org/MPL/
+#
+# Software distributed under the License is distributed on an AS IS basis,
+# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
+# for the specific language governing rights and limitations under the
+# License.
+#
+# The Initial Developer of the Original Code is
+# Miklos Vajna vmik...@frugalware.org
+# Portions created by the Initial Developer are Copyright (C) 2011 the
+# Initial Developer. All Rights Reserved.
+#
+# Major Contributor(s):
+#
+# For minor contributions see the git repository.
+#
+# Alternatively, the contents of this file may be used under the terms of
+# either the GNU General Public License Version 3 or later (the GPLv3+), or
+# the GNU Lesser General Public License Version 3 or later (the LGPLv3+),
+# in which case the provisions of the GPLv3+ or the LGPLv3+ are applicable
+# instead of those above.
+
+import getopt, sys, os, re
+
+class Options:
+Options of this script.
+
+def __init__(self):
+self.input = None
+self.output = None
+self.language = None
+self.template = None
+
+class Entry:
+Represents a single line in an SDF file.
+
+def __init__(self, items):
+self.items = items # list of 15 fields
+path = self.items[1].split('\\')
+self.po = %s/%s/%s.po % (options.input.replace('\\', '/'),
self.items[0], /.join(path[:-1]))
+prefix =
+if len(self.items[5]):
+prefix += %s. % self.items[5]
+if len(self.items[3]):
+prefix += %s. % self.items[3]
+self.keys = []
+# 10..13 are translation types
+for idx in range(10, 14):
+if len(self.items[idx]):
+t = {10:'text', 12:'quickhelptext', 13:'title'}[idx]
+self.keys.append((idx, self.sdf2po(%s#%s.%s%s % (path[-1],
self.items[4], prefix, t
+
+def translate(self, translations):
+Translates text in the entry based on translations.
+
+self.items[9] = options.language
+for idx, key in self.keys:
+try:
+self.items[idx] = translations.data[(self.po, key)]
+
+self.items[14] = 2002-02-02 02:02:02
+except KeyError:
+pass
+self.items[14] = self.items[14].strip()
+
+def sdf2po(self, s):
+Escapes special chars in po key names.
+
+return s.translate(normalizetable)
+
+class Template:
+Represents a reference template in SDF format.
+
+def __init__(self, path):
+sock = open(path)
+self.lines = []
+for line in sock:
+entry = Entry(line.split('\t'))
+if os.path.exists(entry.po):
+self.lines.append(entry)
+
+def translate(self, translations):
+Translates entires in the template based on translations.
+
+sock =
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
I'm attaching an updated patch that fixes po2lo on Windows, where it did
not translate anything due to path translating (/ vs \) issues.
From a1e8bad75279a05bf6a061c53351898eea72433d Mon Sep 17 00:00:00 2001
From: Miklos Vajna vmik...@frugalware.org
Date: Wed, 7 Sep 2011 23:39:15 +0200
Subject: [PATCH] Add po2lo tool
---
solenv/bin/po2lo | 202 ++
1 files changed, 202 insertions(+), 0 deletions(-)
create mode 100755 solenv/bin/po2lo
diff --git a/solenv/bin/po2lo b/solenv/bin/po2lo
new file mode 100755
index 000..08b5519
--- /dev/null
+++ b/solenv/bin/po2lo
@@ -0,0 +1,202 @@
+#!/usr/bin/env python
+# Version: MPL 1.1 / GPLv3+ / LGPLv3+
+#
+# The contents of this file are subject to the Mozilla Public License Version
+# 1.1 (the License); you may not use this file except in compliance with
+# the License or as specified alternatively below. You may obtain a copy of
+# the License at http://www.mozilla.org/MPL/
+#
+# Software distributed under the License is distributed on an AS IS basis,
+# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
+# for the specific language governing rights and limitations under the
+# License.
+#
+# The Initial Developer of the Original Code is
+# Miklos Vajna vmik...@frugalware.org
+# Portions created by the Initial Developer are Copyright (C) 2011 the
+# Initial Developer. All Rights Reserved.
+#
+# Major Contributor(s):
+#
+# For minor contributions see the git repository.
+#
+# Alternatively, the contents of this file may be used under the terms of
+# either the GNU General Public License Version 3 or later (the GPLv3+), or
+# the GNU Lesser General Public License Version 3 or later (the LGPLv3+),
+# in which case the provisions of the GPLv3+ or the LGPLv3+ are applicable
+# instead of those above.
+
+import getopt, sys, os, re
+
+class Options:
+Options of this script.
+
+def __init__(self):
+self.input = None
+self.output = None
+self.language = None
+self.template = None
+
+class Entry:
+Represents a single line in an SDF file.
+
+def __init__(self, items):
+self.items = items # list of 15 fields
+path = self.items[1].split('\\')
+self.po = %s/%s/%s.po % (options.input.replace('\\', '/'),
self.items[0], /.join(path[:-1]))
+prefix =
+if len(self.items[5]):
+prefix += %s. % self.items[5]
+if len(self.items[3]):
+prefix += %s. % self.items[3]
+self.keys = []
+# 10..13 are translation types
+for idx in range(10, 14):
+if len(self.items[idx]):
+t = {10:'text', 12:'quickhelptext', 13:'title'}[idx]
+self.keys.append((idx, self.sdf2po(%s#%s.%s%s % (path[-1],
self.items[4], prefix, t
+
+def translate(self, translations):
+Translates text in the entry based on translations.
+
+self.items[9] = options.language
+for idx, key in self.keys:
+try:
+self.items[idx] = translations.data[(self.po, key)]
+
+self.items[14] = 2002-02-02 02:02:02
+except KeyError:
+pass
+self.items[14] = self.items[14].strip()
+
+def sdf2po(self, s):
+Escapes special chars in po key names.
+
+return s.translate(normalizetable)
+
+class Template:
+Represents a reference template in SDF format.
+
+def __init__(self, path):
+sock = open(path)
+self.lines = []
+for line in sock:
+self.lines.append(Entry(line.split('\t')))
+
+def translate(self, translations):
+Translates entires in the template based on translations.
+
+sock = open(options.output, w)
+for line in self.lines:
+line.translate(translations)
+sock.write(\t.join(line.items)+\r\n)
+sock.close()
+
+class Translations:
+Represents a set of .po files, containing translations.
+
+def __init__(self):
+key = None
+self.data = {}
+for root, dirs, files in os.walk(options.input):
+for file in files:
+path = %s/%s % (root, file)
+sock = open(path)
+buf = []
+multiline = False
+fuzzy = False
+for line in sock:
+if line.startswith(#: ):
+key = line.strip()[3:]
+elif line.startswith(#, fuzzy):
+fuzzy = True
+elif line.startswith(msgstr ):
+trans = line.strip()[8:-1]
+if len(trans):
+if fuzzy:
+fuzzy = False
+else:
+self.setdata(path, key, trans)
+else:
+buf = []
+
