Re: [OE-core] [PATCH] buildhistory.bbclass: Specify lang in decoding strings

2014-01-23 Thread Paul Eggleton 
On Saturday 28 December 2013 22:52:18 Paul Eggleton wrote:
 On Friday 06 December 2013 16:11:54 Khem Raj wrote:
  On systems where default locale is utf-8 we get errors like
  
  File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
  0034: if pkginfo.rconflicts:
  0035: f.write(RCONFLICTS = %s\n % pkginfo.rconflicts)
  0036: f.write(PKGSIZE = %d\n % pkginfo.size)
  0037: f.write(FILES = %s\n % pkginfo.files)
  *** 0038: f.write(FILELIST = %s\n % pkginfo.filelist)
  0039:
  0040: for filevar in pkginfo.filevars:
  0041: filevarpath = os.path.join(pkgpath, latest.%s % filevar)
  0042: val = pkginfo.filevars[filevar]
  Exception: UnicodeEncodeError: 'ascii' codec can't encode character
  u'\xed' in position 337: ordinal not in range(128)
  
  This patch specifies decode to use utf-8 so ascii and utf-8 based
  locales both work
  
  Signed-off-by: Khem Raj raj.k...@gmail.com
  ---
  
   meta/classes/buildhistory.bbclass |2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
  
  diff --git a/meta/classes/buildhistory.bbclass
  b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644
  --- a/meta/classes/buildhistory.bbclass
  +++ b/meta/classes/buildhistory.bbclass
  @@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
  
   key = item[0]
   
   if key.endswith('_' + pkg):
   key = key[:-len(pkg)-1]
  
  -pkgdata[key] = item[1].decode('string_escape')
  +pkgdata[key] = item[1].decode('utf-8', 'string_escape')
  
   pkge = pkgdata.get('PKGE', '0')
   pkgv = pkgdata['PKGV']
 
 Khem, did you test that this actually works? Here it does not - I get
 strings with \n \t in them; reverting this change makes it interpret these
 as it should.

Unless I'm misunderstanding, I think the second parameter here is wrong, since 
according to the Python docs it's supposed to specify what to do when an error 
occurs, not specify another encoding to decode. Should it be 
.decode('utf-8').decode('string_escape') instead?

The problem that I have is I can't reproduce the failure that you observed, so 
I need you to help me to fix this properly.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] buildhistory.bbclass: Specify lang in decoding strings

2014-01-23 Thread Khem Raj 
On Thu, Jan 23, 2014 at 10:26 AM, Paul Eggleton
paul.eggle...@linux.intel.com wrote:
 On Saturday 28 December 2013 22:52:18 Paul Eggleton wrote:
 On Friday 06 December 2013 16:11:54 Khem Raj wrote:
  On systems where default locale is utf-8 we get errors like
 
  File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
  0034: if pkginfo.rconflicts:
  0035: f.write(RCONFLICTS = %s\n % pkginfo.rconflicts)
  0036: f.write(PKGSIZE = %d\n % pkginfo.size)
  0037: f.write(FILES = %s\n % pkginfo.files)
  *** 0038: f.write(FILELIST = %s\n % pkginfo.filelist)
  0039:
  0040: for filevar in pkginfo.filevars:
  0041: filevarpath = os.path.join(pkgpath, latest.%s % filevar)
  0042: val = pkginfo.filevars[filevar]
  Exception: UnicodeEncodeError: 'ascii' codec can't encode character
  u'\xed' in position 337: ordinal not in range(128)
 
  This patch specifies decode to use utf-8 so ascii and utf-8 based
  locales both work
 
  Signed-off-by: Khem Raj raj.k...@gmail.com
  ---
 
   meta/classes/buildhistory.bbclass |2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
 
  diff --git a/meta/classes/buildhistory.bbclass
  b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644
  --- a/meta/classes/buildhistory.bbclass
  +++ b/meta/classes/buildhistory.bbclass
  @@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
 
   key = item[0]
 
   if key.endswith('_' + pkg):
   key = key[:-len(pkg)-1]
 
  -pkgdata[key] = item[1].decode('string_escape')
  +pkgdata[key] = item[1].decode('utf-8', 'string_escape')
 
   pkge = pkgdata.get('PKGE', '0')
   pkgv = pkgdata['PKGV']

 Khem, did you test that this actually works? Here it does not - I get
 strings with \n \t in them; reverting this change makes it interpret these
 as it should.

 Unless I'm misunderstanding, I think the second parameter here is wrong, since
 according to the Python docs it's supposed to specify what to do when an error
 occurs, not specify another encoding to decode. Should it be
 .decode('utf-8').decode('string_escape') instead?


Yes that seems more likely the right thing to do.


 The problem that I have is I can't reproduce the failure that you observed, so
 I need you to help me to fix this properly.

I see the failure on a newly installed debian wheezy with no locale's installed.



 Thanks,
 Paul

 --

 Paul Eggleton
 Intel Open Source Technology Centre
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] buildhistory.bbclass: Specify lang in decoding strings

2013-12-28 Thread Paul Eggleton 
On Friday 06 December 2013 16:11:54 Khem Raj wrote:
 On systems where default locale is utf-8 we get errors like
 
 File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
 0034: if pkginfo.rconflicts:
 0035: f.write(RCONFLICTS = %s\n % pkginfo.rconflicts)
 0036: f.write(PKGSIZE = %d\n % pkginfo.size)
 0037: f.write(FILES = %s\n % pkginfo.files)
 *** 0038: f.write(FILELIST = %s\n % pkginfo.filelist)
 0039:
 0040: for filevar in pkginfo.filevars:
 0041: filevarpath = os.path.join(pkgpath, latest.%s % filevar)
 0042: val = pkginfo.filevars[filevar]
 Exception: UnicodeEncodeError: 'ascii' codec can't encode character
 u'\xed' in position 337: ordinal not in range(128)
 
 This patch specifies decode to use utf-8 so ascii and utf-8 based
 locales both work
 
 Signed-off-by: Khem Raj raj.k...@gmail.com
 ---
  meta/classes/buildhistory.bbclass |2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
 
 diff --git a/meta/classes/buildhistory.bbclass
 b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644
 --- a/meta/classes/buildhistory.bbclass
 +++ b/meta/classes/buildhistory.bbclass
 @@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
  key = item[0]
  if key.endswith('_' + pkg):
  key = key[:-len(pkg)-1]
 -pkgdata[key] = item[1].decode('string_escape')
 +pkgdata[key] = item[1].decode('utf-8', 'string_escape')
 
  pkge = pkgdata.get('PKGE', '0')
  pkgv = pkgdata['PKGV']

Khem, did you test that this actually works? Here it does not - I get strings 
with \n \t in them; reverting this change makes it interpret these as it 
should.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] buildhistory.bbclass: Specify lang in decoding strings

2013-12-06 Thread Khem Raj 
On systems where default locale is utf-8 we get errors like

File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
0034: if pkginfo.rconflicts:
0035: f.write(RCONFLICTS = %s\n % pkginfo.rconflicts)
0036: f.write(PKGSIZE = %d\n % pkginfo.size)
0037: f.write(FILES = %s\n % pkginfo.files)
*** 0038: f.write(FILELIST = %s\n % pkginfo.filelist)
0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, latest.%s % filevar)
0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character
u'\xed' in position 337: ordinal not in range(128)

This patch specifies decode to use utf-8 so ascii and utf-8 based
locales both work

Signed-off-by: Khem Raj raj.k...@gmail.com
---
 meta/classes/buildhistory.bbclass |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass 
b/meta/classes/buildhistory.bbclass
index 1e6d968..4ff39a0 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
 key = item[0]
 if key.endswith('_' + pkg):
 key = key[:-len(pkg)-1]
-pkgdata[key] = item[1].decode('string_escape')
+pkgdata[key] = item[1].decode('utf-8', 'string_escape')
 
 pkge = pkgdata.get('PKGE', '0')
 pkgv = pkgdata['PKGV']
-- 
1.7.10.4

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core