Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-04 Thread fromani 
Francesco Romani has submitted this change and it was merged.

Change subject: startup: Change system default encoding to utf8
..


startup: Change system default encoding to utf8

In Python 2, the system default encoding is 'ascii'. This causes mixing
of unicode and non-ascii strings (e.g. utf8 encoded) to fail with
UnicodeDecodeError or UnicodeEncodeError. The trigger for this failures
is starting using the built-in json library, that returns all values as
unicode strings, even if the value is ascii.

For example:

>>> u'ascii' + '\xd7\x90'
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0:
ordinal not in range(128)

Python tries to decode the second value implicitly, and fails since this
is a utf8 encoded string.

To avoid such issues, the entire application must be changed to use only
strings or only unicode internally, and never mix these types.  In the
distant future, when we run on Python 3, this will be true. For now, the
only way to prevent these issues systematically is to change the system
default encoding to 'utf8'.

Changing the default encoding is done with sys.setdefaultencoding(), but
this function exists only during startup, and can be called only from
sitecustomize module.

We keep now sitecustomize.py module, and change PYTHONPATH so it will be
loaded on startup.

This change is not needed in Python 3 since the default encoding is
already utf8, and Python does not do any implicit decoding or encoding
when mixing 'str' and 'bytes' types.

Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Bug-Url: https://bugzilla.redhat.com/1281940
Signed-off-by: Nir Soffer 
Reviewed-on: https://gerrit.ovirt.org/48661
Continuous-Integration: Jenkins CI
Reviewed-by: Milan Zamazal 
Reviewed-by: Francesco Romani 
Reviewed-by: Piotr Kliczewski 
Reviewed-on: https://gerrit.ovirt.org/51273
---
M debian/vdsm.install
M init/daemonAdapter
M tests/Makefile.am
A tests/unicode_test.py
M vdsm.spec.in
M vdsm/Makefile.am
A vdsm/sitecustomize.py
7 files changed, 84 insertions(+), 0 deletions(-)

Approvals:
  Nir Soffer: Verified
  Jenkins CI: Passed CI tests
  Francesco Romani: Looks good to me, approved



-- 
To view, visit https://gerrit.ovirt.org/51273
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Nir Soffer 
Gerrit-Reviewer: Francesco Romani 
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer 
Gerrit-Reviewer: Piotr Kliczewski 
Gerrit-Reviewer: Yaniv Bronhaim 
Gerrit-Reviewer: gerrit-hooks 
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches

Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-04 Thread automation 
gerrit-hooks has posted comments on this change.

Change subject: startup: Change system default encoding to utf8
..


Patch Set 2:

* #1281940::Update tracker: OK
* Set MODIFIED::bug 1281940#1281940OK

-- 
To view, visit https://gerrit.ovirt.org/51273
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Nir Soffer 
Gerrit-Reviewer: Francesco Romani 
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer 
Gerrit-Reviewer: Piotr Kliczewski 
Gerrit-Reviewer: Yaniv Bronhaim 
Gerrit-Reviewer: gerrit-hooks 
Gerrit-HasComments: No
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches

Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-04 Thread fromani 
Francesco Romani has posted comments on this change.

Change subject: startup: Change system default encoding to utf8
..


Patch Set 1: Code-Review+2

-- 
To view, visit https://gerrit.ovirt.org/51273
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Nir Soffer 
Gerrit-Reviewer: Francesco Romani 
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal 
Gerrit-Reviewer: Nir Soffer 
Gerrit-Reviewer: Oved Ourfali 
Gerrit-Reviewer: Piotr Kliczewski 
Gerrit-Reviewer: Yaniv Bronhaim 
Gerrit-Reviewer: gerrit-hooks 
Gerrit-HasComments: No
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches

Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-03 Thread nsoffer 
Nir Soffer has posted comments on this change.

Change subject: startup: Change system default encoding to utf8
..


Patch Set 1: Verified+1

Verified that we can start now vm with non-ascii name without failing when 
logging the vm xml:

 Thread-107::INFO::2016-01-04 00:25:31,741::vm::1921::virt.vm::(_run) 
vmId=`04a07255-5429-4325-9c22-40aca4c45bbc`::
 http://ovirt.org/vm/tune/1.0;>
מכונה
04a07255-5429-4325-9c22-40aca4c45bbc

(The vm will not start because libvirt does not support such names currently)

Tested vm with ascii name:
- create vm
- start/stop
- migrate
- remove

-- 
To view, visit https://gerrit.ovirt.org/51273
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Nir Soffer 
Gerrit-Reviewer: Francesco Romani 
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal 
Gerrit-Reviewer: Nir Soffer 
Gerrit-Reviewer: Piotr Kliczewski 
Gerrit-Reviewer: gerrit-hooks 
Gerrit-HasComments: No
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches

Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-03 Thread nsoffer 
Hello Piotr Kliczewski, Francesco Romani, Milan Zamazal,

I'd like you to do a code review.  Please visit

https://gerrit.ovirt.org/51273

to review the following change.

Change subject: startup: Change system default encoding to utf8
..

startup: Change system default encoding to utf8

In Python 2, the system default encoding is 'ascii'. This causes mixing
of unicode and non-ascii strings (e.g. utf8 encoded) to fail with
UnicodeDecodeError or UnicodeEncodeError. The trigger for this failures
is starting using the built-in json library, that returns all values as
unicode strings, even if the value is ascii.

For example:

>>> u'ascii' + '\xd7\x90'
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0:
ordinal not in range(128)

Python tries to decode the second value implicitly, and fails since this
is a utf8 encoded string.

To avoid such issues, the entire application must be changed to use only
strings or only unicode internally, and never mix these types.  In the
distant future, when we run on Python 3, this will be true. For now, the
only way to prevent these issues systematically is to change the system
default encoding to 'utf8'.

Changing the default encoding is done with sys.setdefaultencoding(), but
this function exists only during startup, and can be called only from
sitecustomize module.

We keep now sitecustomize.py module, and change PYTHONPATH so it will be
loaded on startup.

This change is not needed in Python 3 since the default encoding is
already utf8, and Python does not do any implicit decoding or encoding
when mixing 'str' and 'bytes' types.

Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Bug-Url: https://bugzilla.redhat.com/1281940
Signed-off-by: Nir Soffer 
Reviewed-on: https://gerrit.ovirt.org/48661
Continuous-Integration: Jenkins CI
Reviewed-by: Milan Zamazal 
Reviewed-by: Francesco Romani 
Reviewed-by: Piotr Kliczewski 
---
M debian/vdsm.install
M init/daemonAdapter
M tests/Makefile.am
A tests/unicode_test.py
M vdsm.spec.in
M vdsm/Makefile.am
A vdsm/sitecustomize.py
7 files changed, 84 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/73/51273/1

diff --git a/debian/vdsm.install b/debian/vdsm.install
index 17ec403..8a659d6 100644
--- a/debian/vdsm.install
+++ b/debian/vdsm.install
@@ -134,6 +134,7 @@
 ./usr/share/vdsm/storage/threadPool.py
 ./usr/share/vdsm/storage/volume.py
 ./usr/share/vdsm/supervdsm.py
+./usr/share/vdsm/sitecustomize.py
 ./usr/share/vdsm/supervdsmServer
 ./usr/share/vdsm/v2v.py
 ./usr/share/vdsm/vdsm
diff --git a/init/daemonAdapter b/init/daemonAdapter
index a112af0..c67ff57 100755
--- a/init/daemonAdapter
+++ b/init/daemonAdapter
@@ -69,12 +69,20 @@
 os.nice(config.getint('vars', 'vdsm_nice'))
 
 env = os.environ.copy()
+
+# Python path is needed to get sitecustomize.py imported from
+# target directory.
+pythonpath = os.path.dirname(self._args.target[0])
+if 'PYTHONPATH' in env:
+pythonpath += ':' + env['PYTHONPATH']
+
 env.update({
 'LIBVIRT_LOG_FILTERS': config.get(
 'vars', 'libvirt_env_variable_log_filters'),
 'LIBVIRT_LOG_OUTPUTS': config.get(
 'vars', 'libvirt_env_variable_log_outputs'),
 'LC_ALL': 'C',
+'PYTHONPATH': pythonpath,
 })
 
 cmd = [self._args.target[0]] + self._args.targetOptions
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 1b33ef8..a9836a1 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -104,6 +104,7 @@
testlibTests.py \
toolTests.py \
transportWrapperTests.py \
+   unicode_test.py \
utilsTests.py \
vdscliTests.py \
vdsClientTests.py \
diff --git a/tests/unicode_test.py b/tests/unicode_test.py
new file mode 100644
index 000..e6f6f35
--- /dev/null
+++ b/tests/unicode_test.py
@@ -0,0 +1,67 @@
+#
+# Copyright 2015 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
+#
+# Refer to the README and

Change in vdsm[ovirt-3.6]: startup: Change system default encoding to utf8

2016-01-03 Thread automation 
gerrit-hooks has posted comments on this change.

Change subject: startup: Change system default encoding to utf8
..


Patch Set 1:

* #1281940::Update tracker: OK
* Check Bug-Url::OK
* Check Public Bug::#1281940::OK, public bug
* Check Product::#1281940::OK, Correct classification oVirt
* Check TM::#1281940::OK, correct target milestone ovirt-3.6.3
* Check merged to previous::OK, change not open on any previous branch

-- 
To view, visit https://gerrit.ovirt.org/51273
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Icc3f072a499ba4034bdbedd09eb60d7a3a32f9c8
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Nir Soffer 
Gerrit-Reviewer: Francesco Romani 
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal 
Gerrit-Reviewer: Piotr Kliczewski 
Gerrit-Reviewer: gerrit-hooks 
Gerrit-HasComments: No
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches