[galaxy-dev] UnicodeDecodeError - suggested note regarding UTF8 encoding in postgres databases
I've finally joined the happy throng suffering from Unicode errors. In my particular case, running the deseq tool from the main toolshed brought down my Galaxy server, and it wouldn't restart. The traceback in paster.log contained this Traceback (most recent call last): [snip] File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2903, in __init__ self._init_metadata() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3306, in _init_metadata self.__buffer_rows() File "/home/galaxy-dev/galaxy/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 3326, in __buffer_rows self.__rowbuffer = collections.deque(self.cursor.fetchmany(size)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 162: ordinal not in range(128) The same error occurred each time I tried to restart the galaxy server until I restored the galaxy database from last night's postgres backup. The problem was my galaxy database in postgres was using SQL_ASCII encoding, which turns out to be a bad idea if you're ever going to write Unicode characters there. (I'm quite disappointed that this is the default encoding for postgres databases on CentOS 6.) The fix was to dump the galaxy database, recreate it with UTF8 encoding, and reload it. Now everything works, and that one tool doesn't crash my galaxy server. I thought it would be worth noting that postgres databases should be created with UTF8 encoding on this page: http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Switching_to_a_database_server I think the right thing to do is to ensure you set the default database encoding for postgres when you run initdb. You only get one chance to get this right. After that, a way to change the default database encoding to UTF8 is given here: https://gist.github.com/ffmike/877447 Scary stuff, but it worked well enough for me. Alternatively, if you didn't think about encodings when you ran initdb, you can think about it every time you run createdb. Hope this helps someone. cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte
Thank you so much Subject: Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte From: d...@bx.psu.edu Date: Thu, 9 May 2013 12:45:25 -0400 CC: galaxy-dev@lists.bx.psu.edu To: ramao_tiago_tibur...@hotmail.com Assuming you checked Galaxy out using mercurial, stop your Galaxy instance and go into the Galaxy root directory and execute: hg pull -u Then start your Galaxy instance back up. On May 9, 2013, at 12:36 PM, Ramon Tiburski wrote:OK. thank you so much, but how can I update my instance? Subject: Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte From: d...@bx.psu.edu Date: Thu, 9 May 2013 11:02:46 -0400 CC: galaxy-dev@lists.bx.psu.edu To: ramao_tiago_tibur...@hotmail.com Hi Ramon, I've committed a patch to the stable branch that should resolve this issue. Please update your Galaxy instance and let us know if this has fixed it for you. Thanks for using Galaxy, Dan On May 9, 2013, at 8:40 AM, Ramon Tiburski wrote:Hi, I uploaded some data and when I click in the eye to see the data I get the following error. I don't know what can I do to fix this problem, I trying install some packges from python but didn't solve my problem. I'm starting think about my operational system, I have ubuntu 12.04 32bit, maybe I need get a 64bit. Someone can help me?Server Error URL: http://localhost:8080/datasets/8c49be448cfe29bc/display/?preview=True Module paste.exceptions.errormiddleware:144 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:194 in handle_request >> return body( environ, start_response ) Module galaxy.web.framework:876 in render >> template.render_context( context ) Module mako.template:319 in render_context >> **kwargs) Module mako.runtime:692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template >> callable_(context, *args, **kwargs) Module _base_mako:42 in render_body >> __M_writer(unicode(next.body())) Module _dataset_large_file_mako:45 in render_body >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' ) >> ))) UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte
Assuming you checked Galaxy out using mercurial, stop your Galaxy instance and go into the Galaxy root directory and execute: hg pull -u Then start your Galaxy instance back up. On May 9, 2013, at 12:36 PM, Ramon Tiburski wrote: > OK. thank you so much, but how can I update my instance? > > Subject: Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte > 0xb9 in position 15: invalid start byte > From: d...@bx.psu.edu > Date: Thu, 9 May 2013 11:02:46 -0400 > CC: galaxy-dev@lists.bx.psu.edu > To: ramao_tiago_tibur...@hotmail.com > > Hi Ramon, > > I've committed a patch to the stable branch that should resolve this issue. > Please update your Galaxy instance and let us know if this has fixed it for > you. > > > Thanks for using Galaxy, > > Dan > > > On May 9, 2013, at 8:40 AM, Ramon Tiburski wrote: > > Hi, > > I uploaded some data and when I click in the eye to see the data I get the > following error. I don't know what can I do to fix this problem, I trying > install some packges from python but didn't solve my problem. I'm starting > think about my operational system, I have ubuntu 12.04 32bit, maybe I need > get a 64bit. > > Someone can help me? > Server Error > > URL: http://localhost:8080/datasets/8c49be448cfe29bc/display/?preview=True > Module paste.exceptions.errormiddleware:144 in __call__ > >> app_iter = self.application(environ, sr_checker) > Module paste.debug.prints:106 in __call__ > >> environ, self.app) > Module paste.wsgilib:543 in intercept_output > >> app_iter = application(environ, replacement_start_response) > Module paste.recursive:84 in __call__ > >> return self.application(environ, start_response) > Module paste.httpexceptions:633 in __call__ > >> return self.application(environ, start_response) > Module galaxy.web.framework.base:132 in __call__ > >> return self.handle_request( environ, start_response ) > Module galaxy.web.framework.base:194 in handle_request > >> return body( environ, start_response ) > Module galaxy.web.framework:876 in render > >> template.render_context( context ) > Module mako.template:319 in render_context > >> **kwargs) > Module mako.runtime:692 in _render_context > >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) > Module mako.runtime:718 in _exec_template > >> callable_(context, *args, **kwargs) > Module _base_mako:42 in render_body > >> __M_writer(unicode(next.body())) > Module _dataset_large_file_mako:45 in render_body > >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' > >> ) ))) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: > invalid start byte > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte
OK. thank you so much, but how can I update my instance? Subject: Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte From: d...@bx.psu.edu Date: Thu, 9 May 2013 11:02:46 -0400 CC: galaxy-dev@lists.bx.psu.edu To: ramao_tiago_tibur...@hotmail.com Hi Ramon, I've committed a patch to the stable branch that should resolve this issue. Please update your Galaxy instance and let us know if this has fixed it for you. Thanks for using Galaxy, Dan On May 9, 2013, at 8:40 AM, Ramon Tiburski wrote:Hi, I uploaded some data and when I click in the eye to see the data I get the following error. I don't know what can I do to fix this problem, I trying install some packges from python but didn't solve my problem. I'm starting think about my operational system, I have ubuntu 12.04 32bit, maybe I need get a 64bit. Someone can help me?Server Error URL: http://localhost:8080/datasets/8c49be448cfe29bc/display/?preview=True Module paste.exceptions.errormiddleware:144 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:194 in handle_request >> return body( environ, start_response ) Module galaxy.web.framework:876 in render >> template.render_context( context ) Module mako.template:319 in render_context >> **kwargs) Module mako.runtime:692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template >> callable_(context, *args, **kwargs) Module _base_mako:42 in render_body >> __M_writer(unicode(next.body())) Module _dataset_large_file_mako:45 in render_body >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' ) >> ))) UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte
Hi Ramon, I've committed a patch to the stable branch that should resolve this issue. Please update your Galaxy instance and let us know if this has fixed it for you. Thanks for using Galaxy, Dan On May 9, 2013, at 8:40 AM, Ramon Tiburski wrote: > Hi, > > I uploaded some data and when I click in the eye to see the data I get the > following error. I don't know what can I do to fix this problem, I trying > install some packges from python but didn't solve my problem. I'm starting > think about my operational system, I have ubuntu 12.04 32bit, maybe I need > get a 64bit. > > Someone can help me? > Server Error > > URL: http://localhost:8080/datasets/8c49be448cfe29bc/display/?preview=True > Module paste.exceptions.errormiddleware:144 in __call__ > >> app_iter = self.application(environ, sr_checker) > Module paste.debug.prints:106 in __call__ > >> environ, self.app) > Module paste.wsgilib:543 in intercept_output > >> app_iter = application(environ, replacement_start_response) > Module paste.recursive:84 in __call__ > >> return self.application(environ, start_response) > Module paste.httpexceptions:633 in __call__ > >> return self.application(environ, start_response) > Module galaxy.web.framework.base:132 in __call__ > >> return self.handle_request( environ, start_response ) > Module galaxy.web.framework.base:194 in handle_request > >> return body( environ, start_response ) > Module galaxy.web.framework:876 in render > >> template.render_context( context ) > Module mako.template:319 in render_context > >> **kwargs) > Module mako.runtime:692 in _render_context > >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) > Module mako.runtime:718 in _exec_template > >> callable_(context, *args, **kwargs) > Module _base_mako:42 in render_body > >> __M_writer(unicode(next.body())) > Module _dataset_large_file_mako:45 in render_body > >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' > >> ) ))) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: > invalid start byte > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte
Hi, I uploaded some data and when I click in the eye to see the data I get the following error. I don't know what can I do to fix this problem, I trying install some packges from python but didn't solve my problem. I'm starting think about my operational system, I have ubuntu 12.04 32bit, maybe I need get a 64bit. Someone can help me?Server Error URL: http://localhost:8080/datasets/8c49be448cfe29bc/display/?preview=True Module paste.exceptions.errormiddleware:144 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:194 in handle_request >> return body( environ, start_response ) Module galaxy.web.framework:876 in render >> template.render_context( context ) Module mako.template:319 in render_context >> **kwargs) Module mako.runtime:692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template >> callable_(context, *args, **kwargs) Module _base_mako:42 in render_body >> __M_writer(unicode(next.body())) Module _dataset_large_file_mako:45 in render_body >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' ) >> ))) UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 15: invalid start byte ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UnicodeDecodeError
Thanks for reporting this. I've attempted to reproduce the workflow related error you're seeing, but have been unable to do so. Do you know what revision your galaxy instance is running? You should not have to avoid special characters in dataset names. If you do run into this issue again, please let me know. -Dannon On Jun 9, 2011, at 5:31 AM, Sarah Diehl wrote: > Hello everybody, > > currently I have some trouble with the following kind of error: > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 155: > ordinal not in range(128) > > We have encountered it twice now. The first time it was related to a shared > library, which could not be opened anymore because of this error. So we > deleted the library and set it up again. > > The second time it was related to a workflow. Clicking on "Run" caused this > error to appear, while editing still worked. This time I saved the error > message and you can find it attached to this mail. The cause of the error was > a < sign in a renaming step inside the workflow (the < was part of the new > name). After removing the < everything worked again. > > However, when using the < in a history item, it works without a problem. > > Unfortunately I cannot reproduce the error currently on my machine. The > computer that caused it was a Mac. > > Should we just try to avoid any special characters or is this actually a bug > which could be fixed? > > Best regards, > Sarah > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] UnicodeDecodeError
Hello everybody, currently I have some trouble with the following kind of error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 155: ordinal not in range(128) We have encountered it twice now. The first time it was related to a shared library, which could not be opened anymore because of this error. So we deleted the library and set it up again. The second time it was related to a workflow. Clicking on "Run" caused this error to appear, while editing still worked. This time I saved the error message and you can find it attached to this mail. The cause of the error was a < sign in a renaming step inside the workflow (the < was part of the new name). After removing the < everything worked again. However, when using the < in a history item, it works without a problem. Unfortunately I cannot reproduce the error currently on my machine. The computer that caused it was a Mac. Should we just try to avoid any special characters or is this actually a bug which could be fixed? Best regards, Sarah 10.1.4.162 - - [09/Jun/2011:10:56:28 +0200] "GET /workflow/run?id=91cc5a0bc3008f7d HTTP/1.1" 500 - "http://solweb2/workflow/list_for_run"; "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16" Error - : 'ascii' codec can't decode byte 0xe2 in position 155: ordinal not in range(128) URL: http://solweb2/workflow/run?id=91cc5a0bc3008f7d File '/galaxy/galaxy_server/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py', line 143 in __call__ app_iter = self.application(environ, start_response) File '/galaxy/galaxy_server/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/galaxy/galaxy_server/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/galaxy/galaxy_server/lib/galaxy/web/framework/base.py', line 145 in __call__ body = method( trans, **kwargs ) File '/galaxy/galaxy_server/lib/galaxy/web/controllers/workflow.py', line 1397 in run incoming=kwargs ) File '/galaxy/galaxy_server/lib/galaxy/web/framework/__init__.py', line 650 in fill_template return self.fill_template_mako( filename, **kwargs ) File '/galaxy/galaxy_server/lib/galaxy/web/framework/__init__.py', line 661 in fill_template_mako return template.render( **data ) File '/galaxy/galaxy_server/eggs/Mako-0.2.5-py2.6.egg/mako/template.py', line 133 in render return runtime._render(self, self.callable_, args, data) File '/galaxy/galaxy_server/eggs/Mako-0.2.5-py2.6.egg/mako/runtime.py', line 364 in _render _render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data)) File '/galaxy/galaxy_server/eggs/Mako-0.2.5-py2.6.egg/mako/runtime.py', line 381 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/galaxy/galaxy_server/eggs/Mako-0.2.5-py2.6.egg/mako/runtime.py', line 414 in _exec_template callable_(context, *args, **kwargs) File '/galaxy/galaxy_server/database/compiled_templates/base.mako.py', line 40 in render_body __M_writer(unicode(next.body())) File '/galaxy/galaxy_server/eggs/Mako-0.2.5-py2.6.egg/mako/runtime.py', line 255 in return lambda *args, **kwargs:callable_(self.context, *args, **kwargs) File '/galaxy/galaxy_server/database/compiled_templates/workflow/run.mako.py', line 226 in render_body __M_writer(unicode(do_inputs( module.get_runtime_inputs(type_filter), step.state.inputs, errors.get( step.id, dict() ), "", step ))) File '/galaxy/galaxy_server/database/compiled_templates/workflow/run.mako.py', line 39 in do_inputs return render_do_inputs(context.locals_(__M_locals),inputs,values,errors,prefix,step,other_values) File '/galaxy/galaxy_server/database/compiled_templates/workflow/run.mako.py', line 391 in render_do_inputs __M_writer(unicode(row_for_param( input, values[ input.name ], other_values, errors, prefix, step ))) File '/galaxy/galaxy_server/database/compiled_templates/workflow/run.mako.py', line 297 in row_for_param return render_row_for_param(context,param,value,other_values,error_dict,prefix,step) File '/galaxy/galaxy_server/database/compiled_templates/workflow/run.mako.py', line 466 in render_row_for_param __M_writer(unicode(param.get_html_field( t, value, other_values ).get_html( str(step.id) + "|" + prefix ))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 155: ordinal not in range(128) CGI Variables - CONTENT_LENGTH: '0' HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' HTTP_ACCEPT_CHARSET: 'ISO-8859-1,utf-8;q=0.7,*;q=0.7' HTTP_ACCEPT_ENCODING: 'gzip,deflate' HTTP_ACCEPT_LANGUAGE: 'en-us,en;q=0.5' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=f611411aa8a0c6655dce25cc8ce7931696473135a41884694eefb704cf34d98429880ec7e01926f8' HTTP_HOST: 'solweb2' HTTP_REFERER: 'http://solweb2/workflow/l
