[galaxy-dev] Acceleration Card Optimization
Hello All, I recently was gifted an ioFusion ioFX acceleration card to help speed my bioinformatics work along and I am wondering how best to integrate it into my instance to make Galaxy more efficient. Is it sufficient to merely ensure that the datasets being operated on are on the flash storage (I guess by placing the main Galaxy directory there) or are there other elements that should also move lest they create bottlenecks elsewhere (tmp directory, files related to proxy/ftp server, tool shed, etc.)? Suggestions will be greatly appreciated. Thanks, Adam The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] ProFTPD Config Issues
Hi Joahcim, Thanks so much for your reply. The problem has actually been resolved thanks to a response from Federico Zambelli. Embarrassingly, it did not occur to me that because the passwords had been created while PBKDF2 encryption was enabled, that I would need to reset them after disabling it :P. The FTP server appears to be working fine now. Thanks, Adam On 09/05/2013 03:21 AM, Joachim Jacob | VIB | wrote: Hi Adam, I remembered having a hard time getting this to work. After it worked, networkadmins decided to disallow ftp connections at our university due to security concerns... I do not remember the solution (...), but I can share my config files (I am using apache instead of nginx): This is my working 'proftp.conf': # This is the ProFTPD configuration file # Server Config - config used for anything outside a VirtualHost or Global context ServerNameBITS Galaxy FTP ServerIdenton Welcome to BITS Galaxy FTP server. ServerTypestandalone #ServerAdminroot@localhost #Umask0066 SyslogFacility DAEMON SyslogLevel debug DefaultRoot~ !adm CreateHome on 700 uid 600 gid 601 PassivePorts39000 4 MaxInstances30 Usergalaxy Group galaxy #UseFtpUsers off #AuthPAMoff # Don't do reverse DNS lookups (hangs on DNS problems) UseReverseDNSoff MaxInstances20 # Define the log formats LogFormatdefault%h %l %u %t \%r\ %s %b LogFormatauth%v [%P] %h %t \%r\ %s # General database support (http://www.proftpd.org/docs/contrib/mod_sql.html) #LoadModule mod_sql.c # Support for base-64 or hex encoded MD5 and SHA1 passwords from SQL tables #LoadModule mod_sql_passwd.c # Postgresql support (requires proftpd-postgresql package) # (http://www.proftpd.org/docs/contrib/mod_sql.html) # LoadModule mod_sql_postgres.c Global # Allow users to overwrite files and change permissions AllowOverwrite on AllowStoreRestart on # Bar use of SITE CHMOD Limit SITE_CHMOD DenyAll /Limit # Bar use of RETR (download) since this is not a public file drop Limit RETR DenyAll /Limit SQLLogFile /var/log/proftpd/proftpd.log # get FTP connection over SSH DefaultServeron SFTPEngine on SFTPLog /var/log/proftpd-sftp.log Port 8822 # Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 SQLPasswordEngine on SQLPasswordEncoding hex RequireValidShell off # Set up mod_sql to authenticate against the Galaxy database SQLEngine on SQLBackend postgres SQLConnectInfo galaxydb@127.0.0.1 galaxyftp ** SQLAuthTypesSHA1 SQLAuthenticate users # An empty directory in case chroot fails SQLDefaultHomedir /mnt/galaxytemp/ftptmp SQLDefaultGID 601 SQLDefaultUID 600 SQLMinID 95 # Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user. SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email,password,'600','601','/mnt/galaxydb/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' /Global Good luck, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/05/2013 02:46 AM, Panzer, Adam wrote: Hello Devs, I've been trying for a while now to get ProFTPD working for our local instance. I have: 1) Modified pg_hba.conf 2) Set up the ProFTPD config files 3) Disabled PBKDF2 password encryption by adding use_pbkdf2 = false to universe_wsgi.ini I've read through the posts from others with this issue and made sure to avoid some of the common pitfalls like forgetting to install/enable mod_sql.c, mod_sql_postgres.c, and LoadModule mod_sql_passwd.c or having an incorrect/999 uid/gid in proftpd.conf. Even so, when I attempt to access the server through FileZilla, it continues to kick back login errors for all users: Status:Resolving address of localhost Status:Connecting to 127.0.0.1:21... Status:Connection established, waiting for welcome message... Response:220 ProFTPD 1.3.4a Server (Jay2) [:::127.0.0.1] Command:USER panze...@kids.wustl.edu Response:331 Password required for panze...@kids.wustl.edu Command:PASS *** Response:530 Login incorrect. Error:Critical error Error:Could not connect to server I have included the contents of the relevant config files and logs below
[galaxy-dev] ProFTPD Config Issues
Hello Devs, I've been trying for a while now to get ProFTPD working for our local instance. I have: 1) Modified pg_hba.conf 2) Set up the ProFTPD config files 3) Disabled PBKDF2 password encryption by adding use_pbkdf2 = false to universe_wsgi.ini I've read through the posts from others with this issue and made sure to avoid some of the common pitfalls like forgetting to install/enable mod_sql.c, mod_sql_postgres.c, and LoadModule mod_sql_passwd.c or having an incorrect/999 uid/gid in proftpd.conf. Even so, when I attempt to access the server through FileZilla, it continues to kick back login errors for all users: Status:Resolving address of localhost Status:Connecting to 127.0.0.1:21... Status:Connection established, waiting for welcome message... Response:220 ProFTPD 1.3.4a Server (Jay2) [:::127.0.0.1] Command:USER panze...@kids.wustl.edu Response:331 Password required for panze...@kids.wustl.edu Command:PASS *** Response:530 Login incorrect. Error:Critical error Error:Could not connect to server I have included the contents of the relevant config files and logs below. The proftpd log says that the user name is wrong, but the sqllog seems to indicate a successful hit on the user but a failed password authentication (again, despite disabling PBKDF2). I would be most grateful if someone could tell me what I'm doing wrong. Thanks, Adam - PACKAGE VERSIONS - PostgreSQL 9.2 Nginx 1.1.19 ProFTPD 1.3.4a - PG_HBA.CONF - local all all trust hostall all 127.0.0.1/32trust hostall all ::1/128 trust hostall all 0.0.0.0/0 md5 - PROFTPD.CONF - Include /etc/proftpd/modules.conf UseIPv6on IdentLookupsoff ServerTypestandalone DeferWelcomeoff MultilineRFC2228on DefaultServeron ShowSymlinkson TimeoutNoTransfer600 TimeoutStalled600 TimeoutIdle1200 DisplayLoginwelcome.msg DisplayChdir .message true ListOptions-l DenyFilter\*.*/ Port21 PassivePorts 3 4 MaxInstances30 Usergalaxy Groupgalaxy Umask077 AllowOverwriteon AuthOrdermod_sql.c SQLDefaultGID1001 SQLDefaultUID1001 TransferLog /var/log/proftpd/xferlog SystemLog /var/log/proftpd/proftpd.log IfModule mod_quotatab.c QuotaEngine off /IfModule IfModule mod_ratio.c Ratios off /IfModule IfModule mod_delay.c DelayEngine on /IfModule IfModule mod_ctrls.c ControlsEngineoff ControlsMaxClients2 ControlsLog /var/log/proftpd/controls.log ControlsInterval 5 ControlsSocket/var/run/proftpd/proftpd.sock /IfModule IfModule mod_ctrls_admin.c AdminControlsEngine off /IfModule Include /etc/proftpd/conf.d/ - GALAXY.CONF (galaxy specific conf.d ProFTPD config file) - ServerNameJay2 DefaultRoot ~ CreateHome on dirmode 700 AllowOverwrite on AllowStoreRestart on Limit SITE_CHMOD DenyAll /Limit Limit RETR DenyAll /Limit AuthPAM off SQLPasswordEngine on SQLPasswordEncoding hex SQLEngine on SQLBackend postgres SQLConnectInfo galaxy@localhost:5432 galaxy galaxy SQLAuthTypesSHA1 SQLAuthenticate users SQLDefaultHomedir /var/lib/proftpd/empty SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email,password,'1001','1001','/home/galaxy/galaxy-supp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' SQLLogFile /var/log/proftpd/sqlLog.txt - PROFTPD.LOG - Sep 04 19:03:51 PCF10-WKS1572 proftpd[4846] PCF10-WKS1572 (localhost[127.0.0.1]): FTP session closed. Sep 04 19:14:57 PCF10-WKS1572 proftpd[5034] PCF10-WKS1572 (localhost[127.0.0.1]): FTP session opened. Sep 04 19:14:57 PCF10-WKS1572 proftpd[5034] PCF10-WKS1572 (localhost[127.0.0.1]): USER panze...@kids.wustl.edu (Login failed): No such user found. - SQLLOG.TXT - Sep 04 19:14:57 mod_sql/4.3[5034]: defaulting to 'postgres' backend Sep 04 19:14:57 mod_sql/4.3[5034]: backend module 'mod_sql_postgres/4.0.4' Sep 04 19:14:57 mod_sql/4.3[5034]: backend api'mod_sql_api_v1' Sep 04 19:14:57 mod_sql/4.3[5034]: sql_sess_init Sep 04 19:14:57 mod_sql/4.3[5034]: entering postgres cmd_defineconnection Sep 04 19:14:57 mod_sql/4.3[5034]: name: 'default' Sep 04 19:14:57 mod_sql/4.3[5034]: user: 'galaxy' Sep 04 19:14:57 mod_sql/4.3[5034]: host: 'localhost' Sep 04 19:14:57 mod_sql/4.3[5034]:db: 'galaxy' Sep 04 19:14:57
[galaxy-dev] Problem with Tools Relying on R
Hello All, I have recently deployed a local instance of Galaxy at work and while I've been able to troubleshoot problems with all other tool types, I have a persistent issue with using any tool that relies on R. For deseq: Traceback (most recent call last): File /home/adam/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 109, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File /home/adam/galaxy-dist/lib/galaxy/jobs/__init__.py, line 999, in finish self.sa_session.flush() File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/scoping.py, line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py, line 1356, in flush self._flush(objects) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py, line 1434, in _flush flush_context.execute() File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py, line 261, in execute UOWExecutor().execute(self, tasks) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py, line 753, in execute self.execute_save_steps(trans, task) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py, line 768, in execute_save_steps self.save_objects(trans, task) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py, line 759, in save_objects task.mapper._save_obj(task.polymorphic_tosave_objects, trans) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/mapper.py, line 1413, in _save_obj c = connection.execute(statement.values(value_params), params) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py, line 824, in execute return Connection.executors[c](self, object, multiparams, params) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py, line 874, in _execute_clauseelement return self.__execute_context(context) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py, line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py, line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File /home/adam/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py, line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE job SET update_time=?, stdout=?, stderr=? WHERE job.id = ?' ['2013-05-10 06:12:01.564757', '', 'Error: Loading required package: Biobase\nLoading required package: BiocGenerics\nLoading required package: methods\nLoading required package: parallel\n\nAttaching package: \xe2\x80\x98BiocGenerics\xe2\x80\x99\n\nThe following objects are masked from \xe2\x80\x98package:parallel\xe2\x80\x99:\n\nclusterApply, clusterApplyLB, clusterCall, clusterEvalQ,\nclusterExport, clusterMap, parApply, parCapply, parLapply,\nparLapplyLB, parRapply, parSapply, parSapplyLB\n\nThe following object is masked from \xe2\x80\x98package:stats\xe2\x80\x99:\n\n xtabs\n\nThe following objects are masked from \xe2\x80\x98package:base\xe2\x8 0\x99:\n\nanyDuplicated, as.data.frame, cbind, colnames, duplicated, eval,\nFilter, Find, get, intersect, lapply, Map, mapply, match, mget,\n order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,\nrbind, Reduce, rep.int, rownames, sapply, setdiff, sort, table,\ntapply, union, unique, unlist\n\nWelcome to Bioconductor\n\nVignettes contain introductory material; view with\n\'browseVignettes()\'. To cite Bioconductor, see\n \'citation(Biobase)\', and for packages \'citation(pkgname)\'.\n\nLoading required package: locfit\nlocfit 1.5-9.1 \t 2013-03-22\nLoading required package: lattice\nWarning message:\nIn xy.coords(x, y, xlabel, ylabel, log) :\n 9 y values = 0 omitted from logarithmic plot\nError in hclustfun(distfun(x)) : \n NA/NaN/Inf in foreign function call (arg 11)\nCalls: heatmap - hclustfun\nExecution halted\n\n', 116] For cummerbund: Traceback (most recent call last): File /home/adam/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 109, in queue_job job_wrapper.finish( stdout, stderr,
