Hi Starr

I will try to answer some of your questions. Next time I recommend to ask just one question (with a corresponding subject line). This will make it much easier for the community to help you, and for others with similar problems to find the correct e-mail thread.

see below for my answers...

On 11/06/2013 11:43 PM, Hazard, E. Starr wrote:
Hello,
I am a new user of Galaxy.
I have a Galaxy instance running (sort of)  on a local research cluster.
I issued the command "hg update stable” today and it retrieved no files.
SO presume I am up-to-date on the stable release. I start the instance
as a user named “galaxy”.
Right now I am still running in “local” mode. I hope to migrate to DRMAA
LSF eventually.
I have tried to set up ProFTP to upload files but have not succeeded so
I use Galaxy Web-upload.
The upload was working nicely and I had  added  a couple of  new tools
and they were working with the uploaded files.
Getting LSF/DRMAA to work was giving me fits and ultimately I deleted
all my history files in an effort to start over.
Presently, files being uploaded appear in history as say job 1 ( in a
new history) The job status in the history panel of the web GUI
changes from purple to yellow then then to red indicating some sort of
error. There is no viewable error text captured, but I can click on the
“eye” icon and see the
first megabyte of the data (for tiny files I can see the entire content
and it’s intact). In the Galaxy file system, however, these files appear
but have a different number , say, dataset_399.dat

On my system the uploaded files appear in
/PATH/galaxy-dist/database/files/000

My first question is why is the data going into the “000” subdirectory
and not one “owned’ by the user who is uploading?

all files are owned by galaxy (independent of who is uploading them and/or generate new files). The first 1000 files are stored in '000', the next 1000 files in '001', and so on.

Access permission is handled by the SQLite database (unless you have already switched to a PostgreSQL database).


My second question is why is the dataset being labeled as
dataset_399.dat and not dataset_001.dat?

The number is given by the SQLite database. When you want a clean start, you need to remove all files from ~/database/files/000 and start with an empty database.

My third question is why do the uploaded files not appear as selectable
options ( say I have paired-end fastq files and tool wants to have
choices about filenames)? This problem is present for programs that seek
one input file as well.

When you uploaded the files, was the 'format' properly recognized?


  I presume that Galaxy is confused because the numbering in history is
not the same as the numbering in the file upload archive (e.g.
/PATH/galaxy-dist/database/files/000 in my case) so my last question is
how do I “reset” my system to get the dataset and history numbers to be
the same?

The numbering in the history has nothing to do with the file numbering. See above wrt my comment about a 'clean start'


Regards, Hans-Rudolf


Here’s how I launch the Galaxy instance

  sh /shared/app/Galaxy/galaxy-dist/run.sh -v --daemon
--pid-file=Nov6Localdaemon.pid.txt  --log-file=Nov6Local1639daemon.log.txt

Entering daemon mode


Here are the last lines of the log

Starting server in PID 26236.

serving on 0.0.0.0:8089 view at http://127.0.0.1:8089


galaxy.tools.actions.upload_common DEBUG 2013-11-06 16:48:49,624
Changing ownership of
/shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4
with: /usr/bin/sudo -E scripts/external_chown_script.py
/shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4
hazards 502

galaxy.tools.actions.upload_common WARNING 2013-11-06 16:48:49,750
Changing ownership of uploaded file
/shared/app/Galaxy/galaxy-dist/database/tmp/upload_file_data_QZGHm4
failed: sudo: no tty present and no askpass program specified


galaxy.tools.actions.upload_common DEBUG 2013-11-06 16:48:49,751
Changing ownership of
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO with:
/usr/bin/sudo -E scripts/external_chown_script.py
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO hazards 502

galaxy.tools.actions.upload_common WARNING 2013-11-06 16:48:49,775
Changing ownership of uploaded file
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO failed: sudo: no
tty present and no askpass program specified


galaxy.tools.actions.upload_common INFO 2013-11-06 16:48:49,805 tool
upload1 created job id 170


galaxy.jobs DEBUG 2013-11-06 16:48:50,678 (170) Persisting job
destination (destination id: local)

galaxy.jobs.handler INFO 2013-11-06 16:48:50,698 (170) Job dispatched

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:50,994 (170) executing:
python /shared/app/Galaxy/galaxy-dist/tools/data_source/upload.py
/depot/shared/app/Galaxy/galaxy-dist
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO
406:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/dataset_406_files:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat

galaxy.jobs DEBUG 2013-11-06 16:48:51,030 (170) Persisting job
destination (destination id: local)

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:53,335 execution
finished: python
/shared/app/Galaxy/galaxy-dist/tools/data_source/upload.py
/depot/shared/app/Galaxy/galaxy-dist
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpEsyGfO
406:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/dataset_406_files:/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat

galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:53,463 executing
external set_meta script for job 170:
/depot/shared/app/Galaxy/galaxy-dist/set_metadata.sh
/shared/app/Galaxy/galaxy-dist/database/files
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170 .
/depot/shared/app/Galaxy/galaxy-dist/universe_wsgi.ini
/shared/app/Galaxy/galaxy-dist/database/tmp/tmpTq22ot
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy.json
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_in_HistoryDatasetAssociation_410_pUkPU9,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_kwds_HistoryDatasetAssociation_410_iouZ6k,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_out_HistoryDatasetAssociation_410_3c6Nj2,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_results_HistoryDatasetAssociation_410_A6L2LO,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat,/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/metadata_override_HistoryDatasetAssociation_410_8Wub22


galaxy.jobs.runners.local DEBUG 2013-11-06 16:48:54,789 execution of
external set_meta for job 170 finished

galaxy.jobs DEBUG 2013-11-06 16:48:54,815 (170) Changing ownership of
working directory with: /usr/bin/sudo -E
scripts/external_chown_script.py
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170
galaxy 50982

galaxy.jobs ERROR 2013-11-06 16:48:54,839 (170) Failed to change
ownership of
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170,
failing

Traceback (most recent call last):

   File
"/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line
868, in finish

     self.reclaim_ownership()

   File
"/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line
1452, in reclaim_ownership

     self._change_ownership( self.galaxy_system_pwent[0], str(
self.galaxy_system_pwent[3] ) )

   File
"/depot/shared/app/Galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line
1438, in _change_ownership

     assert p.returncode == 0

AssertionError

galaxy.jobs DEBUG 2013-11-06 16:48:54,876 fail(): Moved
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170/galaxy_dataset_406.dat
to /shared/app/Galaxy/galaxy-dist/database/files/000/dataset_406.dat




ls -ltr /shared/app/Galaxy/galaxy-dist/database/files/000


-rw------- 1 galaxy galaxy 93 Nov6 16:48 dataset_406.dat


 From
http://dev.list.galaxyproject.org/Submitting-jobs-as-a-real-user-without-using-chown-please-td4658586.html

I tried this as sudo user


  scripts/external_chown_script.py
/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/170
svcgalaxy 100000

Traceback (most recent call last):

   File "scripts/external_chown_script.py", line 16, in <module>

     import drmaa

   File "build/bdist.linux-x86_64/egg/drmaa/__init__.py", line 41, in
<module>

   File "build/bdist.linux-x86_64/egg/drmaa/wrappers.py", line 43, in
<module>

RuntimeError: could not find drmaa library. Please specify its full path
using the environment variable DRMAA_LIBRARY_PATH



I have no idea why that refers to DRMAA anything because in this case I
was running local
In fact however I get that same error when I start galaxy with  this
”job_conf.xml”

  cat job_conf.xml

<?xml version="1.0"?>

<job_conf>

     <plugins>

         <!-- "workers" is the number of threads for the runner's work
queue.

              The default from <plugins> is used if not defined for a
<plugin>.

           -->

         <plugin id="drmaa" type="runner"
load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="8"/>

     </plugins>

      <handlers default="handlers">

              <!-- Additional job handlers - the id should match the
name of a

                                [server:<id>] in universe_wsgi.ini.

                                         -->

         <handler id="main" tags="handlers"/>

       </handlers>

     <destinations default="drmaa">

         <!-- Destinations define details about remote resources and how
jobs

              should be executed on those remote resources.

          -->

         <destination id="drmaa" runner="drmaa" tags="hpcc3.musc.edu">

             <!-- "TODO: The real user options should maybe not be
considered runner params."

              -->

             <param
id="galaxy_external_runjob_script">scripts/drmaa_external_runner.py</param>

             <param
id="galaxy_external_killjob_script">scripts/drmaa_external_killer.py</param>

             <param
id="galaxy_external_chown_script">scripts/external_chown_script.py</param>

         </destination>

     </destinations>

</job_conf>



The other difference is that in the DRMAA case, the GUI upload job in
the HISTORY panel goes “grey” and stays that way; nothing is ever uploaded.

Summary:
In local mode I can upload  files but there are errors in HISTORY and I
cannot get tools to see the uploaded files.
In DMAA mode I cannot get the GUI upload to happen at all.

BTW in user “galaxy”’s .bashrc file I have this line

export
DRMAA_LIBRARY_PATH=/opt/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libdrmaa.so


I’m baffled.


Thanks

Starr

/shared/app/Galaxy/galaxy-dist/database/files



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Reply via email to