Re: [aroma.affymetrix] Error when analyzing CEL files using process() function: No permission to modify existing file: /probeData...
I've updated aroma.affymetrix v2.11.8 and friends. Update by:
source(http://callr.org/install#aroma.affymetrix;)
I'd be keen to hear whether this solves your problem or not.
/Henrik
On Thu, Sep 4, 2014 at 1:47 PM, Henrik Bengtsson h...@biostat.ucsf.edu wrote:
Hi,
good that it works for you with this workaround. The fix in aroma is
fairly simple(*). I'll let you know when an updated version is
available.
(*) DETAILS: This has to do with base::file.copy() and its argument
'copy.mode' which was introduced in R 2.13.0 (April 2011). The thing
is that they set the default to TRUE (whereas the previous behavior
was equivalent to FALSE), meaning any files copied will inherit the
file permissions from the source. Various steps in the aroma pipeline
*copies* the source CEL file and uses that as a template to store
updates signals. This is what hits you. I'll updating to make use of
base::file.copy(..., copy.mode=FALSE) such that one can work with also
read-only CEL files.
Thanks for reporting on this (not too unlikely) use case
Henrik
On Thu, Sep 4, 2014 at 1:35 PM, Taylor Raborn taylor.rab...@gmail.com wrote:
Hi Henrik:
Thank you for the prompt reply. You are right- the .CEL files are all read
only (see their permissions, below)
rtraborn@Mason:
/N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2
ls -lkhtra
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL
After changing the permissions, the job performed without issue (I'm showing
you only the results of 1/9 for the sake of brevity):
csBC - process(bc,verbose=verbose)
Background correcting data set...
Number of arrays: 9
Array #1 ('GSM709330_HSB194-S1C-L') of 9...
Adjusting PM signals only
Obtaining signals...
Obtaining signals...done
Applying normal+exponential signal model...
Applying normal+exponential signal model...done
Writing adjusted probe signals...
Adding temporary suffix from file...
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Suffix: .tmp
Rename existing file?: FALSE
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Adding temporary suffix from file...done
Creating CEL file for results, if missing...
Creating CEL file for results, if missing...done
Writing adjusted intensities...
Writing adjusted intensities...done
Dropping temporary suffix from file...
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Suffix: .tmp
Regular expression for suffix: \.tmp$
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Renaming existing file...
Result: TRUE
Renaming existing file...done
Dropping temporary suffix from file...done
Writing adjusted probe signals...done
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 709078 37.91166886 62.4 1166886 62.4
Vcells 31586985 241.0 47054688 359.0 46991036 358.6
AffymetrixCelFile:
Name: GSM709330_HSB194-S1C-L
Tags:
Full name: GSM709330_HSB194-S1C-L
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
File size: 62.73 MB (65775403 bytes)
RAM: 0.00 MB
File format: v4 (binary; XDA)
Platform: Affymetrix
Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
Timestamp: 2011-01-25 12:15:03
Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
-
There must be a file permissions transfer behavior that takes place on a
multi-user cluster environment that doesn't take place on other types of
machines, because I was unable to reproduce this error elsewhere. I didn't
bother changing the raw .CEL files' permissions because I never experienced
a problem of this nature with .CEL files.
Let me know if you patch this up so I can pull the latest code from the
aroma repo.
Thanks for your help with this and for taking a look at the issue so
quickly.
Best regards,
Taylor
On Thursday, September 4, 2014 1:37:36 PM UTC-4, Henrik Bengtsson wrote:
Hi,
interesting. A quick guess is that the input file, i.e.
rawData/NIK_2014/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
is write protected (e.g. file owned by someone else). This could be
the reason, because internally that file is copied and used as a
template and the copy may inherit the
Re: [aroma.affymetrix] Error when analyzing CEL files using process() function: No permission to modify existing file: /probeData...
Hi Henrik:
Thank you for patching this so quickly. I didn't realize the defaults of
file.copy() had changed in R 2.13.0.
I'll test this once my current job (n = ~400 CEL files) is complete, and
I'll let you know what I find.
Taylor
On Friday, September 5, 2014 4:15:56 PM UTC-4, Henrik Bengtsson wrote:
I've updated aroma.affymetrix v2.11.8 and friends. Update by:
source(http://callr.org/install#aroma.affymetrix;)
I'd be keen to hear whether this solves your problem or not.
/Henrik
On Thu, Sep 4, 2014 at 1:47 PM, Henrik Bengtsson h...@biostat.ucsf.edu
javascript: wrote:
Hi,
good that it works for you with this workaround. The fix in aroma is
fairly simple(*). I'll let you know when an updated version is
available.
(*) DETAILS: This has to do with base::file.copy() and its argument
'copy.mode' which was introduced in R 2.13.0 (April 2011). The thing
is that they set the default to TRUE (whereas the previous behavior
was equivalent to FALSE), meaning any files copied will inherit the
file permissions from the source. Various steps in the aroma pipeline
*copies* the source CEL file and uses that as a template to store
updates signals. This is what hits you. I'll updating to make use of
base::file.copy(..., copy.mode=FALSE) such that one can work with also
read-only CEL files.
Thanks for reporting on this (not too unlikely) use case
Henrik
On Thu, Sep 4, 2014 at 1:35 PM, Taylor Raborn taylor...@gmail.com
javascript: wrote:
Hi Henrik:
Thank you for the prompt reply. You are right- the .CEL files are all
read
only (see their permissions, below)
rtraborn@Mason:
/N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2
ls -lkhtra
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL
After changing the permissions, the job performed without issue (I'm
showing
you only the results of 1/9 for the sake of brevity):
csBC - process(bc,verbose=verbose)
Background correcting data set...
Number of arrays: 9
Array #1 ('GSM709330_HSB194-S1C-L') of 9...
Adjusting PM signals only
Obtaining signals...
Obtaining signals...done
Applying normal+exponential signal model...
Applying normal+exponential signal model...done
Writing adjusted probe signals...
Adding temporary suffix from file...
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Suffix: .tmp
Rename existing file?: FALSE
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Adding temporary suffix from file...done
Creating CEL file for results, if missing...
Creating CEL file for results, if missing...done
Writing adjusted intensities...
Writing adjusted intensities...done
Dropping temporary suffix from file...
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Suffix: .tmp
Regular expression for suffix: \.tmp$
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Renaming existing file...
Result: TRUE
Renaming existing file...done
Dropping temporary suffix from file...done
Writing adjusted probe signals...done
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 709078 37.91166886 62.4 1166886 62.4
Vcells 31586985 241.0 47054688 359.0 46991036 358.6
AffymetrixCelFile:
Name: GSM709330_HSB194-S1C-L
Tags:
Full name: GSM709330_HSB194-S1C-L
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
File size: 62.73 MB (65775403 bytes)
RAM: 0.00 MB
File format: v4 (binary; XDA)
Platform: Affymetrix
Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
Timestamp: 2011-01-25 12:15:03
Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
-
There must be a file permissions transfer behavior that takes place on
a
multi-user cluster environment that doesn't take place on other types
of
machines, because I was unable to reproduce this error elsewhere. I
didn't
bother changing the raw .CEL
Re: [aroma.affymetrix] Error when analyzing CEL files using process() function: No permission to modify existing file: /probeData...
Hi Henrik:
Thank you for the prompt reply. You are right- the .CEL files are all read
only (see their permissions, below)
rtraborn@Mason:
/N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2
ls -lkhtra
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL
After changing the permissions, the job performed without issue (I'm
showing you only the results of 1/9 for the sake of brevity):
csBC - process(bc,verbose=verbose)
Background correcting data set...
Number of arrays: 9
Array #1 ('GSM709330_HSB194-S1C-L') of 9...
Adjusting PM signals only
Obtaining signals...
Obtaining signals...done
Applying normal+exponential signal model...
Applying normal+exponential signal model...done
Writing adjusted probe signals...
Adding temporary suffix from file...
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Suffix: .tmp
Rename existing file?: FALSE
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Adding temporary suffix from file...done
Creating CEL file for results, if missing...
Creating CEL file for results, if missing...done
Writing adjusted intensities...
Writing adjusted intensities...done
Dropping temporary suffix from file...
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Suffix: .tmp
Regular expression for suffix: \.tmp$
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Renaming existing file...
Result: TRUE
Renaming existing file...done
Dropping temporary suffix from file...done
Writing adjusted probe signals...done
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 709078 37.91166886 62.4 1166886 62.4
Vcells 31586985 241.0 47054688 359.0 46991036 358.6
AffymetrixCelFile:
Name: GSM709330_HSB194-S1C-L
Tags:
Full name: GSM709330_HSB194-S1C-L
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
File size: 62.73 MB (65775403 bytes)
RAM: 0.00 MB
File format: v4 (binary; XDA)
Platform: Affymetrix
Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
Timestamp: 2011-01-25 12:15:03
Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
-
There must be a file permissions transfer behavior that takes place on a
multi-user cluster environment that doesn't take place on other types of
machines, because I was unable to reproduce this error elsewhere. I didn't
bother changing the raw .CEL files' permissions because I never experienced
a problem of this nature with .CEL files.
Let me know if you patch this up so I can pull the latest code from the
aroma repo.
Thanks for your help with this and for taking a look at the issue so
quickly.
Best regards,
Taylor
On Thursday, September 4, 2014 1:37:36 PM UTC-4, Henrik Bengtsson wrote:
Hi,
interesting. A quick guess is that the input file, i.e.
rawData/NIK_2014/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
is write protected (e.g. file owned by someone else). This could be
the reason, because internally that file is copied and used as a
template and the copy may inherit the file permissions. What does ls
-l say about the above file?
If this is the case, I need to update the copying such that the new
file has proper permissions.
Thanks for the report
Henrik
On Thu, Sep 4, 2014 at 10:15 AM, Taylor Raborn taylor...@gmail.com
javascript: wrote:
Hi Henrik:
I just come across an unusual error when using aroma.affymetrix on our
high-memory HPC machine. It may or may not be platform-specific, and
I'll
give you all the information I can in the hopes of isolating and
identifying
the error.
Traceback:
traceback()
18: stop(cond)
17: throw.Exception(Exception(...))
16: throw(Exception(...))
15: throw.default(No permission to modify existing file: , pathname)
14: throw(No permission to modify existing file: , pathname)
13: getWritablePathname.Arguments(static, ...)
12: getWritablePathname(static, ...) at text#1
11: Arguments$getWritablePathname(pathname, mustExist = TRUE)
10: renameFile.default(srcPathname, pathname, ...)
9: renameFile(srcPathname, pathname, ...)
8: renameTo.GenericDataFile(res, filename = pathname, verbose =
less(verbose))
7:
Re: [aroma.affymetrix] Error when analyzing CEL files using process() function: No permission to modify existing file: /probeData...
Hi,
good that it works for you with this workaround. The fix in aroma is
fairly simple(*). I'll let you know when an updated version is
available.
(*) DETAILS: This has to do with base::file.copy() and its argument
'copy.mode' which was introduced in R 2.13.0 (April 2011). The thing
is that they set the default to TRUE (whereas the previous behavior
was equivalent to FALSE), meaning any files copied will inherit the
file permissions from the source. Various steps in the aroma pipeline
*copies* the source CEL file and uses that as a template to store
updates signals. This is what hits you. I'll updating to make use of
base::file.copy(..., copy.mode=FALSE) such that one can work with also
read-only CEL files.
Thanks for reporting on this (not too unlikely) use case
Henrik
On Thu, Sep 4, 2014 at 1:35 PM, Taylor Raborn taylor.rab...@gmail.com wrote:
Hi Henrik:
Thank you for the prompt reply. You are right- the .CEL files are all read
only (see their permissions, below)
rtraborn@Mason:
/N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2
ls -lkhtra
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL
-r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL
After changing the permissions, the job performed without issue (I'm showing
you only the results of 1/9 for the sake of brevity):
csBC - process(bc,verbose=verbose)
Background correcting data set...
Number of arrays: 9
Array #1 ('GSM709330_HSB194-S1C-L') of 9...
Adjusting PM signals only
Obtaining signals...
Obtaining signals...done
Applying normal+exponential signal model...
Applying normal+exponential signal model...done
Writing adjusted probe signals...
Adding temporary suffix from file...
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Suffix: .tmp
Rename existing file?: FALSE
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Adding temporary suffix from file...done
Creating CEL file for results, if missing...
Creating CEL file for results, if missing...done
Writing adjusted intensities...
Writing adjusted intensities...done
Dropping temporary suffix from file...
Temporary pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
Suffix: .tmp
Regular expression for suffix: \.tmp$
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
Renaming existing file...
Result: TRUE
Renaming existing file...done
Dropping temporary suffix from file...done
Writing adjusted probe signals...done
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 709078 37.91166886 62.4 1166886 62.4
Vcells 31586985 241.0 47054688 359.0 46991036 358.6
AffymetrixCelFile:
Name: GSM709330_HSB194-S1C-L
Tags:
Full name: GSM709330_HSB194-S1C-L
Pathname:
probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
File size: 62.73 MB (65775403 bytes)
RAM: 0.00 MB
File format: v4 (binary; XDA)
Platform: Affymetrix
Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
Timestamp: 2011-01-25 12:15:03
Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
-
There must be a file permissions transfer behavior that takes place on a
multi-user cluster environment that doesn't take place on other types of
machines, because I was unable to reproduce this error elsewhere. I didn't
bother changing the raw .CEL files' permissions because I never experienced
a problem of this nature with .CEL files.
Let me know if you patch this up so I can pull the latest code from the
aroma repo.
Thanks for your help with this and for taking a look at the issue so
quickly.
Best regards,
Taylor
On Thursday, September 4, 2014 1:37:36 PM UTC-4, Henrik Bengtsson wrote:
Hi,
interesting. A quick guess is that the input file, i.e.
rawData/NIK_2014/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
is write protected (e.g. file owned by someone else). This could be
the reason, because internally that file is copied and used as a
template and the copy may inherit the file permissions. What does ls
-l say about the above file?
If this is the case, I need to update the copying such that the new
file has proper permissions.
Thanks for the report
Henrik
On Thu, Sep 4, 2014 at 10:15 AM, Taylor Raborn taylor...@gmail.com
wrote:
Hi
