Hi Henrik:
Thank you for patching this so quickly. I didn't realize the defaults of
file.copy() had changed in R 2.13.0.
I'll test this once my current job (n = ~400 CEL files) is complete, and
I'll let you know what I find.
Taylor
On Friday, September 5, 2014 4:15:56 PM UTC-4, Henrik Bengtsson wrote:
>
> I've updated aroma.affymetrix v2.11.8 and friends. Update by:
>
> source("http://callr.org/install#aroma.affymetrix";)
>
> I'd be keen to hear whether this solves your problem or not.
>
> /Henrik
>
> On Thu, Sep 4, 2014 at 1:47 PM, Henrik Bengtsson <h...@biostat.ucsf.edu
> <javascript:>> wrote:
> > Hi,
> >
> > good that it works for you with this workaround. The fix in aroma is
> > fairly simple(*). I'll let you know when an updated version is
> > available.
> >
> > (*) DETAILS: This has to do with base::file.copy() and its argument
> > 'copy.mode' which was introduced in R 2.13.0 (April 2011). The thing
> > is that they set the default to TRUE (whereas the previous behavior
> > was equivalent to FALSE), meaning any files copied will inherit the
> > file permissions from the source. Various steps in the aroma pipeline
> > *copies* the source CEL file and uses that as a template to store
> > updates signals. This is what hits you. I'll updating to make use of
> > base::file.copy(..., copy.mode=FALSE) such that one can work with also
> > read-only CEL files.
> >
> > Thanks for reporting on this (not too unlikely) use case
> >
> > Henrik
> >
> > On Thu, Sep 4, 2014 at 1:35 PM, Taylor Raborn <taylor...@gmail.com
> <javascript:>> wrote:
> >> Hi Henrik:
> >>
> >> Thank you for the prompt reply. You are right- the .CEL files are all
> read
> >> only (see their permissions, below)
> >>
> >> rtraborn@Mason:
> >>
> /N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2>
>
> >> ls -lkhtra
> >>
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL
> >>
> >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL
> >>
> >>
> >> After changing the permissions, the job performed without issue (I'm
> showing
> >> you only the results of 1/9 for the sake of brevity):
> >>
> >>> csBC <- process(bc,verbose=verbose)
> >>
> >> Background correcting data set...
> >>
> >> Number of arrays: 9
> >>
> >> Array #1 ('GSM709330_HSB194-S1C-L') of 9...
> >>
> >> Adjusting PM signals only
> >>
> >> Obtaining signals...
> >>
> >> Obtaining signals...done
> >>
> >> Applying normal+exponential signal model...
> >>
> >> Applying normal+exponential signal model...done
> >>
> >> Writing adjusted probe signals...
> >>
> >> Adding temporary suffix from file...
> >>
> >> Pathname:
> >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
> >>
> >> Suffix: .tmp
> >>
> >> Rename existing file?: FALSE
> >>
> >> Temporary pathname:
> >>
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
> >>
> >> Adding temporary suffix from file...done
> >>
> >> Creating CEL file for results, if missing...
> >>
> >> Creating CEL file for results, if missing...done
> >>
> >> Writing adjusted intensities...
> >>
> >> Writing adjusted intensities...done
> >>
> >> Dropping temporary suffix from file...
> >>
> >> Temporary pathname:
> >>
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
> >>
> >> Suffix: .tmp
> >>
> >> Regular expression for suffix: \.tmp$
> >>
> >> Pathname:
> >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
> >>
> >> Renaming existing file...
> >>
> >> Result: TRUE
> >>
> >> Renaming existing file...done
> >>
> >> Dropping temporary suffix from file...done
> >>
> >> Writing adjusted probe signals...done
> >>
> >> used (Mb) gc trigger (Mb) max used (Mb)
> >>
> >> Ncells 709078 37.9 1166886 62.4 1166886 62.4
> >>
> >> Vcells 31586985 241.0 47054688 359.0 46991036 358.6
> >>
> >> AffymetrixCelFile:
> >>
> >> Name: GSM709330_HSB194-S1C-L
> >>
> >> Tags:
> >>
> >> Full name: GSM709330_HSB194-S1C-L
> >>
> >> Pathname:
> >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
> >>
> >> File size: 62.73 MB (65775403 bytes)
> >>
> >> RAM: 0.00 MB
> >>
> >> File format: v4 (binary; XDA)
> >>
> >> Platform: Affymetrix
> >>
> >> Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
> >>
> >> Timestamp: 2011-01-25 12:15:03
> >>
> >> Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
> >>
> >>
> >> -----
> >>
> >>
> >> There must be a file permissions transfer behavior that takes place on
> a
> >> multi-user cluster environment that doesn't take place on other types
> of
> >> machines, because I was unable to reproduce this error elsewhere. I
> didn't
> >> bother changing the raw .CEL files' permissions because I never
> experienced
> >> a problem of this nature with .CEL files.
> >>
> >>
> >> Let me know if you patch this up so I can pull the latest code from the
> >> aroma repo.
> >>
> >>
> >> Thanks for your help with this and for taking a look at the issue so
> >> quickly.
> >>
> >>
> >> Best regards,
> >>
> >>
> >> Taylor
> >>
> >>
> >>
> >> On Thursday, September 4, 2014 1:37:36 PM UTC-4, Henrik Bengtsson
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> interesting. A quick guess is that the input file, i.e.
> >>>
> >>> rawData/NIK_2014/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
> >>>
> >>> is write protected (e.g. file owned by someone else). This could be
> >>> the reason, because internally that file is copied and used as a
> >>> template and the copy may inherit the file permissions. What does ls
> >>> -l say about the above file?
> >>>
> >>> If this is the case, I need to update the copying such that the new
> >>> file has proper permissions.
> >>>
> >>> Thanks for the report
> >>>
> >>> Henrik
> >>>
> >>>
> >>> On Thu, Sep 4, 2014 at 10:15 AM, Taylor Raborn <taylor...@gmail.com>
> >>> wrote:
> >>> > Hi Henrik:
> >>> >
> >>> > I just come across an unusual error when using aroma.affymetrix on
> our
> >>> > high-memory HPC machine. It may or may not be platform-specific, and
> >>> > I'll
> >>> > give you all the information I can in the hopes of isolating and
> >>> > identifying
> >>> > the error.
> >>> >
> >>> > Traceback:
> >>> >
> >>> >> traceback()
> >>> >
> >>> > 18: stop(cond)
> >>> >
> >>> > 17: throw.Exception(Exception(...))
> >>> >
> >>> > 16: throw(Exception(...))
> >>> >
> >>> > 15: throw.default("No permission to modify existing file: ",
> pathname)
> >>> >
> >>> > 14: throw("No permission to modify existing file: ", pathname)
> >>> >
> >>> > 13: getWritablePathname.Arguments(static, ...)
> >>> >
> >>> > 12: getWritablePathname(static, ...) at <text>#1
> >>> >
> >>> > 11: Arguments$getWritablePathname(pathname, mustExist = TRUE)
> >>> >
> >>> > 10: renameFile.default(srcPathname, pathname, ...)
> >>> >
> >>> > 9: renameFile(srcPathname, pathname, ...)
> >>> >
> >>> > 8: renameTo.GenericDataFile(res, filename = pathname, verbose =
> >>> > less(verbose))
> >>> >
> >>> > 7: renameTo(res, filename = pathname, verbose = less(verbose))
> >>> >
> >>> > 6: createFrom.AffymetrixCelFile(this, filename = pathnameT, path =
> NULL,
> >>> >
> >>> > verbose = less(verbose))
> >>> >
> >>> > 5: createFrom(this, filename = pathnameT, path = NULL, verbose =
> >>> > less(verbose))
> >>> >
> >>> > 4: bgAdjustRma.AffymetrixCelFile(df, path = outputPath, pmonly =
> pmonly,
> >>> >
> >>> > addJitter = addJitter, jitterSd = jitterSd, overwrite =
> force,
> >>> >
> >>> > verbose = verbose, .deprecated = FALSE)
> >>> >
> >>> > 3: bgAdjustRma(df, path = outputPath, pmonly = pmonly, addJitter =
> >>> > addJitter,
> >>> >
> >>> > jitterSd = jitterSd, overwrite = force, verbose = verbose,
> >>> >
> >>> > .deprecated = FALSE)
> >>> >
> >>> > 2: process.RmaBackgroundCorrection(bc, verbose = verbose)
> >>> >
> >>> > 1: process(bc, verbose = verbose)
> >>> >
> >>> >
> >>> > ----
> >>> >
> >>> > sessionInfo:
> >>> >
> >>> >> sessionInfo()
> >>> >
> >>> > R version 3.0.1 (2013-05-16)
> >>> >
> >>> > Platform: x86_64-unknown-linux-gnu (64-bit)
> >>> >
> >>> >
> >>> > locale:
> >>> >
> >>> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> >>> >
> >>> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
> >>> >
> >>> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
> >>> >
> >>> > [7] LC_PAPER=C LC_NAME=C
> >>> >
> >>> > [9] LC_ADDRESS=C LC_TELEPHONE=C
> >>> >
> >>> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >>> >
> >>> >
> >>> > attached base packages:
> >>> >
> >>> > [1] tools parallel stats graphics grDevices utils
> datasets
> >>> >
> >>> > [8] methods base
> >>> >
> >>> >
> >>> > other attached packages:
> >>> >
> >>> > [1] R.cache_0.10.0 base64enc_0.1-2 aroma.apd_0.5.0
> >>> >
> >>> > [4] preprocessCore_1.24.0 affyio_1.30.0 Biobase_2.20.1
> >>> >
> >>> > [7] BiocGenerics_0.6.0 aroma.light_1.32.0
> matrixStats_0.10.0
> >>> >
> >>> > [10] aroma.affymetrix_2.12.0 aroma.core_2.12.1
> R.devices_2.11.0
> >>> >
> >>> > [13] R.filesets_2.6.0 R.utils_1.33.0 R.oo_1.18.0
> >>> >
> >>> > [16] affxparser_1.34.2 affy_1.40.0
> R.methodsS3_1.6.1
> >>> >
> >>> > [19] PSCBS_0.43.0 DNAcopy_1.36.0
> >>> > BiocInstaller_1.12.1
> >>> >
> >>> >
> >>> > loaded via a namespace (and not attached):
> >>> >
> >>> > [1] digest_0.6.4 R.huge_0.8.0 R.rsp_0.19.0 zlibbioc_1.11.1
> >>> >
> >>> >
> >>> >
> >>> > Here's how I produced the error. Note that my directory structure
> >>> > matches
> >>> > that on the aroma-project.org site, shown here:
> >>> >
> >>> >> chipType <- "HuEx-1_0-st-v2"
> >>> >
> >>> >
> >>> >> cdf <- AffymetrixCdfFile$byChipType(chipType,
> >>> >> tags="coreR3,A20071112,EP")
> >>> >
> >>> >
> >>> >> print(cdf)
> >>> >
> >>> > AffymetrixCdfFile:
> >>> >
> >>> > Path: annotationData/chipTypes/HuEx-1_0-st-v2
> >>> >
> >>> > Filename: HuEx-1_0-st-v2,coreR3,A20071112,EP.cdf
> >>> >
> >>> > File size: 38.25 MB (40108891 bytes)
> >>> >
> >>> > Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
> >>> >
> >>> > RAM: 0.00MB
> >>> >
> >>> > File format: v4 (binary; XDA)
> >>> >
> >>> > Dimension: 2560x2560
> >>> >
> >>> > Number of cells: 6553600
> >>> >
> >>> > Number of units: 18708
> >>> >
> >>> > Cells per unit: 350.31
> >>> >
> >>> > Number of QC units: 1
> >>> >
> >>> >
> >>> >> cs <- AffymetrixCelSet$byName("NIK_2014", cdf=cdf)
> >>> >
> >>> >
> >>> >> print(cs)
> >>> >
> >>> > AffymetrixCelSet:
> >>> >
> >>> > Name: NIK_2014
> >>> >
> >>> > Tags:
> >>> >
> >>> > Path: rawData/NIK_2014/HuEx-1_0-st-v2
> >>> >
> >>> > Platform: Affymetrix
> >>> >
> >>> > Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP
> >>> >
> >>> > Number of arrays: 9
> >>> >
> >>> > Names: GSM709330_HSB194-S1C-L, GSM709331_HSB194-S1C-R,
> >>> > GSM709332_HSB194-STC-L, ..., GSM709338_HSB194-VFC-R [9]
> >>> >
> >>> > Time period: 2011-01-25 12:15:03 -- 2011-01-25 13:48:02
> >>> >
> >>> > Total file size: 564.87MB
> >>> >
> >>> > RAM: 0.02MB
> >>> >
> >>> >
> >>> >> csBC <- process(bc,verbose=verbose)
> >>> >
> >>> > Background correcting data set...
> >>> >
> >>> > Number of arrays: 9
> >>> >
> >>> > Array #1 ('GSM709330_HSB194-S1C-L') of 9...
> >>> >
> >>> > Adjusting PM signals only
> >>> >
> >>> > Obtaining signals...
> >>> >
> >>> > Obtaining signals...done
> >>> >
> >>> > Applying normal+exponential signal model...
> >>> >
> >>> > Applying normal+exponential signal model...done
> >>> >
> >>> > Writing adjusted probe signals...
> >>> >
> >>> > Adding temporary suffix from file...
> >>> >
> >>> > Pathname:
> >>> >
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL
> >>> >
> >>> > Suffix: .tmp
> >>> >
> >>> > Rename existing file?: FALSE
> >>> >
> >>> > Temporary pathname:
> >>> >
> >>> >
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp
> >>> >
> >>> > Adding temporary suffix from file...done
> >>> >
> >>> > Creating CEL file for results, if missing...
> >>> >
> >>> > [2014-09-04 12:38:54] Exception: No permission to modify existing
> file:
> >>> >
> >>> >
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp.tmp
>
>
> >>> >
> >>> >
> >>> > at #13. getWritablePathname.Arguments(static, ...)
> >>> >
> >>> > - getWritablePathname.Arguments() is in environment
> 'R.utils'
> >>> >
> >>> >
> >>> > at #12. getWritablePathname(static, ...)
> >>> >
> >>> > - getWritablePathname() is in environment 'R.utils'
> >>> >
> >>> > - originating from '<text>'
> >>> >
> >>> >
> >>> > at #11. Arguments$getWritablePathname(pathname, mustExist = TRUE)
> >>> >
> >>> > - Arguments$getWritablePathname() is local of the calling
> >>> > function
> >>> >
> >>> >
> >>> > at #10. renameFile.default(srcPathname, pathname, ...)
> >>> >
> >>> > - renameFile.default() is in environment 'R.utils'
> >>> >
> >>> >
> >>> > at #09. renameFile(srcPathname, pathname, ...)
> >>> >
> >>> > - renameFile() is in environment 'R.utils'
> >>> >
> >>> >
> >>> > at #08. renameTo.GenericDataFile(res, filename = pathname, verbose
> =
> >>> > less(verbose))
> >>> >
> >>> > - renameTo.GenericDataFile() is in environment
> 'R.filesets'
> >>> >
> >>> >
> >>> > at #07. renameTo(res, filename = pathname, verbose =
> less(verbose))
> >>> >
> >>> > - renameTo() is in environment 'R.filesets'
> >>> >
> >>> >
> >>> > at #06. createFrom.AffymetrixCelFile(this, filename = pathnameT,
> path
> >>> > =
> >>> > NULL,
> >>> >
> >>> > verbose = less(verbose))
> >>> >
> >>> > - createFrom.AffymetrixCelFile() is in environment
> >>> > 'aroma.affymetrix'
> >>> >
> >>> >
> >>> > at #05. createFrom(this, filename = pathnameT, path = NULL,
> verbose =
> >>> > less(verbose))
> >>> >
> >>> > - createFrom() is in environment 'aroma.affymetrix'
> >>> >
> >>> >
> >>> > at #04. bgAdjustRma.AffymetrixCelFile(df, path = outputPath,
> pmonly =
> >>> > pmonly,
> >>> >
> >>> > addJitter = addJitter, jitterSd = jitterSd, overwrite
> =
> >>> > force,
> >>> >
> >>> > verbose = verbose, .deprecated = FALSE)
> >>> >
> >>> > - bgAdjustRma.AffymetrixCelFile() is in environment
> >>> > 'aroma.affymetrix'
> >>> >
> >>> >
> >>> > at #03. bgAdjustRma(df, path = outputPath, pmonly = pmonly,
> addJitter
> >>> > =
> >>> > addJitter,
> >>> >
> >>> > jitterSd = jitterSd, overwrite = force, verbose =
> verbose,
> >>> >
> >>> > .deprecated = FALSE)
> >>> >
> >>> > - bgAdjustRma() is in environment 'aroma.affymetrix'
> >>> >
> >>> >
> >>> > at #02. process.RmaBackgroundCorrection(bc, verbose = verbose)
> >>> >
> >>> > - process.RmaBackgroundCorrection() is in environment
> >>> > 'aroma.affymetrix'
> >>> >
> >>> >
> >>> > at #01. process(bc, verbose = verbose)
> >>> >
> >>> > - process() is in environment 'aroma.core'
> >>> >
> >>> >
> >>> > Error: No permission to modify existing file:
> >>> >
> >>> >
> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp.tmp
>
>
> >>> >
> >>> > Creating CEL file for results, if missing...done
> >>> >
> >>> > Writing adjusted probe signals...done
> >>> >
> >>> > Array #1 ('GSM709330_HSB194-S1C-L') of 9...done
> >>> >
> >>> > Background correcting data set...done
> >>> >
> >>> > --------------------
> >>> >
> >>> >
> >>> > I tried to find a solution from the group archives and other places
> to
> >>> > no
> >>> > avail. It's worth mentioning that I get this problem even if I open
> up
> >>> > permissions to the entire directory. The *CEL.tmp.tmp file appears
> to be
> >>> > written with write-protections, which is baffling to me. The machine
> is
> >>> > a
> >>> > large, high-memory cluster from my institution, and the error may be
> >>> > associated with this.
> >>> >
> >>> >
> >>> > Thanks in advance for any insight you can provide with this.
> >>> >
> >>> >
> >>> > Best regards,
> >>> >
> >>> >
> >>> > Taylor
> >>> >
> >>> >
> >>> > --
> >>> > --
> >>> > When reporting problems on aroma.affymetrix, make sure 1) to run the
> >>> > latest
> >>> > version of the package, 2) to report the output of sessionInfo() and
> >>> > traceback(), and 3) to post a complete code example.
> >>> >
> >>> >
> >>> > You received this message because you are subscribed to the Google
> >>> > Groups
> >>> > "aroma.affymetrix" group with website http://www.aroma-project.org/.
>
> >>> > To post to this group, send email to aroma-af...@googlegroups.com
> >>> > To unsubscribe and other options, go to
> >>> > http://www.aroma-project.org/forum/
> >>> >
> >>> > ---
> >>> > You received this message because you are subscribed to the Google
> >>> > Groups
> >>> > "aroma.affymetrix" group.
> >>> > To unsubscribe from this group and stop receiving emails from it,
> send
> >>> > an
> >>> > email to aroma-affymetr...@googlegroups.com.
> >>> > For more options, visit https://groups.google.com/d/optout.
> >>
> >> --
> >> --
> >> When reporting problems on aroma.affymetrix, make sure 1) to run the
> latest
> >> version of the package, 2) to report the output of sessionInfo() and
> >> traceback(), and 3) to post a complete code example.
> >>
> >>
> >> You received this message because you are subscribed to the Google
> Groups
> >> "aroma.affymetrix" group with website http://www.aroma-project.org/.
> >> To post to this group, send email to aroma-af...@googlegroups.com
> <javascript:>
> >> To unsubscribe and other options, go to
> http://www.aroma-project.org/forum/
> >>
> >> ---
> >> You received this message because you are subscribed to the Google
> Groups
> >> "aroma.affymetrix" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to aroma-affymetr...@googlegroups.com <javascript:>.
> >> For more options, visit https://groups.google.com/d/optout.
>
--
--
When reporting problems on aroma.affymetrix, make sure 1) to run the latest
version of the package, 2) to report the output of sessionInfo() and
traceback(), and 3) to post a complete code example.
You received this message because you are subscribed to the Google Groups
"aroma.affymetrix" group with website http://www.aroma-project.org/.
To post to this group, send email to aroma-affymetrix@googlegroups.com
To unsubscribe and other options, go to http://www.aroma-project.org/forum/
---
You received this message because you are subscribed to the Google Groups
"aroma.affymetrix" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to aroma-affymetrix+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
