Hi Christian.
On Wed, Jul 21, 2010 at 2:59 PM, cstratowa
<christian.strat...@vie.boehringer-ingelheim.com> wrote:
> Dear Henrik,
>
> Thank you for this extensive explanation and sorry for the late reply
> but I was pretty busy.
>
> Yes, it did work before! As I mentioned with versions
> aroma.affymetrix_1.1.0 and earlier I have never had a problem doing
> the analyses on cluster nodes.
>
> Looking at the source code of different versions of saveObject() I
> realize that using "saveObject(..,safe=FALSE)" would be the same as
> using saveObject() from R.utils_0.9.1. Thus in principle this could
> solve my problem. Is this correct?
>
> Sadly, method AffymetrixCelSet::getAverageFile() in
> aroma.affymetrix_1.6.2 does not allow to pass parameter "safe=FALSE"
> to saveObject(). Is it possible for you to change it?
I have decided to remove that debug code that calls saveObject(),
because it is not really needed anymore. The main reason why I remove
it is because it is obsolete code. The intention of that code snippet
in getAverageFile() was never to protect against race conditions (it
was just an unplanned side effect).
Until next release, you can get a patched version as:
library("aroma.affymetrix");
downloadPackagePatch("aroma.affymetrix");
Note, as I said in my previous reply, by processing (=here calling
getAverageFile() on) the same data set on multiple hosts, you are
potentially running into race conditions resulting in corrupt data.
You should at least be aware of it and understand why this is the
case.
>
> It is still not clear to me why you create first a temporary file
> which you then rename (although you mention power failures etc).
> However, would it be possible to add a random number to the temporary
> filename, e.g. "*.tmp.1948234", so that the problem with the existing
> temporary file could be avoided?
The main purpose of writing to a temporary file and then renaming is
to make sure that the file is complete. If something happens while
writing the temporary file, the final file will not exist/be created.
If one would write to the final file from the beginning, there is no
way for us to know if the file was correctly created or not. So,
writing via a temporary file, we effectively have a way of creating
files in one atomic action.
>
> Probably you only need to change line 59 to:
>
> pathnameT <- sprintf("%s.tmp.%i", pathname,
> as.integer(runif(1,1,99999999)))
In order not to corrupt the temporary file, we check if it already
exist as a protection for being overwritten/added to by another
process. Yes, you could randomize the name of the temporary file,
lowering the risk of two hosts writing to the same temporary file.
However, when done, both hosts will try to rename their temporary
files to the same pathname. If done at the same time, we still may
have problems.
>
> Regarding your suggestion to wrap getAverageFile() in Mutex calls I
> have no idea if there exists an R-package for this purpose. Neither
> Rmpi nor snow seem to be suitable for this purpose (at least not
> without a complete re-write of my package).
Yes, I neither know of a functional mutex implementation in R. You
can achieve some by utilizing the lock mechanisms of data base servers
(not SqlLite), but nothing ready is available to my knowledge.
Again, you seem to assume that aroma.* takes care of potential race
conditions for you - it does not. It only tries to detect them
without warranty - and indeed, the reason why got the error in the
first place indicates that you are pushing the system and that race
conditions may very well happen. If you run things in parallel and
you are updating/writing the *same data resource*, you should really
have protection against race conditions. This is a generic problem
unrelated to aroma.*.
/Henrik
>
> One other question:
> Is it allowed to delete the contents of directory .Rcache/
> aroma.affymetrix/idChecks?
Yes, it should be safe to delete any .Rcache/ as long as no R session
is in the process of writing to it. It's a cache containing redundant
information.
>
> Best regards
> Christian
>
> On Jul 2, 12:47 am, Henrik Bengtsson <h...@stat.berkeley.edu> wrote:
>
>
>
> > Hi Christian.
>
> > On Tue, Jun 29, 2010 at 3:39 PM, cstratowa
>
> > <christian.strat...@vie.boehringer-ingelheim.com> wrote:
> > > Dear Henrik,
>
> > > Until now I have used aroma.affymetrix_1.1.0 with R-2.8.1 and could
> > > run my analysis on our sge-cluster w/o any problems.
>
> > > Now I have upgraded to R-2.11.1 and to aroma.affymetrix_1.6.2 and are
> > > curently testing with 8 chips whether my package based on
> > > aroma.affymetrix still works on the cluster. The normalization step on
> > > a server did run fine, howeever, distributing the 8 samples on the
> > > cluster to run GladModel() resulted in the problem that 3 of 8 cluster
> > > nodes did stop with the following error message:
>
> > > Loading required package: GLAD
> > > ...
> > > Loading required package: RColorBrewer
> > > Loading required package: Cairo
> > > Error in list(`computeCN(aroma, model = model, arrays = arrays[i],
> > > chromosomes = 1:23, ref` = <environment>, :
>
> > > [2010-06-29 15:08:49] Exception: Cannot save to file. Temporary file
> > > already exists: ~/.Rcache/aroma.affymetrix/idChecks/
> > > a1c33926939ee43fbed83ae69301d215.tmp
> > > at throw(Exception(...))
> > > at throw.default("Cannot save to file. Temporary file already
> > > exists: ", pathn
> > > at throw("Cannot save to file. Temporary file already exists: ",
> > > pathnameT)
> > > at saveObject.default(list(key = key, keyIds = lapply(key, digest2),
> > > id = id),
> > > at saveObject(list(key = key, keyIds = lapply(key, digest2), id =
> > > id), idPathn
> > > at getAverageFile.AffymetrixCelSet(ces, force = force, verbose =
> > > less(verbose)
> > > at NextMethod(generic = "getAverageFile", object = this, indices =
> > > indices, ..
> > > at getAverageFile.ChipEffectSet(ces, force = force, verbose =
> > > less(verbose))
> > > at NextMethod(generic = "getAverageFile", object = this, ...)
> > > at getAverageFile.SnpChipEffectSet(ces, force = force, verbose =
> > > less(verbose)
> > > at NextMethod(generic = "getAverageFile", object = this, ...)
> > > at getAverageFile.CnChipEffectS
> > > Calls: computeCN ... saveObject.default -> throw -> throw.default ->
> > > throw -> throw.Exception
> > > Execution halted
>
> > > Interestingly, on the other 5 nodes GladModel() seems to run fine.
>
> > > Do you have any idea what the reason for this problem might be?
>
> > This seems to be due to a race condition, because several processes
> > calls getAverageFile() on the same data set (set of data files). It
> > has nothing to do with the GladModel - that is only calling
> > getAverageFile() in order to calculate the average signal across all
> > samples in the data set.
>
> > More precisely, in this particular case it is saveObject() of R.utils
> > that detects that there already exist a temporary file (added file
> > name extension *.tmp) that is currently being created and written to
> > by another process. This temporary file is renamed to its final name
> > when done. The reason why didn't observe it before is most likely
> > because this additional feature was added to saveObject() in R.utils
> > v1.2.4:
>
> > Version: 1.2.4 [2009-10-30]
> > o ROBUSTIFICATION: Lowered the risk for saveObject() to leave an
> > imcomplete file due to say power failures etc. This is done by
> > first writing to a temporary file, which is then renamed. If the
> > temporary file already exists, an exception is thrown.
>
> > Ok, that's the details explaining the error message and the traceback
> > you report.
>
> > So, did it work before? Did you get valid estimates? Probably,
> > because the way getAverageFile() is written it is unlikely that a
> > corrupt result file is created. For sure is that the calculations
> > where done multiple times if there were race conditions.
>
> > I'd like to put out a little disclaimer that although I try write
> > methods so that they work even when there are race conditions.
> > However, as you've noticed, I am also very conservative, that is, I
> > rather detect the race condition and throw an exception, than silently
> > ignore it. Then plan is to loosen this up in the future. I just like
> > to say this here so that you understand my current design
> > decisions/plans.
>
> > I have to think about this particular case, because I could loosen up
> > getAverageFile() a bit, I think. However, at the moment it is better
> > if you take care of the race conditions yourself. Assume you current
> > code looks something like this:
>
> > fln <- FragmentLengthNormalization(ces);
> > cesN <- process(fln);
> > seg <- GladModel(cesN);
> > process(seg);
>
> > Then first you should know that the latter two lines are
> > computationally identical to [it is only slightly more complicated if
> > you use chip type pairs]:
>
> > ceR <- getAverageFile(cesN);
> > seg <- GladModel(cesN, ceR);
> > process(seg);
>
> > So, if you can synchronize the averaging by (conceptually only):
>
> > mutex <- waitForMutex("foo");
> > ceR <- getAverageFile(cesN);
> > releaseMutex(mutex);
>
> > then it should all be fine. Replace waitForMutex()/releaseMutex()
> > with your favorite synchronization mechanism. FYI, if there would be
> > a cross-platform bullet proof and generic synchronization mechanism in
> > R, I would internally add synchronization to lots of methods.
>
> > Hope this helps(?)
>
> > Henrik
>
> > >> sessionInfo()
> > > R version 2.11.1 (2010-05-31)
> > > x86_64-unknown-linux-gnu
>
> > > locale:
> > > [1] C
>
> > > attached base packages:
> > > [1] stats graphics grDevices utils datasets methods
> > > base
>
> > > other attached packages:
> > > [1] Cairo_1.4-5 RColorBrewer_1.0-2
> > > GLAD_2.10.0
> > > [4] biasnp_0.2.54 RODBC_1.3-1
> > > aroma.affymetrix_1.6.2
> > > [7] aroma.apd_0.1.7 affxparser_1.20.0
> > > R.huge_0.2.0
> > > [10] aroma.core_1.6.2 aroma.light_1.16.0
> > > matrixStats_0.2.1
> > > [13] R.rsp_0.3.6 R.cache_0.3.0
> > > R.filesets_0.8.2
> > > [16] digest_0.4.2 R.utils_1.4.2
> > > R.oo_1.7.3
> > > [19] R.methodsS3_1.2.0
>
> > > loaded via a namespace (and not attached):
> > > [1] tools_2.11.1
> > > Warning message:
> > > 'DESCRIPTION' file has 'Encoding' field and re-encoding is not
> > > possible
>
> > > Best regards
> > > Christian
>
> > > P.S.:
> > > Is Google groups still the place to post questions?
>
> > > --
> > > When reporting problems on aroma.affymetrix, make sure 1) to run the
> > > latest version of the package, 2) to report the output of sessionInfo()
> > > and traceback(), and 3) to post a complete code example.
>
> > > You received this message because you are subscribed to the Google Groups
> > > "aroma.affymetrix" group with websitehttp://www.aroma-project.org/.
> > > To post to this group, send email to aroma-affymetrix@googlegroups.com
> > > To unsubscribe and other options, go tohttp://www.aroma-project.org/forum/
--
When reporting problems on aroma.affymetrix, make sure 1) to run the latest
version of the package, 2) to report the output of sessionInfo() and
traceback(), and 3) to post a complete code example.
You received this message because you are subscribed to the Google Groups
"aroma.affymetrix" group with website http://www.aroma-project.org/.
To post to this group, send email to aroma-affymetrix@googlegroups.com
To unsubscribe and other options, go to http://www.aroma-project.org/forum/
