Re: [Bioc-devel] Error in HDF5 - Package submission - Not detected locally
Hi Tiago,
On 9/16/19 06:35, Tiago Lubiana Alves wrote:
> Hello Mike,
>
> Thank you for the detailed explanation.
>
> You are right, for the vignette, I can download it from the ExperimentHub
> subset the pbmc3k dataset in the first few lines. The main point of having
> a new dataset was to use it in the examples of functions. The full dataset
> is too big and the examples would take too long otherwise.
>
> Still, if someone knows how to "transform from an HDF5 back
> SingleCellExperiment to using the data in memory", it would be of great
> help.
Just to clarify, it's not that you have to choose between using the full
on-disk dataset or using a small in-memory subset in your vignette. A
3rd option is to start your vignette with the full on-disk dataset and
to immediately subset it. Like what Pete is doing in his 3-line snippet
but without the last line (replacement of on-disk assay with in-memory
assay).
H.
> Best,
> Tiago
>
>
> Tiago Lubiana
> Mestrando em Bioinformática, FCF/USP
> *Computational Systems Biology Laboratory (CSBL)*
> Telefone (laboratório): +55 (11) 2648-0240
> Telefone (pessoal): +55 (11) 954258000
>
>
> On Mon, Sep 16, 2019 at 5:39 AM Mike Smith wrote:
>
>> Hi Tiago,
>>
>> I suspect what has happened here is that when create the mini_pbmc3k
>> object, you're doing this by subsetting the PBMC, 3k scRNA-seq data from
>> ExperimenHub. The assay data for that are are actually stored in an HDF5
>> file which will be downloaded and stored in your ExperimentHub cache on
>> your local machine. When you take the subset for your new object it still
>> points to the same HDF5 file, and when you save mini_pbmc3k all it actually
>> saves is the location of the HDF5 rather than putting the data in the
>> object. This works locally, but naturally the BioC build server doesn't
>> have the original HDF5 in exactly the same location, and so it fails.
>>
>> Is there a reason you don't want to use the whole dataset and make the
>> first section of the vignette demonstrate how to download it via
>> ExperimentHub? Alternatively there is a way to transform from an HDF5 back
>> SingleCellExperiment to using the data in memory, but I can't remember it
>> right now - hopefully someone else will be along shortly.
>>
>> This also seems like the perfect opportunity to point out that our next
>> BioC Developers' Forum (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_pipermail_bioc-2Ddevel_2019-2DSeptember_015499.html=DwIFaQ=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo=TDo6rxwdtAE-kwlyPoYbbHn9JI69Wdw3scAgxje60ck=
>> ) will
>> have a discussion on object serilaisation which should cover exactly this
>> type of issue.
>>
>> Best,
>> Mike
>>
>>
>>
>> On Sat, 14 Sep 2019 at 23:26, Tiago Lubiana Alves <
>> tiago.lubiana.al...@usp.br> wrote:
>>
>>> Hello,
>>>
>>> I am having a problem with a package submission build.
>>>
>>> This is the package issue:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Bioconductor_Contributions_issues_1241=DwIFaQ=eRAMFD45gAfqt84VtBcfhQ=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA=s7A_Q0fNE8fpu3Ob4M_wxwPfLdlY2QkGSbNwInWB2mo=HRcbdQHP2-fSp-0NnFZw8H0NrGgT8IerVORN96IaUrs=
>>> And this is the ERROR:
>>>
>>> HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
>>>#000: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5F.c line 509
>>> in H5Fopen(): unable to open file
>>> major: File accessibilty
>>> minor: Unable to open file
>>>#001: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5Fint.c line
>>> 1498 in H5F_open(): unable to open file: time = Fri Sep 13 15:05:35
>>> 2019
>>> , name = '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605',
>>> tent_flags = 0
>>> (...)
>>>
>>> Quitting from lines 46-51 (fcoex.Rmd)
>>> Error: processing vignette 'fcoex.Rmd' failed with diagnostics:
>>> failed to open file '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605'
>>> --- failed re-building 'fcoex.Rmd'
>>>
>>>
>>> It looks like something is pointing in the wrong direction, but I have not
>>> been able to figure out exactly what. I've tried re-saving the data file
>>> ("mini_pbmc3k"), which is loaded in the vignette chunk of the error, but
>>> that did not help.
>>>
>>> Does anyone have a suggestion of what might be happening?
>>>
>>> Thank you very much for your time,
>>> Best,
>>> Tiago
>>>
>>> Tiago Lubiana
>>> Mestrando em Bioinformática, FCF/USP
>>> *Computational Systems Biology Laboratory (CSBL)*
>>> Telefone (laboratório): +55 (11) 2648-0240
>>> Telefone (pessoal): +55 (11) 954258000
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ___
>>> Bioc-devel@r-project.org mailing list
>>>
Re: [Bioc-devel] Error in HDF5 - Package submission - Not detected locally
Hi Tiago,
The following will create a similarly sized subset of the PBMC3k
dataset with the counts in-memory as a sparse matrix:
pbmc3k <- TENxPBMCData::TENxPBMCData("pbmc3k")
mini_pbmc3k <- pbmc3k[1:1700, 1:600]
assay(mini_pbmc3k) <- as(assay(mini_pbmc3k), "dgCMatrix")
Cheers,
Pete
On Mon, 16 Sep 2019 at 23:38, Tiago Lubiana Alves
wrote:
>
> Hello Mike,
>
> Thank you for the detailed explanation.
>
> You are right, for the vignette, I can download it from the ExperimentHub
> subset the pbmc3k dataset in the first few lines. The main point of having
> a new dataset was to use it in the examples of functions. The full dataset
> is too big and the examples would take too long otherwise.
>
> Still, if someone knows how to "transform from an HDF5 back
> SingleCellExperiment to using the data in memory", it would be of great
> help.
> Best,
> Tiago
>
>
> Tiago Lubiana
> Mestrando em Bioinformática, FCF/USP
> *Computational Systems Biology Laboratory (CSBL)*
> Telefone (laboratório): +55 (11) 2648-0240
> Telefone (pessoal): +55 (11) 954258000
>
>
> On Mon, Sep 16, 2019 at 5:39 AM Mike Smith wrote:
>
> > Hi Tiago,
> >
> > I suspect what has happened here is that when create the mini_pbmc3k
> > object, you're doing this by subsetting the PBMC, 3k scRNA-seq data from
> > ExperimenHub. The assay data for that are are actually stored in an HDF5
> > file which will be downloaded and stored in your ExperimentHub cache on
> > your local machine. When you take the subset for your new object it still
> > points to the same HDF5 file, and when you save mini_pbmc3k all it actually
> > saves is the location of the HDF5 rather than putting the data in the
> > object. This works locally, but naturally the BioC build server doesn't
> > have the original HDF5 in exactly the same location, and so it fails.
> >
> > Is there a reason you don't want to use the whole dataset and make the
> > first section of the vignette demonstrate how to download it via
> > ExperimentHub? Alternatively there is a way to transform from an HDF5 back
> > SingleCellExperiment to using the data in memory, but I can't remember it
> > right now - hopefully someone else will be along shortly.
> >
> > This also seems like the perfect opportunity to point out that our next
> > BioC Developers' Forum (
> > https://stat.ethz.ch/pipermail/bioc-devel/2019-September/015499.html) will
> > have a discussion on object serilaisation which should cover exactly this
> > type of issue.
> >
> > Best,
> > Mike
> >
> >
> >
> > On Sat, 14 Sep 2019 at 23:26, Tiago Lubiana Alves <
> > tiago.lubiana.al...@usp.br> wrote:
> >
> >> Hello,
> >>
> >> I am having a problem with a package submission build.
> >>
> >> This is the package issue:
> >> https://github.com/Bioconductor/Contributions/issues/1241
> >> And this is the ERROR:
> >>
> >> HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
> >> #000: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5F.c line 509
> >> in H5Fopen(): unable to open file
> >> major: File accessibilty
> >> minor: Unable to open file
> >> #001: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5Fint.c line
> >> 1498 in H5F_open(): unable to open file: time = Fri Sep 13 15:05:35
> >> 2019
> >> , name = '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605',
> >> tent_flags = 0
> >> (...)
> >>
> >> Quitting from lines 46-51 (fcoex.Rmd)
> >> Error: processing vignette 'fcoex.Rmd' failed with diagnostics:
> >> failed to open file '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605'
> >> --- failed re-building 'fcoex.Rmd'
> >>
> >>
> >> It looks like something is pointing in the wrong direction, but I have not
> >> been able to figure out exactly what. I've tried re-saving the data file
> >> ("mini_pbmc3k"), which is loaded in the vignette chunk of the error, but
> >> that did not help.
> >>
> >> Does anyone have a suggestion of what might be happening?
> >>
> >> Thank you very much for your time,
> >> Best,
> >> Tiago
> >>
> >> Tiago Lubiana
> >> Mestrando em Bioinformática, FCF/USP
> >> *Computational Systems Biology Laboratory (CSBL)*
> >> Telefone (laboratório): +55 (11) 2648-0240
> >> Telefone (pessoal): +55 (11) 954258000
> >>
> >> [[alternative HTML version deleted]]
> >>
> >> ___
> >> Bioc-devel@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >>
> >
>
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Error in HDF5 - Package submission - Not detected locally
Hello Mike,
Thank you for the detailed explanation.
You are right, for the vignette, I can download it from the ExperimentHub
subset the pbmc3k dataset in the first few lines. The main point of having
a new dataset was to use it in the examples of functions. The full dataset
is too big and the examples would take too long otherwise.
Still, if someone knows how to "transform from an HDF5 back
SingleCellExperiment to using the data in memory", it would be of great
help.
Best,
Tiago
Tiago Lubiana
Mestrando em Bioinformática, FCF/USP
*Computational Systems Biology Laboratory (CSBL)*
Telefone (laboratório): +55 (11) 2648-0240
Telefone (pessoal): +55 (11) 954258000
On Mon, Sep 16, 2019 at 5:39 AM Mike Smith wrote:
> Hi Tiago,
>
> I suspect what has happened here is that when create the mini_pbmc3k
> object, you're doing this by subsetting the PBMC, 3k scRNA-seq data from
> ExperimenHub. The assay data for that are are actually stored in an HDF5
> file which will be downloaded and stored in your ExperimentHub cache on
> your local machine. When you take the subset for your new object it still
> points to the same HDF5 file, and when you save mini_pbmc3k all it actually
> saves is the location of the HDF5 rather than putting the data in the
> object. This works locally, but naturally the BioC build server doesn't
> have the original HDF5 in exactly the same location, and so it fails.
>
> Is there a reason you don't want to use the whole dataset and make the
> first section of the vignette demonstrate how to download it via
> ExperimentHub? Alternatively there is a way to transform from an HDF5 back
> SingleCellExperiment to using the data in memory, but I can't remember it
> right now - hopefully someone else will be along shortly.
>
> This also seems like the perfect opportunity to point out that our next
> BioC Developers' Forum (
> https://stat.ethz.ch/pipermail/bioc-devel/2019-September/015499.html) will
> have a discussion on object serilaisation which should cover exactly this
> type of issue.
>
> Best,
> Mike
>
>
>
> On Sat, 14 Sep 2019 at 23:26, Tiago Lubiana Alves <
> tiago.lubiana.al...@usp.br> wrote:
>
>> Hello,
>>
>> I am having a problem with a package submission build.
>>
>> This is the package issue:
>> https://github.com/Bioconductor/Contributions/issues/1241
>> And this is the ERROR:
>>
>> HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
>> #000: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5F.c line 509
>> in H5Fopen(): unable to open file
>> major: File accessibilty
>> minor: Unable to open file
>> #001: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5Fint.c line
>> 1498 in H5F_open(): unable to open file: time = Fri Sep 13 15:05:35
>> 2019
>> , name = '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605',
>> tent_flags = 0
>> (...)
>>
>> Quitting from lines 46-51 (fcoex.Rmd)
>> Error: processing vignette 'fcoex.Rmd' failed with diagnostics:
>> failed to open file '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605'
>> --- failed re-building 'fcoex.Rmd'
>>
>>
>> It looks like something is pointing in the wrong direction, but I have not
>> been able to figure out exactly what. I've tried re-saving the data file
>> ("mini_pbmc3k"), which is loaded in the vignette chunk of the error, but
>> that did not help.
>>
>> Does anyone have a suggestion of what might be happening?
>>
>> Thank you very much for your time,
>> Best,
>> Tiago
>>
>> Tiago Lubiana
>> Mestrando em Bioinformática, FCF/USP
>> *Computational Systems Biology Laboratory (CSBL)*
>> Telefone (laboratório): +55 (11) 2648-0240
>> Telefone (pessoal): +55 (11) 954258000
>>
>> [[alternative HTML version deleted]]
>>
>> ___
>> Bioc-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] Error in HDF5 - Package submission - Not detected locally
Hi Tiago,
I suspect what has happened here is that when create the mini_pbmc3k
object, you're doing this by subsetting the PBMC, 3k scRNA-seq data from
ExperimenHub. The assay data for that are are actually stored in an HDF5
file which will be downloaded and stored in your ExperimentHub cache on
your local machine. When you take the subset for your new object it still
points to the same HDF5 file, and when you save mini_pbmc3k all it actually
saves is the location of the HDF5 rather than putting the data in the
object. This works locally, but naturally the BioC build server doesn't
have the original HDF5 in exactly the same location, and so it fails.
Is there a reason you don't want to use the whole dataset and make the
first section of the vignette demonstrate how to download it via
ExperimentHub? Alternatively there is a way to transform from an HDF5 back
SingleCellExperiment to using the data in memory, but I can't remember it
right now - hopefully someone else will be along shortly.
This also seems like the perfect opportunity to point out that our next
BioC Developers' Forum (
https://stat.ethz.ch/pipermail/bioc-devel/2019-September/015499.html) will
have a discussion on object serilaisation which should cover exactly this
type of issue.
Best,
Mike
On Sat, 14 Sep 2019 at 23:26, Tiago Lubiana Alves <
tiago.lubiana.al...@usp.br> wrote:
> Hello,
>
> I am having a problem with a package submission build.
>
> This is the package issue:
> https://github.com/Bioconductor/Contributions/issues/1241
> And this is the ERROR:
>
> HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 0:
> #000: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5F.c line 509
> in H5Fopen(): unable to open file
> major: File accessibilty
> minor: Unable to open file
> #001: C:/hdf5_build/CMake-hdf5-1.10.5/hdf5-1.10.5/src/H5Fint.c line
> 1498 in H5F_open(): unable to open file: time = Fri Sep 13 15:05:35
> 2019
> , name = '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605',
> tent_flags = 0
> (...)
>
> Quitting from lines 46-51 (fcoex.Rmd)
> Error: processing vignette 'fcoex.Rmd' failed with diagnostics:
> failed to open file '/home/lubianat/.cache/ExperimentHub/5486ffbe0e3_1605'
> --- failed re-building 'fcoex.Rmd'
>
>
> It looks like something is pointing in the wrong direction, but I have not
> been able to figure out exactly what. I've tried re-saving the data file
> ("mini_pbmc3k"), which is loaded in the vignette chunk of the error, but
> that did not help.
>
> Does anyone have a suggestion of what might be happening?
>
> Thank you very much for your time,
> Best,
> Tiago
>
> Tiago Lubiana
> Mestrando em Bioinformática, FCF/USP
> *Computational Systems Biology Laboratory (CSBL)*
> Telefone (laboratório): +55 (11) 2648-0240
> Telefone (pessoal): +55 (11) 954258000
>
> [[alternative HTML version deleted]]
>
> ___
> Bioc-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>
[[alternative HTML version deleted]]
___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel
