Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
Hi Aaron and Martin, In any case, a distance matrix calculated from such an array would be fine, as long as the dimensions are equal to the number of cells. The question is whether it is needed by enough packages to warrant a slot in the base SCE class; I will discuss this with Davide and Vlad. Yeah, that was my point. since it’s cumbersome and non-natural to have 3D data in there, a distance matrix would fit better. BQ_BEGIN The SingleCellExperiment package makes no statement on whether downstream users/packages want to (or not) use the tidy-verse or ggplot2. It simply provides the minimal class and methods; convenience wrappers are left to the discretion of each package developer. scater, for example, implements a few dplyr verbs for SCE objects. BQ_END Of course, but only fortify is ggplot2-specific. But anyway; Martin pointed out: BQ_BEGIN FWIW, SummarizedExperiment (hence SingleCellExperimet) uses $ (and $<-; also [[ and [[<-) to access (with auto-completion) colData. And it wouldn't be a good idea to have $ access one element (counts) and $<- modify another (rowData), or to mix what $ and [[ access. Martin BQ_END Ah, I see! names and .DollarNames.SummarizedExperiment are also defined! Well, that’s good enough i think, although this seems cumbersome: assays(sce)$log_counts[ 'Actb' , ] Cheers, Phil Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
On 08/08/2017 03:59 AM, Angerer, Philipp wrote: Hi Aaron, I guess this would be a question for the SummarizedExperiment developers, though personally, I never liked ExpressionSet's inclination to slap names on everything. Too bad we’re bound to SummarizedExperiment’s “rows” and “cols”. Since they always refer to features and samples, respectively: Why not name them that? There’s already too many APIs in too many programming languages that confusingly have one or the other convention – if whe know which is which, why not name them after that knowledge? BQ_BEGIN It probably wouldn't be a good idea to store distances as expression matrices. However, if there is a need for it, we can add a new slot for distance matrices. I think SC3 has a similar requirement, so perhaps this would be more generally useful than I first thought. You can post an issue on the github repository to remind Davide or me to do it. BQ_END Distance matrices (cell×cell) can’t only come from cell×gene matrices. You can e.g. use dynamic time warping to create them from cell×gene×time arrays. BQ_BEGIN Finally, I'm not sure what advantages those ergonomics provide. Indeed, if every package defines its own plot() S4 method for SingleCellExperiment, they will clobber each other in the dispatch table, resulting in some interesting results dependent on package loading order. If you have destiny-specific data and methods, best to keep them separate rather than stuffing them into the SCE object. BQ_END I wrote that I could e.g. create a plot_dm method, which plots a diffusion map stored in a SCE. Also I didn’t mean the plot method with ergonomics. I meant fortify , names , $ , and [[ . Those would be very useful, as you could just do things like the following, and have autocompletion: sce$Predicate1 <- sce$SampleMeta1 > 40 # `$` accesses counts (by gene) and rowData. `$<-` sets rowData qplot(Gene1, Gene2, colour = Predicate1, data = sce) # fortify creates a data.frame containing cbind(t(counts), rowData) FWIW, SummarizedExperiment (hence SingleCellExperimet) uses $ (and $<-; also [[ and [[<-) to access (with auto-completion) colData. And it wouldn't be a good idea to have $ access one element (counts) and $<- modify another (rowData), or to mix what $ and [[ access. Martin Just as you can do now with DiffusionMap objects. Also I’m not sure if i got rowData and the “t” right in the above code ;) I meant cbind(counts as cell×gene, sampleMeta as cell×n_meta) Best, Phil Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel This email message may contain legally privileged and/or...{{dropped:2}} ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
> I guess this would be a question for the > SummarizedExperiment developers, though personally, I never liked > ExpressionSet's inclination to slap names on everything. > > Too bad we’re bound to SummarizedExperiment’s “rows” and “cols”. Since > they always refer to features and samples, respectively: Why not name > them that? > > There’s already too many APIs in too many programming languages that > confusingly have one or the other convention – if whe know which is > which, why not name them after that knowledge? *shrug* + *meh*. As I said, I'm the wrong person to complain to about this. Though I don't have particularly strong feelings either way. > It probably wouldn't be a good idea to store distances as expression > matrices. However, if there is a need for it, we can add a new slot > for distance matrices. I think SC3 has a similar requirement, so > perhaps this would be more generally useful than I first thought. > You can post an issue on the github repository to remind Davide or > me to do it. > > Distance matrices (cell×cell) can’t only come from cell×gene matrices. > You can e.g. use dynamic time warping to create them from cell×gene×time > arrays. I don't think there's direct support for >2-dimensional arrays in SE objects. You might be able to put them in, but I don't know how well it will interact with the subsetting machinery. One solution is to split it up by the third dimension and store each matrix as a separate assay. In any case, a distance matrix calculated from such an array would be fine, as long as the dimensions are equal to the number of cells. The question is whether it is needed by enough packages to warrant a slot in the base SCE class; I will discuss this with Davide and Vlad. > Finally, I'm not sure what advantages those ergonomics provide. > Indeed, if every package defines its own plot() S4 method for > SingleCellExperiment, they will clobber each other in the dispatch > table, resulting in some interesting results dependent on package > loading order. If you have destiny-specific data and methods, best > to keep them separate rather than stuffing them into the SCE object. > > I wrote that I could e.g. create a plot_dm method, which plots a > diffusion map stored in a SCE. > > Also I didn’t mean the plot method with ergonomics. I meant |fortify|, > |names|, |$|, and |[[|. Those would be very useful, as you could just do > things like the following, and have autocompletion: > > sce$Predicate1 <- sce$SampleMeta1 > 40# `$` accesses counts (by gene) > and rowData. `$<-` sets rowData > qplot(Gene1, Gene2, colour = Predicate1, data = sce) # fortify creates a > data.frame containing cbind(t(counts), rowData) The SingleCellExperiment package makes no statement on whether downstream users/packages want to (or not) use the tidy-verse or ggplot2. It simply provides the minimal class and methods; convenience wrappers are left to the discretion of each package developer. scater, for example, implements a few dplyr verbs for SCE objects. Cheers, Aaron ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
On 08/08/2017 03:59 AM, Angerer, Philipp wrote: Hi Aaron, I guess this would be a question for the SummarizedExperiment developers, though personally, I never liked ExpressionSet's inclination to slap names on everything. Too bad we’re bound to SummarizedExperiment’s “rows” and “cols”. Since they always refer to features and samples, respectively: Why not name them that? Language is a funny thing. In the ExpressionSet world, 'features' were actually a misnomer, since they refer to spots (probes) on the microarray, rather than summarized expression values of individual genes. Rectangular data from other assays might well label the observations made on each sample as something different from 'feature'. Likewise, the columns were called 'phenoData', describing the phenotypes of the samples, but phenotype has a different meanings in different disciplines (hey wait, my experiment used two genetically different mice, we're talking about _geno_types, not phenotypes!). And of course 'sample' has statistical meanings that only sometimes applies. In the end it seemed better to use generic terms for a data class meant for general use. Martin There’s already too many APIs in too many programming languages that confusingly have one or the other convention – if whe know which is which, why not name them after that knowledge? BQ_BEGIN It probably wouldn't be a good idea to store distances as expression matrices. However, if there is a need for it, we can add a new slot for distance matrices. I think SC3 has a similar requirement, so perhaps this would be more generally useful than I first thought. You can post an issue on the github repository to remind Davide or me to do it. BQ_END Distance matrices (cell×cell) can’t only come from cell×gene matrices. You can e.g. use dynamic time warping to create them from cell×gene×time arrays. BQ_BEGIN Finally, I'm not sure what advantages those ergonomics provide. Indeed, if every package defines its own plot() S4 method for SingleCellExperiment, they will clobber each other in the dispatch table, resulting in some interesting results dependent on package loading order. If you have destiny-specific data and methods, best to keep them separate rather than stuffing them into the SCE object. BQ_END I wrote that I could e.g. create a plot_dm method, which plots a diffusion map stored in a SCE. Also I didn’t mean the plot method with ergonomics. I meant fortify , names , $ , and [[ . Those would be very useful, as you could just do things like the following, and have autocompletion: sce$Predicate1 <- sce$SampleMeta1 > 40 # `$` accesses counts (by gene) and rowData. `$<-` sets rowData qplot(Gene1, Gene2, colour = Predicate1, data = sce) # fortify creates a data.frame containing cbind(t(counts), rowData) Just as you can do now with DiffusionMap objects. Also I’m not sure if i got rowData and the “t” right in the above code ;) I meant cbind(counts as cell×gene, sampleMeta as cell×n_meta) Best, Phil Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel This email message may contain legally privileged and/or...{{dropped:2}} ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
Hi Aaron, I guess this would be a question for the SummarizedExperiment developers, though personally, I never liked ExpressionSet's inclination to slap names on everything. Too bad we’re bound to SummarizedExperiment’s “rows” and “cols”. Since they always refer to features and samples, respectively: Why not name them that? There’s already too many APIs in too many programming languages that confusingly have one or the other convention – if whe know which is which, why not name them after that knowledge? BQ_BEGIN It probably wouldn't be a good idea to store distances as expression matrices. However, if there is a need for it, we can add a new slot for distance matrices. I think SC3 has a similar requirement, so perhaps this would be more generally useful than I first thought. You can post an issue on the github repository to remind Davide or me to do it. BQ_END Distance matrices (cell×cell) can’t only come from cell×gene matrices. You can e.g. use dynamic time warping to create them from cell×gene×time arrays. BQ_BEGIN Finally, I'm not sure what advantages those ergonomics provide. Indeed, if every package defines its own plot() S4 method for SingleCellExperiment, they will clobber each other in the dispatch table, resulting in some interesting results dependent on package loading order. If you have destiny-specific data and methods, best to keep them separate rather than stuffing them into the SCE object. BQ_END I wrote that I could e.g. create a plot_dm method, which plots a diffusion map stored in a SCE. Also I didn’t mean the plot method with ergonomics. I meant fortify , names , $ , and [[ . Those would be very useful, as you could just do things like the following, and have autocompletion: sce$Predicate1 <- sce$SampleMeta1 > 40 # `$` accesses counts (by gene) and rowData. `$<-` sets rowData qplot(Gene1, Gene2, colour = Predicate1, data = sce) # fortify creates a data.frame containing cbind(t(counts), rowData) Just as you can do now with DiffusionMap objects. Also I’m not sure if i got rowData and the “t” right in the above code ;) I meant cbind(counts as cell×gene, sampleMeta as cell×n_meta) Best, Phil Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 [[alternative HTML version deleted]] ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
perspective, if the DiffusionMap() function vomits out a lot of metadata fields, that might not be desirable if only the final diffusion coordinates are of interest. In such cases, I would find it easier to just extract the coordinates and store it in reducedDim<- manually. Whether this is done from a DiffusionMap or SingleCellExperiment output makes little difference to me. Finally, I'm not sure what advantages those ergonomics provide. Indeed, if every package defines its own plot() S4 method for SingleCellExperiment, they will clobber each other in the dispatch table, resulting in some interesting results dependent on package loading order. If you have destiny-specific data and methods, best to keep them separate rather than stuffing them into the SCE object. Our vision for the SCE class is to coordinate inputs into many packages across a long, long workflow. A little detour into destiny's classes for a small portion of the workflow doesn't pose much trouble, as long as any relevant statistics can be extracted and stored in the SCE object when it moves to the next stage of the workflow. -Aaron > -------- > *Von: *"Aaron Lun" <a...@wehi.edu.au> > *An: *"bioc-devel" <bioc-devel@r-project.org> > *Gesendet: *Montag, 31. Juli 2017 10:38:03 > *Betreff: *Re: [Bioc-devel] transitioning scater/scran to > SingleCellExperiment > > Dear developers, > > Both scater and scran will be migrating to the SingleCellExperiment > class (https://bioconductor.org/packages/SingleCellExperiment) in the > next BioC release. This is based on a SummarizedExperiment and provides > a more modern user interface, as well as supporting different matrix > representations (e.g., dgCMatrix, HDF5Matrix). > > We note that there are a number of Bioconductor packages that depend > on/import/suggest scater or scran, which we have listed below: > > scDD > scone > SIMLR > splatter > Glimma > SC3 > phenopath > switchde > > To the maintainers of these packages, we advise switching from SCESet to > SingleCellExperiment as soon as possible; the former will be deprecated > in the next release cycle. There are several things to note here: > > - The SCESet previously contained a number of slots relating to > distances and clustering results. These are no longer present in the > SingleCellExperiment, in line with the minimalist design philosophy of > that package. If these are necessary, we suggest extending the > SingleCellExperiment class in your own packages(*). > > - For packages that depend directly on methods in scater or scran, a > number of methods have been removed. This aims to simplify the analysis > workflow and code maintenance by reducing redundancy. Please ensure that > your package does not need those missing methods by CHECKing it against > the experimental versions(**) of these two packages: > > https://github.com/LTLA/scran > https://github.com/davismcc/scater/tree/future > > If there are any issues with the switch, please let us know and we will > do our best to figure out the most appropriate fix. > > Regards, > > Aaron, Davis and Davide > > (*): If there is popular demand for some slots, we may consider > including it in the base SingleCellExperiment object. > > (**): These versions are highly experimental and fluid, and results are > likely to be unstable over the coming month. Nonetheless, if something > is breaking, it is best that we know sooner rather than later. Or in > other words, don't start complaining when it's close to release time. > ___ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > Helmholtz Zentrum München > Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH) > Ingolstädter Landstr. 1 > 85764 Neuherberg > www.helmholtz-muenchen.de > Aufsichtsratsvorsitzende: MinDir'in Bärbel Brumme-Bothe > Geschäftsführer: Prof. Dr. Günther Wess, Heinrich Baßler, Dr. Alfons Enhsen > Registergericht: Amtsgericht München HRB 6466 > USt-IdNr: DE 129521671 ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
Hi! As I told you at the HCA hackathon, I’m interested in switching over destiny! I think the class is a really cool idea and seems very well thought out. Interestingly the design decisions coverged very well with [ https://github.com/theislab/scanpy#readme | scanpy ] ’s [ https://www.pydoc.io/pypi/scanpy-0.2.3/autoapi/data_structs/ann_data/index.html#data_structs.ann_data.AnnData | AnnData ] class that I helped Alex design. Scanpy makes heavy use of HDF5 serialization. I think we should quickly converge on a serialization format (keys and so on) so that AnnData and SingleCellExperiment can have interoperability via HDF5! The only point of criticism is that you, while staying specific to single cell data, named the dimensions “rows” and “columns” instead of e.g. “samples” and “variables”. Alex and me came to the conclusion that ExpressionSet ’s way of returning a named vector for dims is a good idea, and having the dimensions named for their roles reduces confusion. I have “two” “questions” regarding destiny, with some feature requests hiding in the second one: 1. destiny accepts either an expression matrix or a distance matrix (both with optional metadata). Currently the signature is this: DiffusionMap(data = ExpressionSet | data.frame | matrix | Matrix, distance = NULL | "euclidean" | "cosine" | "rankcor" ) DiffusionMap(data = NULL | data.frame , # Metadata distance = matrix | dist | symmetricMatrix) The idea is that both when providing expressions and when providing a distance matrix, you should be able to provide metadata. I’m not super happy with my approach, since the current methods of providing metadata differ. However, ExpressionSet and SingleCellExperiment are both specific for expression data. I think neither can hold dist objects as data. Is it valid and a good idea to neither store counts not exprs, but e.g. SingleCellExperiment(assays = list(dists = some_mat)) ? It wouldn’t be sliced properly, for example, and it being symmetric would mean that column and row metadata is the same… Is it a good idea to require assays to have certain names (e.g. “exprs” or “dists” here)? 2. The reducedDim methods would be able to store and retrieve diffusion components in a SingleCellExperiment , while destiny’s dataset method stores the original data used to create a DiffusionMap . What do you think is the best approach? Just conversions between the two classes? Or also deprecate DiffusionMap objects and create a diffusion_map function that returns a SingleCellExperiment object with the reduced dimensions and all the necessary metadata for further methods like e.g. DPT? I think for the latter, SingleCellExperiment isn’t quite cool enough yet :P. I’d like to have the full ergonomics of DiffusionMap : * A names method (returning gene and per-cell-metadata names) * Gene/per-cell-metadata access by $ and [[ . * A fortify method that makes everything available in ggplot2. (E.g. ggplot(dm, aes(DC1, DC2, colour = Condition)) works!) I can do without the remaining methods (or provide them in destiny), as they are are neither general purpose enough for SingleCellExperiment nor really necessary, e.g. I can add an alias plot(a_dm_object) → plot_dm(a_sce_object) . Cheers, Philipp Von: "Aaron Lun" <a...@wehi.edu.au> An: "bioc-devel" <bioc-devel@r-project.org> Gesendet: Montag, 31. Juli 2017 10:38:03 Betreff: Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment Dear developers, Both scater and scran will be migrating to the SingleCellExperiment class (https://bioconductor.org/packages/SingleCellExperiment) in the next BioC release. This is based on a SummarizedExperiment and provides a more modern user interface, as well as supporting different matrix representations (e.g., dgCMatrix, HDF5Matrix). We note that there are a number of Bioconductor packages that depend on/import/suggest scater or scran, which we have listed below: scDD scone SIMLR splatter Glimma SC3 phenopath switchde To the maintainers of these packages, we advise switching from SCESet to SingleCellExperiment as soon as possible; the former will be deprecated in the next release cycle. There are several things to note here: - The SCESet previously contained a number of slots relating to distances and clustering results. These are no longer present in the SingleCellExperiment, in line with the minimalist design philosophy of that package. If these are necessary, we suggest extending the SingleCellExperiment class in your own packages(*). - For packages that depend directly on methods in scater or scran, a number of methods have been removed. This aims to simplify the analysis workflow and code maintenance by reducing redundancy. Please ensure that your package does not need those missing methods by CHECKing it against the experimental versi
Re: [Bioc-devel] transitioning scater/scran to SingleCellExperiment
Dear developers, Both scater and scran will be migrating to the SingleCellExperiment class (https://bioconductor.org/packages/SingleCellExperiment) in the next BioC release. This is based on a SummarizedExperiment and provides a more modern user interface, as well as supporting different matrix representations (e.g., dgCMatrix, HDF5Matrix). We note that there are a number of Bioconductor packages that depend on/import/suggest scater or scran, which we have listed below: scDD scone SIMLR splatter Glimma SC3 phenopath switchde To the maintainers of these packages, we advise switching from SCESet to SingleCellExperiment as soon as possible; the former will be deprecated in the next release cycle. There are several things to note here: - The SCESet previously contained a number of slots relating to distances and clustering results. These are no longer present in the SingleCellExperiment, in line with the minimalist design philosophy of that package. If these are necessary, we suggest extending the SingleCellExperiment class in your own packages(*). - For packages that depend directly on methods in scater or scran, a number of methods have been removed. This aims to simplify the analysis workflow and code maintenance by reducing redundancy. Please ensure that your package does not need those missing methods by CHECKing it against the experimental versions(**) of these two packages: https://github.com/LTLA/scran https://github.com/davismcc/scater/tree/future If there are any issues with the switch, please let us know and we will do our best to figure out the most appropriate fix. Regards, Aaron, Davis and Davide (*): If there is popular demand for some slots, we may consider including it in the base SingleCellExperiment object. (**): These versions are highly experimental and fluid, and results are likely to be unstable over the coming month. Nonetheless, if something is breaking, it is best that we know sooner rather than later. Or in other words, don't start complaining when it's close to release time. ___ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel