.
---
From: CF-metadata [mailto:cf-metadata-boun...@cgd.ucar.edu] On Behalf Of Steve
Hankin
Sent: Thursday, September 19, 2013 10:03 AM
To: Corey Bettenhausen
Cc: CF Metadata List
Subject: Re: [CF-metadata] Towards recognizing and exploiting hierarchical
groups (Charlie Zender
Hi Philip,
Please read my response to Bryan Lawrence's post on another thread.
It pertains to many of your points. And I may duplicate parts below...
Hi All,
I like Steve Hankin's point (below) about 'powerful' versus
'interoperable' . I hadn't thought about it quite that way before :-).
On Sep 18, 2013, at 12:32 PM, Steve Hankin wrote:
On 9/18/2013 7:56 AM, Roy Mendelssohn - NOAA Federal wrote:
Hi All:
NASA has used hierarchies for years, and appears committed to them. So,
either it is done in an ad hoc way, or through a standard. That doesn't
mean CF is the place
Hi all,
Again, I may be unaware of all the possible uses of hierarchies, but
here's our experience with CMIP.
It seems to me if hierarchies are for the purpose of organizing
datasets (or organizing a bunch of files), this should fall outside CF's
purview because a single hierarchy is rarely
On Sep 19, 2013, at 11:29 AM, Karl Taylor wrote:
Hi all,
Again, I may be unaware of all the possible uses of hierarchies, but here's
our experience with CMIP.
It seems to me if hierarchies are for the purpose of organizing datasets
(or organizing a bunch of files), this should fall
On 9/19/2013 9:05 AM, Corey Bettenhausen wrote:
On Sep 19, 2013, at 11:29 AM, Karl Taylor wrote:
Hi all,
Again, I may be unaware of all the possible uses of hierarchies, but here's our
experience with CMIP.
It seems to me if hierarchies are for the purpose of organizing datasets (or
Hi all -
Did I misunderstand the original proposal?
Could we possibly have a proposal written up as a ticket on the Trac
system?
I'm finding it difficult to find a description of what's being
suggested, in
the many emails in this thread.
I realize we're very far from a detailed description
Hi Martin,
Interesting point of view.
Le 18/09/2013 09:43, Schultz, Martin a écrit :
Dear all,
maybe we should rephrase the question behind this discussion: How can
hierarchies and/or groups be implemented without breaking CF?
CF datasets can certainly be aggregated without loss of
Hi all,
No need to respond further to my post. I missed that the groups were
*within* the file. I have now looked at the netCDF documentation about
groups and will hopefully contribute more cogent remarks later.
cheers,
Karl
On 9/19/13 9:05 AM, Corey Bettenhausen wrote:
On Sep 19, 2013,
Hi Bryan,
Responses interleaved.
Best,
cz
Le 18/09/2013 05:57, Bryan Lawrence a écrit :
Hi Charlie
(Before I disagree with you, like most on the list, I'm glad we're
having the conversation on this topic, it needs to be had, so thanks!)
I find this particular example completely
Hello Steve,
Responses interleaved.
Best,
cz
Le 18/09/2013 09:32, Steve Hankin a écrit :
On 9/18/2013 7:56 AM, Roy Mendelssohn - NOAA Federal wrote:
Hi All:
NASA has used hierarchies for years, and appears committed to them. So,
either it is done in an ad hoc way, or through a
Hi Karl,
Please see new thread
Are ensembles a compelling use case for group-aware metadata?
It is a thinly veiled reference to CMIP5, which I don't mention
by name so as not to associate it/you with our proposal.
Nevertheless, many on the list know what CMIP5 is and it has helped
climate
Hi Steve,
Your stern but steady caution about the tension between flexibility
and interoperability is well-taken here.
What have been the down sides to the
use of groups and hierarchies?
Those who respond to this please be careful to distinguish between
HDF5 _allowed_ hierarchies and
On Sep 19, 2013, at 13:58, Charlie Zender zen...@uci.edu wrote:
Nothing in our proposal mandates a single way of representing hierarchies. Au
contraire.
The concern by the hierarchy-wary is not that you'd mandate a single way, but
that by providing an open mechanism, you would encourage a
Hello Nan,
Le 19/09/2013 12:16, Nan Galbraith a écrit :
Hi all -
Did I misunderstand the original proposal?
Could we possibly have a proposal written up as a ticket on the Trac
system?
I'd prefer not to submit anything to the Trac system
until/unless there is consensus from many doubters
Hi Corey,
Le 19/09/2013 09:05, Corey Bettenhausen a écrit :
On Sep 19, 2013, at 11:29 AM, Karl Taylor wrote:
Hi all,
Again, I may be unaware of all the possible uses of hierarchies, but here's
our experience with CMIP.
It seems to me if hierarchies are for the purpose of organizing
All,
I'm glad we are discussing this topic, but the fact that large data
providers are already distributing data using groups and hierarchies
is not a compelling reason to endorse this practice through CF. After
all, a lot of data providers are currently distributing scientific
data in any
Hi Charlie
(Before I disagree with you, like most on the list, I'm glad we're having
the conversation on this topic, it needs to be had, so thanks!)
I find this particular example completely unhelpful, not least because I
don't see the utility for doing so. However, I can see that others might
Hi All:
NASA has used hierarchies for years, and appears committed to them. So, either
it is done in an ad hoc way, or through a standard. That doesn't mean CF is
the place for the standard, just that it would be nice to have one.
I would point out that every major modern programming
On 9/18/2013 7:56 AM, Roy Mendelssohn - NOAA Federal wrote:
Hi All:
NASA has used hierarchies for years, and appears committed to them. So, either
it is done in an ad hoc way, or through a standard. That doesn't mean CF is
the place for the standard, just that it would be nice to have
Dear Charlie
Thank you for your interesting post and the discussion.
As a data analyst, I have a different view from NASA. I dislike hierarchies
and directories. I prefer things to be as flat as possible, with each item
thoroughly described by its own independent metadata, using tools to
Hi Folks
CMIP5 is illuminating in a number of ways ... not least because it is
impossible to come up with a *natural* hierarchy for consumers of the data
(as opposed to the producers). But even the producers have different ways
of organising their material (running members of different ensembles
Bryan has beaten me to the points I would have made. I think hierarchies are
over rated at the interface level. Examples abound of where they have been
abandoned: hierarchal vs relational DBs, XML databases and tools (save us from
xquery for Netcdf!).
Under the hood hierarchies are often
Hi.
I strongly support the idea of adding groups to CF. As a data producer and
consumer, I vastly prefer to have collections of similar items grouped together
rather than laying about in a single large bin. (I also make extensive use of
folders on my computer!) I am currently building
HI.
I've got a few of questions. People have suggested using '.' delimiters to
build full variable names (path names). Since '.' has historically been a
legal part of a variable name, would it perhaps be better to use ':', '\', or
'|' instead? And why is '/' forbidden? Is it used
)
--
Message: 1
Date: Tue, 17 Sep 2013 09:51:51 -0400
From: Jim Biard jim.bi...@noaa.gov
To: cf-metadata@cgd.ucar.edu List cf-metadata@cgd.ucar.edu
Subject: Re: [CF-metadata] Towards recognizing and exploiting
hierarchicalgroups
Message-ID: 0b9e777c-c317-4ce9-acea-cfdf93c2c...@noaa.gov
Hello Jim,
Just a note to affirm that the gradualist approach you recommend below
seems the natural to me. Settle the meanings of groups and scope first.
This is the foundation.
If the foundation seems solid, then take-on the definition of specific
group-aware metadata features aimed at
Dear Jonathan,
Thanks for your input. I expect many here share your views.
Directories full of flat files are often a sensible way to organize.
Easy to search for and ingest what with ls or grep and options.
What limits such searchability within hierarchical files?
Hierarchical files look just
Hello Bryan,
Thanks for your perspective/counterexample about whether CMIP5 is in
fact a natural candidate for hierarchies. I agree with your points
about the utility of flat systems of objects, while retaining my urge
to hierarchically organize some objects sometimes. Everyone packs
their
Hi Stephen,
I also advocate keeping support for groups simple.
The simplest support imaginable is that groups should be completely
self-contained without attribute inheritance. This would go in the
right direction, yet I think attribute inheritance, at least of global
metadata, ought to be
Hi Martin,
Let me enumerate a third disaggregation method to your list:
There should probably be two ways for converting group files to flat
files:
a) flatten everything into one file with . separated name spaces
b) flatten groups into individual files (tools like NCO could then
use the
Hello John,
As I understand it, your starting concern is to be able to put
things into folder-equivalents, pure and simple.
Our original post describes my goals. Groups are a logical place to
start because they must exist for inheritance to be meaningful.
I am for having 1. groups as
Hi All:
I am old and slow, and I must be missing something, because at this point most
of the discussion has been about the desirability of files with groups and
hierarchies. Again, unless I am missing something, there already are data
providers who are distributing data using groups and
Dear all,
I'm also glad to see this discussion surface. Since I first presented
Developing Conventions for netCDF-4 at the 2007 GO-ESSP meeting:
http://www.unidata.ucar.edu/presentations/Rew/nc4-conventions.pdf
I've been hoping that netCDF-4 feature adoption would begin to gain
traction in
Charlie Co,
Also, regardless of whether these hierarchical structures are stored
in NetCDF4 or flattened NetCDF3, we get a big boost in
interoperability when we write datasets with known featureTypes
(profile, time series collection, swath, etc), because then workflows
that have performed a
Hi Charlie,
Great that you have opened the door onto this discussion topic. Total
agreement from my pov that group-awareness in CF is an area that is
crying to be explored and solved. Your analysis of technical details
-- e.g. attribute scope and inheritance by group descendents, etc. --
Hi Martin,
When NCO's ncecat aggregates multiple files into a single file, it
does as you describe and puts each into its own group with all the
global metadata now as group metadata. What should be the new global
metadata of the group file? neceat duplicates the first file's
metadata for that,
Hi Russ,
Thanks for your input and link to an earlier presentation of yours.
Agree that the proposal only applies to group hierarchies, i.e., to
groups representable by the Common Data Model 2/extended/enhanced
which for practical purposes means groups exposed by the netCDF4 API.
Your way of
NASA has recently convened an Earth Science Data System Working Group to
explore existing conventions for data and products stored in HDF and to
make recommendations for future developments. The CF Conventions are an
important element in this work, as many scientists and users are
interested in
39 matches
Mail list logo
