Keywords: DataAccWG
Attendees:
Robyn Allsman
Ani Thakar
Maria Nieto-Santisteban
Ray Plante
David Fleming
Kem Cook
Sergei Nikolaev
Jacek Becla
Provenance
==========
The agenda should say:
"reading material: 'Provenance Object attributes'",
not 'Provenance Use Case'
Different types of provenance objects:
Data Product, Job, Pipeline, Processing Component
- yes, it the right approach to have dedicated
provenance types for each of these
- good start, seems like a complete list
Annotation
- should we allow users to annotate objects?
(add free-form "notes")
- if we do that, we should have Annotation table
separate from other provenance data, and use
another table to link annotations with annotated
objects
- this will insure annotated objects don't need
an update when annotation is added
- what is it that we want to attach annotation to?
object or provenance?
- need to identify use case(s) that will tell us whether
we need annotation and what to annotate
user provenance
- are we going to have any provenance related to users/actors?
For instance
- real name that acts as given actor
- privileges for given actor
- not sure, leave it open, check with Jeff/Tim...
Variables/policies
- policies are simply config files. We will need to capture
contents of these files (very important)
- format very simple: <keyword>=<value>
- we need to parse these files, extract information
and store as floats/ints/strings (rather than store as
one big string)
- likely will want to use XML here...
Yes, we have more than enough info to start putting
this in uml
--> action item for Jacek
schema inspect
==============
- Component Diagram: connection from Data Access Framework
to Pipeline Construction System (PCS)
Should there be similar connection to "User Interface Services"?
- do we want to expose to users information about available
optimizations like indices, clustering etc?
--> probably not, UIS is mostly for retrieving and visualizing,
PCS will probably be exposed more to the outside world
that it is now
- applications should not worry about details of retrieving data
- how does an application developer communicate to pipeline
construction system what input data is needed?
- if DataAccess Framework knew what data is needed and how it will
be used, it could do extra optimizations ahead, eg. prebuild
certain indices or pre-cache data
NCSA will be looking into input/output for modules, this should
help answer some of the above questions
It would be useful for app developer to inquire about performance
and for Data Access Framework to send expected performance spec,
for instance:
- Q: "what performance I might get if I try to access file X"
- A: "you will get data right away, it is local"
Difficult to estimate expected performance
- no standard right now
- depends on load
- we could collect stats and use them to predict performance
- at SDSS simply have cut off time, no attempts
to do reliable estimate
- grid community is trying to develop algorithms for
performance estimation
- worth exploring
- but do we know how effectively an application can use
this information?
- it is certainly useful for a user sitting at terminal
to know if it will take minutes or whole night...
AOB
====
Tim's action item:
"Take one use case and work it through
- best usecase: "recreate calibrated image"
- Tim will do it or delegate to somebody else"
- Tim not available, will re-check next time
"Jacek will find more on how babar uses checksums,
how catches file corruption"
- no progress yet
Jacek
P.S. Sorry I'm sending the notes with few days delay...
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We have a DataAccWG telecon tomorrow (Friday) at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- Provenance (reading material: "Provenance Use Case" email thread)
- "schema inspect" (brainstorming: how an application
developer should communicate with Data Access Framework
to maximize job's performance)
Jacek
_______________________________________________
LSST-data mailing list
[EMAIL PROTECTED]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[EMAIL PROTECTED]
http://www.lsstmail.org/mailman/listinfo/lsst-data
