Keywords: DataAccWG
Sergei,
--> start working on generating schema from uml usecases
- looking at schema will help spot missing things
- will learn how to do it, the sooner the better
Are we talking about starting from scratch? Or should we use UW schema as
the basis, esp. since Nicole and Russ are making it compliant with use
cases?
The idea is to generate "brand new" schema based on use cases,
and then compare it to what we already have in the prototype.
UW schema covers only a small part of what we are doing.
Yes, the idea is to reuse Nicole and Russ work.
Also, we should start working on 'coding' standard
for schema
- first step: list issues, things like
- naming convention
- varchar vs text vs char
- etc
This is also related to cross-operability between DBMS. It's no secret
that while MySQL, MS SQL and PostgreSQL are all "SQL", they do not
understand each other schema, unless special care is taken to write it so
that they would. We've seen it when trying to port MySQL to MS SQL and
MySQL to PostgreSQL. So, for the benefit of all groups, the schema should
not contain things specific to any DBMS version.
The idea is that you automatically generate schema from EA for
each database system. So you have one "generic" model in EA,
and generate mysql-specific schema, sql server-specific schema
etc. At this point we really wanted to make sure we and
Nicole/Russ are using the same formatting style and programming style
so that the EA-generated schemas have the same 'look and feel'.
Database-specific things are unavoidable (you do want to use correct
engine, and not run out of rows in table etc, right?)
There was one other item on agenda: what schema and what data to use for
future tests. Did you guys cover this?
Yes, we won't be making any drastic moves: will continue tests
with the prototype schema in the next few months.
Generating schema from EA is an exercise focused on understanding
how to do it, and on spotting holes in the uml model
Is UW also doing ingest tests?
No, I don't think they are doing any tests for the sake of testing
database, their focus is on focused on getting specific task done,
and they just 'happen' to be using mysql to achieve that.
I have some latest info on ingest tests at Livermore.
We have two DBMS engines, PostgreSQL (8.1.1) and MySQL (5.0.17), but the
tests were done using MySQL only so far.
Here's the rig:
Quad-CPU board, 4 2x Opteron 870 processors @ 2GHz
4 GB RAM
1.8TB RAID5 for data
Data are SM (5 year), schema is still the old basic one we designed.
The numbers for ingest are:
290K rows/min into Source table, 1.1billion rows takes 2.5 days something
hours 100K rows/min into DIASource table, 34M rows takes about 6.5 hours
these are pretty stable, i.e. linear scaling as long as disk has enough
free space. Indexing has to be turned off to ingest, otherwise it becomes
logarithmic. Time to index 34M rows is ~30 min.
ok, thanks, I think we will be in a position to compare these results
to ours in the next few weeks, so let's get back to that then
Jacek
I have a brief memo for the performance (attached). Will be happy to
answer any questions.
Cheers,
Sergei.
------------------------------------------------------------------------
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
