Aloha,
Sounds like your data case/data model is better suited to a SQL database
(Relational DB instead of Object DB).
Zope works with MySQL, PostgreSQL etc. via ZSQLMethods.
Look into using those, with your existing data going into a SQL DB (or
maybe it is already?), if you want/need to access and manipulate the
data from within zope applications.
Then you also have the option of accessing that data via any
app/platform that can talk to your SQL DB.
I have a project I am working on where a portion of the data is very
RDB-type data, while the rest is more zope/object content-management stuff.
So the RDB-type data will be in a MySQL DB, and, accessible through the
same zope-based platform that also provides the CMS for document/object
management.
Meanwhile if other stakeholders now or later want access to the RDB-type
data they don't have to use zope, they can use whatever they want that
will talk to MySQL.
cheers,
John S.
Allen Huang wrote:
Sorry Andreas..
My product is rewritten in python placed in the product folder. So yes,
it is a instance of a class.
In the future, I will be storing more data into one instance from other
dbase file (The total columns may vary).
I'm still in the developing stage and this is just a test run to know
that is the length of processing time I'm look at. I want each entry to
be an instance because I'm planning to create other interactive
functions (edit and query(ZCatalog maybe, I'm not sure yet) and
more ) for particular info.
I've never thought of using a BTree because I don't know enough about
it. I'll look into it but will BTree still be a better choice than
making instance if I'm going to make interactive functions?
Any other suggestions?
----- Original Message ----
From: Andreas Jung <[EMAIL PROTECTED]>
To: Allen Huang <[EMAIL PROTECTED]>; Zope <[email protected]>
Sent: Tuesday, January 9, 2007 1:55:34 PM
Subject: Re: [Zope] need advice on mass data processing
--On 8. Januar 2007 19:28:32 -0800 Allen Huang <[EMAIL PROTECTED]> wrote:
> I have a data file that has over 110000 entry of 3 column data (string,
> float, float)
> currently I have written my program so it will do an entry by entry
> processing with zope. This operation is like this
> 1. read data (the data file)
> 2. create product (a python product that store three field data: one
> string and two float data) 3. update product (update the three field
> entries)
Please name things the right way. A "Product" is basically a Zope/Python
package that contains definitions of classes, scripts, templates etc.
You mean instances of a particular class?
>
> when I first tried it out with the first 1000 entries it took about 30
> seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.
You're creating 110k instances for storing a string and two floats?
If yes, that's stupid idea.
You can persistent large amounts of data within a single instances
by using Zope BTrees.
> It not every day that you have to process over 110000 data entries but
> processing over 60 minutes is still kind of long.
>
What kind of processing?
-aj
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
------------------------------------------------------------------------
_______________________________________________
Zope maillist - [email protected]
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
--
John Schinnerer - MA, Whole Systems Design
------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[EMAIL PROTECTED]
http://eco-living.net
_______________________________________________
Zope maillist - [email protected]
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
