Aloha,

Sounds like your data case/data model is better suited to a SQL database (Relational DB instead of Object DB).


Zope works with MySQL, PostgreSQL etc. via ZSQLMethods.
Look into using those, with your existing data going into a SQL DB (or maybe it is already?), if you want/need to access and manipulate the data from within zope applications.

Then you also have the option of accessing that data via any app/platform that can talk to your SQL DB.

I have a project I am working on where a portion of the data is very RDB-type data, while the rest is more zope/object content-management stuff.

So the RDB-type data will be in a MySQL DB, and, accessible through the same zope-based platform that also provides the CMS for document/object management. Meanwhile if other stakeholders now or later want access to the RDB-type data they don't have to use zope, they can use whatever they want that will talk to MySQL.

cheers,
John S.


Allen Huang wrote:

Sorry Andreas..
My product is rewritten in python placed in the product folder. So yes, it is a instance of a class. In the future, I will be storing more data into one instance from other dbase file (The total columns may vary). I'm still in the developing stage and this is just a test run to know that is the length of processing time I'm look at. I want each entry to be an instance because I'm planning to create other interactive functions (edit and query(ZCatalog maybe, I'm not sure yet) and more ) for particular info. I've never thought of using a BTree because I don't know enough about it. I'll look into it but will BTree still be a better choice than making instance if I'm going to make interactive functions? Any other suggestions?


----- Original Message ----
From: Andreas Jung <[EMAIL PROTECTED]>
To: Allen Huang <[EMAIL PROTECTED]>; Zope <zope@zope.org>
Sent: Tuesday, January 9, 2007 1:55:34 PM
Subject: Re: [Zope] need advice on mass data processing

--On 8. Januar 2007 19:28:32 -0800 Allen Huang <[EMAIL PROTECTED]> wrote:

 > I have a data file that has over 110000 entry of 3 column data (string,
 > float, float)
 > currently I have written my program so it will do an entry by entry
 > processing with zope. This operation is like this
 > 1. read data (the data file)
 > 2. create product (a python product that store three field data: one
 > string and two float data) 3. update product (update the three field
 > entries)

Please name things the right way. A "Product" is basically a Zope/Python
package that contains definitions of classes, scripts, templates etc.

You mean instances of a particular class?

 >
 > when I first tried it out with the first 1000 entries it took about 30
 > seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.

You're creating 110k instances for storing a string and two floats?
If yes, that's stupid idea.

You can persistent large amounts of data within a single instances
by using Zope BTrees.

 > It not every day that you have to process over 110000 data entries but
 > processing over 60 minutes is still kind of long.
 >

What kind of processing?

-aj


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com


------------------------------------------------------------------------

_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

--
John Schinnerer - MA, Whole Systems Design
------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
[EMAIL PROTECTED]
http://eco-living.net
_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Reply via email to