Re: [Metakit] Question on compiling Metakit for Mandrake 10 (Python 2.3.3)
Quoting Mikhail Sobolev [EMAIL PROTECTED]: On Thu, Jun 24, 2004 at 12:48:06PM -0400, [EMAIL PROTECTED] wrote: I have a question on compiling Metakit 2.4.9.3 for Mandrake 10 with Python 2.3.3. I get the following error when I type python compile.py build in the ~/metakit/2.4.9.3/metakit-2.4.9.3/python directory: ... error: invalid Python installation: unable to open /usr/lib/python2.3/config/Makefile (No such file or directory) Which is good, because there is no /config directory. Do I need to have the python source on my system to install metakit? I am using Debian. For this particular directory, Debian requires python2.3-dev package, which includes everything under /usr/lib/python2.3/config I had to install libpython2.3-devel to get the files. Once that was installed I needed to do the ../unix/configure from the /build directory. Then after make...make install I was able to do python setup.py build and it worked. Hope this helps Thanks, it helped a lot! Wayne _ Metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
[Metakit] Question on compiling Metakit for Mandrake 10 (Python 2.3.3)
Hello, I have a question on compiling Metakit 2.4.9.3 for Mandrake 10 with Python 2.3.3. How do you do it? :) I get the following error when I type python compile.py build in the ~/metakit/2.4.9.3/metakit-2.4.9.3/python directory: running build running build_py running build_ext running config error: invalid Python installation: unable to open /usr/lib/python2.3/config/Makefile (No such file or directory) Which is good, because there is no /config directory. Do I need to have the python source on my system to install metakit? Thanks for any help, Wayne _ Metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question
Thanks Jean-Claude.. a couple of questions (below): - Original Message - From: Jean-Claude Wippler [EMAIL PROTECTED] To: Metakit list [EMAIL PROTECTED] Sent: Friday, January 02, 2004 1:49 PM Subject: Re: [Metakit] Question chris mollis wrote: I have a question about the best way to validate information on reads/writes to the db. For example, I'd like to make sure that data that is written out during a particular commit (by calculating a hash of data written, perhaps) can be verified again when the database is re-opened at a later date (possibly calculating the hash again and then checking this against the previous hash). What do you recommend to be the best way to do something like this? Should I override DataWrite/DataRead methods of c4_FileStrategy to calculate hashes on read and write operations? Good questions. There are several aspects to consider. The first one is really what sort of validation you are after: if you need to verify storage in general, then one could argue that there really is no other option than full file checksums, and even then it'll depend on the sort of validation as to when and how often you need to do it. Such checksums could be done outside MK, i.e. after commits and before opens. CM: agreed.. this was my first thought, but seemed a bit much if I just wanted to check deltas between reads and writes... Another point to be made is that MK is a database: it does not read or write all data each time the datafile is used. By validating writes on commit, you'll be checking only what it changed, not the entire datafile. Due to the way data is stored, the data written can be all over the datafile, it's not necessarily contiguous (though individual columns are). CM: right, this is what I thought.. I had a feeling it was going to be more difficult than I first thought :) It gets worse: MK usually loads data by mapping a file into memory. That means no read system calls take place at all in most cases: the data is mapped to a range of addresses and paged in via O/S page faults when accessed, which is a matter of following pointers. CM: Ok.. I think I will need to disable memory-mapped IO (as you point out below).. If you really insist on doing this in some sort of fine-grained manner, my suggestion would be to use a custom c4_Strategy class as you mention yourself, in combination with a *second* MK datafile. The invariant is that MK always writes entire columns - I suspect that it is possible to detect the column boundaries written by intercepting DataWrite(). The main call comes from column.cpp line 1532. Or it may be necessary to introduce two extra strategy members which get called once in each call to c4_Column::SaveNow(): - strategy_.DataInit(pos_) - unmodified while (iter...) loop - strategy_.DataDone(_size) The DataInit would reset a checksum field in the strategy object (and remember pos_), the DataWrite calls would incrementally update the checksum, and the DataDone call would save a pos,size,check triple in the second MK datafile. It'll take some extra logic to make this work across multiple commits, i.e. when space gets re-used, but that ought to be doable. You may want to use hashed views for the secondary MK file, to make it snappy. CM: I'll have to work through an example of how this gets done and get back to you with questions.. I'm a bit confused about what actually gets written in the second file and how it's verified during access... (I'll have to test out some stuff before I can ask an intelligent question here :).. The most important problem to deal with is *when* to verify such saved checksums. If it has to be done during access, then I can't think of any other way than to disable memory mapping (by overriding c4_Strategy::ResetFileMapping with a dummy which does nothing). That makes MK slower and makes it use considerably more temp memory, however - so you'll have to think hard whether that is really what you want. If you just want to checksum occasionally, then you could iterate through all triples in the secondary MK file and verify each of the ranges. CM: again, I'm a bit confused as to what the triples are... I assume that these are just file markers (file position, size, along with the hash)... I'll try it out.. thanks for the advice! Another idea would be to save checksums per fixed-size block, say 4 Kb. That means DataWrite would track checksums, but it may need to read some data of the disk to deal with writes which are not exactly on block boundaries. This needs some thought to optimize, since most DataWrite calls will not be aligned nicely. Then again, DataWrite does get called in mostly sequential order, since it writes entire columns most of the time. -jcw _ Metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit _ Metakit mailing list - [EMAIL PROTECTED] http
Re: [Metakit] Question
chris mollis wrote: I have a question about the best way to validate information on reads/writes to the db. For example, I'd like to make sure that data that is written out during a particular commit (by calculating a hash of data written, perhaps) can be verified again when the database is re-opened at a later date (possibly calculating the hash again and then checking this against the previous hash). What do you recommend to be the best way to do something like this? Should I override DataWrite/DataRead methods of c4_FileStrategy to calculate hashes on read and write operations? Good questions. There are several aspects to consider. The first one is really what sort of validation you are after: if you need to verify storage in general, then one could argue that there really is no other option than full file checksums, and even then it'll depend on the sort of validation as to when and how often you need to do it. Such checksums could be done outside MK, i.e. after commits and before opens. Another point to be made is that MK is a database: it does not read or write all data each time the datafile is used. By validating writes on commit, you'll be checking only what it changed, not the entire datafile. Due to the way data is stored, the data written can be all over the datafile, it's not necessarily contiguous (though individual columns are). It gets worse: MK usually loads data by mapping a file into memory. That means no read system calls take place at all in most cases: the data is mapped to a range of addresses and paged in via O/S page faults when accessed, which is a matter of following pointers. If you really insist on doing this in some sort of fine-grained manner, my suggestion would be to use a custom c4_Strategy class as you mention yourself, in combination with a *second* MK datafile. The invariant is that MK always writes entire columns - I suspect that it is possible to detect the column boundaries written by intercepting DataWrite(). The main call comes from column.cpp line 1532. Or it may be necessary to introduce two extra strategy members which get called once in each call to c4_Column::SaveNow(): - strategy_.DataInit(pos_) - unmodified while (iter...) loop - strategy_.DataDone(_size) The DataInit would reset a checksum field in the strategy object (and remember pos_), the DataWrite calls would incrementally update the checksum, and the DataDone call would save a pos,size,check triple in the second MK datafile. It'll take some extra logic to make this work across multiple commits, i.e. when space gets re-used, but that ought to be doable. You may want to use hashed views for the secondary MK file, to make it snappy. The most important problem to deal with is *when* to verify such saved checksums. If it has to be done during access, then I can't think of any other way than to disable memory mapping (by overriding c4_Strategy::ResetFileMapping with a dummy which does nothing). That makes MK slower and makes it use considerably more temp memory, however - so you'll have to think hard whether that is really what you want. If you just want to checksum occasionally, then you could iterate through all triples in the secondary MK file and verify each of the ranges. Another idea would be to save checksums per fixed-size block, say 4 Kb. That means DataWrite would track checksums, but it may need to read some data of the disk to deal with writes which are not exactly on block boundaries. This needs some thought to optimize, since most DataWrite calls will not be aligned nicely. Then again, DataWrite does get called in mostly sequential order, since it writes entire columns most of the time. -jcw _ Metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question on how to remove top level views
Berk, Murat wrote: We used to store a view name in one view (instead of marking cell as subview) so that we can do getAs(name) and use it. I do not want to change a lot of things, but when we try to delete the rows in the first view, the only thing we can do is to remove all elements of the second view but I cannot really delete it. view1 name field1 field2 field3 name1 data... name2 data... __name1__ prop1 prop2 prop3 __name2__ prop1 prop2 prop3 When we delete name1 from the first view, I want to remove the whole view called __name1__. How can I do this.. storage.GetAs(__name1__) IOW, omit the usual [...] part. Am I missing something? Since everything is a view including database itself, in theory this should be possible. Yes. The one confusing part could be that the view won't go away until committed. All properties, including subviews, including therefore also top-level views, stay around afetr a restructure, and only truly vanish on commit (that makes it possible to re-structure, copy over, then commit). -jcw ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
RE: [Metakit] Question on how to remove top level views
Thanks I did not know that GetAs(name) will delete the view :) Murat -Original Message- From: Jean-Claude Wippler [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 11, 2003 3:18 AM To: Metakit mailing list Subject: Re: [Metakit] Question on how to remove top level views Berk, Murat wrote: We used to store a view name in one view (instead of marking cell as subview) so that we can do getAs(name) and use it. I do not want to change a lot of things, but when we try to delete the rows in the first view, the only thing we can do is to remove all elements of the second view but I cannot really delete it. view1 name field1 field2 field3 name1 data... name2 data... __name1__ prop1 prop2 prop3 __name2__ prop1 prop2 prop3 When we delete name1 from the first view, I want to remove the whole view called __name1__. How can I do this.. storage.GetAs(__name1__) IOW, omit the usual [...] part. Am I missing something? Since everything is a view including database itself, in theory this should be possible. Yes. The one confusing part could be that the view won't go away until committed. All properties, including subviews, including therefore also top-level views, stay around afetr a restructure, and only truly vanish on commit (that makes it possible to re-structure, copy over, then commit). -jcw ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
[Metakit] Question on how to remove top level views
We have a database from the days where subviews are not as functional as now. We used to store a view name in one view (instead of marking cell as subview) so that we can do getAs(name) and use it. I do not want to change a lot of things, but when we try to delete the rows in the first view, the only thing we can do is to remove all elements of the second view but I cannot really delete it. view1 name field1 field2 field3 name1 data... name2 data... __name1__ prop1 prop2 prop3 __name2__ prop1 prop2 prop3 When we delete name1 from the first view, I want to remove the whole view called __name1__. How can I do this.. Since everything is a view including database itself, in theory this should be possible. Murat ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question about large databases
--- mARK bLOORE [EMAIL PROTECTED] wrote: actually, if metakit can't memory-map the database file (eg because it is too big), it will revert to normal i/o. this means that you can open a huge file, but access will be slower. one odd wrinkle: the release version of metakit catches the error when it fails to map a file, and switches to normal i/o. the debug version reports the error and dies! = -- mARK bLOORE [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question about large databases
mARK bLOORE wrote: actually, if metakit can't memory-map the database file (eg because it is too big), it will revert to normal i/o. this means that you can open a huge file, but access will be slower. This has been my experience as well. I'm actually using databases of 500MB in size with very reasonable performance. Just make sure that your joins are 1 to 1 :) -- Brian Kelley [EMAIL PROTECTED] Whitehead Institute for Biomedical Research 617 258-6191 ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question about large databases
actually, if metakit can't memory-map the database file (eg because it is too big), it will revert to normal i/o. this means that you can open a huge file, but access will be slower. --- Jacob Levy [EMAIL PROTECTED] wrote: On operating systems on which Metakit finds enough support for using memory mappings, it will load the entire database into a memory mapped region of memory, and then the OS takes care of reading/writing the data from/to the actual storage as needed, based on memory accesses. This does mean that for those operating systems, the largest possible storage size is limited by the available virtual memory. = -- mARK bLOORE [EMAIL PROTECTED] __ Do you Yahoo!? Yahoo! SiteBuilder - Free, easy-to-use web site design software http://sitebuilder.yahoo.com ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
[Metakit] Question about large databases
Hi, I've got a concern about what might happen if a database file starts getting *really* big - like say 20MB or more. When I first create the storage object: db = metakit.storage(my-big-db.mk, 1) Does the entire contents of the database get loaded into memory, or is database content pulled in as needed? If the whole database doesn't get loaded into memory lock, stock and barrel, what about views. When I load a view: vw = db.view(bigBloatedTable) do all the view's rows get loaded into memory? Or is this data just loaded as needed? Thanking you all muchly once again for your kind help, Cheers David ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question about large databases
On operating systems on which Metakit finds enough support for using memory mappings, it will load the entire database into a memory mapped region of memory, and then the OS takes care of reading/writing the data from/to the actual storage as needed, based on memory accesses. This does mean that for those operating systems, the largest possible storage size is limited by the available virtual memory. --JYL David McNab wrote: Hi, I've got a concern about what might happen if a database file starts getting *really* big - like say 20MB or more. When I first create the storage object: db = metakit.storage(my-big-db.mk, 1) Does the entire contents of the database get loaded into memory, or is database content pulled in as needed? If the whole database doesn't get loaded into memory lock, stock and barrel, what about views. When I load a view: vw = db.view(bigBloatedTable) do all the view's rows get loaded into memory? Or is this data just loaded as needed? Thanking you all muchly once again for your kind help, Cheers David ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question - editable joins?
On 17 Jul 2003 at 23:42, David McNab wrote: I note that normal views are of type 'PyView', whereas views returned by a join() are of type 'PyROViewer', and that I can't edit any of the data in these views. Is there any way I can edit the values in a join()-derived view, without having to hunt down the corresponding rows/fields in the original view(s)? (Using Mk4py) No. The reason filter returns an index view instead of doing it's own remapwith is so code can get to the proper rows in the base view. You might also look at view.map(). The approach used by filter and map (passing in a callable) generally result in faster code than doing the looping in Python. -- Gordon http://www.mcmillan-inc.com/ ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question on Views
Hi Barbara,
The problem might be that you are passing vJustMeasured and vTreeTable
to CAreaBased by value. This makes a copy of the objects, which may
not be allowed. You might be better passing a reference:
CAreaBased::CAreaBased(c4_View vJustMeasured, c4_View vTreeTable)
{
int nRows = vTreeTable.GetSize();
if(nRows 0) vJustMeasured.SetSize();
}
Copying the c4_View might be the cause of your error and is definitely
quite inefficient!
Cheers,
Steve.
On Friday, Jan 24, 2003, at 19:43 Europe/London, Barbara Menzel wrote:
Here's an example of the type of constructor code I'm getting errors
on...
CAreaBased::CAreaBased(c4_View vJustMeasured, c4_View vTreeTable)
{
int nRows = vTreeTable.GetSize();
if(nRows 0) vJustMeasured.SetSize();
}
It is appropriately prototyped in the header file and when the object
is
instantiated, there would be a view (vJustMeasured) equal in size to
the
tree table which is then visible to the class CAreaBased. Using
c4_View in
the constructor parameter list is what generates the unrecognized type
error.
(Embedded image moved to file: pic11773.gif)
Jean-Claude
Wippler To: Metakit
mailing list [EMAIL PROTECTED]
[EMAIL PROTECTED] cc:
Sent by: Subject: Re: [Metakit]
Question on Views
metakit-admin@eq
ui4.com
01/22/2003 02:12
AM
Barbara Menzel wrote:
We are using MetaKit with Visual C++. Often, we find there is a need
to
initialize or perform some initial action on a view within a new
class.
I've tried passing the view into the object via the constructor and
get
several compile errors, primarily, c4_view is not a recognized type.
That's a typo, I assume: c4_View, not c4_view, right?
However, passing a view as a parameter in a member function, there are
no
errors and everything works fine. The view can even be updated within
the
member function and returned with the updates included. Has anyone
tried
this or something similar with any success?
Could you post a brief extract of the code you would like to get
working? I'm having trouble understanding exactly what part is not
doing what you expect.
-jcw
___
metakit mailing list - [EMAIL PROTECTED]
http://www.equi4.com/mailman/listinfo/metakit
pic11773.gif
___
metakit mailing list - [EMAIL PROTECTED]
http://www.equi4.com/mailman/listinfo/metakit
Re: [Metakit] Question on Views
Barbara Menzel wrote: We are using MetaKit with Visual C++. Often, we find there is a need to initialize or perform some initial action on a view within a new class. I've tried passing the view into the object via the constructor and get several compile errors, primarily, c4_view is not a recognized type. That's a typo, I assume: c4_View, not c4_view, right? However, passing a view as a parameter in a member function, there are no errors and everything works fine. The view can even be updated within the member function and returned with the updates included. Has anyone tried this or something similar with any success? Could you post a brief extract of the code you would like to get working? I'm having trouble understanding exactly what part is not doing what you expect. -jcw ___ metakit mailing list - [EMAIL PROTECTED] http://www.equi4.com/mailman/listinfo/metakit
[Metakit] Question on Views
Hello! We are using MetaKit with Visual C++. Often, we find there is a need to initialize or perform some initial action on a view within a new class. I've tried passing the view into the object via the constructor and get several compile errors, primarily, c4_view is not a recognized type. However, passing a view as a parameter in a member function, there are no errors and everything works fine. The view can even be updated within the member function and returned with the updates included. Has anyone tried this or something similar with any success? (Embedded image moved to file: pic07483.gif) attachment: pic07483.gif
