Hi,
That sounds like a better approach to me too, however if the problem
is because the file is > 2GB (or whatever the limit is on Windows)
then it still won't work.
All the Best
Dave
On 5 Jul 2007, at 13:20, Andre Garzia wrote:
Alejandro,
if this is that kind of XML that has a simple record structure and is
repeated over and over again like a phone book, then why don't you
break it
into smaller edible chunks and insert it into something like SQLite or
Valentina chunk by chunk. By using a RDBMS you'll be able to query
and make
sense of the XML data easily, and those databases will have no problem
dealing with large data sets.
because, even if you manage to load 8gb of data in Rev,
manipulating it will
be kind slow I think, just imagine the loops needed to make cross
references
like find everyone who was born in july and is between 30 and 40
years....
I'd make a little software to go piece by piece inserting this into a
database and then begin again from there.
Andre
On 7/4/07, Alejandro Tejada <[EMAIL PROTECTED]> wrote:
Hi all,
Recently, i was extracting data
from a 8 gigabyte ANSI text file
(a XML customer database), but after
processing approximately 3.5 gigabyte
of data, Revolution quits itself and
Windows XP presents the familiar dialog
asking to notify the Developer of this
error.
The log file that i saved, while using
the stack, shows that after reading character
3,758,096,384 (that is more than 3 thousand million
of characters) the stack could not read anymore
into the XML database and start repeating the
same last line of text that it reads.
Notice that i checked the processor and memory use
with Windows Task Manager and everything was normal.
The stack was using between a 30 to 70 % of processor
and memory use was between 45 MB and 125 MB.
The code used is similar to this:
repeat until tCounter = 8589934592 -- 8 Gigabites
read from file tData from char tCounter for 10000
-- reading 10,000 characters from database
-- these character are placed in the variable: it
put processDATA(it) into tProcessedData
write tProcessedData to tNewFile
put tCounter && last line of it & cr after URL tLOG
add 10000 to tCounter
end repeat
etc...
I have repeated the test at least 3 times :((
and the results are almost the same, with a small
difference between the character where stack quits,
while reading this 8 Gigabyte size XML database.
I have checked for strange characters in that part of
the database, when i splitted the file in many parts,
but have not found any.
Every insight that you could provide to process
this database from start to end is more
than welcome. :)
Thanks in advance.
alejandro
Visit my site:
http://www.geocities.com/capellan2000/
____________________________________________________________
________________________
Sucker-punch spam with award-winning protection.
Try the free Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/features_spam.html
_______________________________________________
use-revolution mailing list
[email protected]
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
_______________________________________________
use-revolution mailing list
[email protected]
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
_______________________________________________
use-revolution mailing list
[email protected]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
