I am using a text array (only 1 element) The array is created on import of the text from the disk file and is NOT resized (except to clear) after this. The element(s) of the array are filled via Receive Packet(Doc;Array_Element;1,500,000,000)
Original Issue - I tired to import a large (50meg) document -- time to import was excessive. The 50 meg file I was trying to import failed to complete after running over night (~16 hours). Resolutions: - New text parsing code -- this code keeps track of how much of the text has been processed via longints, and does NOT manipulate the text (being parsed) directly, only getting substrings from it, and never changing it's size. - New routine to replace/wrap substring - Pass text to be parsed to various methods VIA POINTER* My original code, truncated the source text to the next character after the text to Find (delimiter) This, added SERIOUS time overhead. The import process includes creating slightly more then 332,000 records My new code imports the same 50 megs file (interpretedly) in about 7 minutes. My new code imports the same 50 meg file (compiled) in just over 1 minute. 4D v12 (stand alone), OSX 10.6.8, Mac Mini 8gb RAM, 2.4ghz Core 2 duo, spinning metal hard drive * using the new code, and passing the Source text (to be parsed) as a text parameter my new import routine took about an hour to complete. Chip On Thu, 17 Nov 2016 14:23:06 -0500, Charles Miller wrote: > On Thu, Nov 17, 2016 at 11:43 AM, Arnaud de Montard <[email protected]> wrote: > >> huge text in a text array makes it much easier to manipulate, but, at the >> end, 4D memory is the same. In my example of 6,6Gb file, it was not a >> solution. > > > Also it might be creation of array. Remember that every time 4D resizes an > array especially bigger, it looks for a block of memory that can hold it > all. In effect copying array over and over > > Think of it this way > > Array text($Somtext;0) > insert element($Somtext;size of array(Somtext)+1) > $Somtext{size of array(Somtext)}:="bkjbkjbkb" `copy one > > > insert element($Somtext;size of array(Somtext)+1) > $Somtext{size of array(Somtext)}:="bkjbkjbkb" `copy two > > etc each looking for larger chunks of continuos memory > > Regards > > Chuck > > > -- > ----------------------------------------------------------------------------------------- > Chuck Miller Voice: (617) 739-0306 Fax: (617) 232-1064 > Informed Solutions, Inc. > Brookline, MA 02446 USA Registered 4D Developer > Providers of 4D, Sybase & SQL Sever connectivity > http://www.informed-solutions.com > ----------------------------------------------------------------------------------------- > This message and any attached documents contain information which may be > confidential, subject to privilege or exempt from disclosure under > applicable law. These materials are intended only for the use of the > intended recipient. If you are not the intended recipient of this > transmission, you are hereby notified that any distribution, disclosure, > printing, copying, storage, modification or the taking of any action in > reliance upon this transmission is strictly prohibited. Delivery of this > message to any person other than the intended recipient shall not > compromise or waive such confidentiality, privilege or exemption > from disclosure as to this communication. > ********************************************************************** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:[email protected] > ********************************************************************** ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:[email protected] **********************************************************************
