This has been a very interesting discussion. I decided to test the various
techniques using the code I pasted below.
Essentially 6 different ways of building up a large text variable.
1) simplest method, add text directly to the variable
2) use a 2048 buffer, add text to a buffer, once that hits 2048 bytes copy it
to the text var, clear and repeat
3) use a 4096 buffer, add text to a buffer, once that hits 4096 bytes copy it
to the text var, clear and repeat
4) Use TEXT TO BLOB just appending and then copy to text var at end
5) Use TEXT TO BLUB but with a 2048 buffer and only copy to blob once the
buffer hits 2048 bytes
6) Use TEXT TO BLUB but with a 4096 buffer and only copy to blob once the
buffer hits 4096 bytes
The code loops through target text var sizes up to 10MB in 128k increments.
Each way is run 5 times and then the result is averaged and appended to an
array. Once the 1-3 techniques exceed 600ms I stop running them since they
behave so badly.
At the end of the test the results are placed in the clipboard (for pasting
into Excel and then graphed) and in an Alert.
Based on this test, technique #5/#6 are the fastest by far.
Techniques 4-6 seem to have linear performance.
For 10 MB text var, the #4 came in at 1613ms and #5 came in at 290.6ms
Very interesting. I am currently using techniques #3 when I am creating exports
and reports to disk. Going to switch to the #5.
Downsize of the blob techniques is that you need double the amount of memory if
you need to convert the blob back to a text variable like I did in this
experiment.
Dani Beaubien
Open Road Development
ARRAY LONGINT($sizes;0)
ARRAY REAL($r1;0)
ARRAY REAL($r2;0)
ARRAY REAL($r3;0)
ARRAY REAL($r4;0)
ARRAY REAL($r5;0)
ARRAY REAL($r6;0)
C_BLOB($blob)
C_TEXT($textToAdd;$result)
$textToAdd:="123456789 "
C_BOOLEAN($skip1;$skip3;$skip4)
C_LONGINT($targetVarSize;$maxSize;$outerLoopMax)
$targetVarSize:=0
$maxSize:=1024*1024*10 // 10 MB
$outerLoopMax:=5
C_LONGINT($vs1;$ve1)
C_LONGINT($vs2;$ve2)
C_LONGINT($vs3;$ve3)
C_LONGINT($vs4;$ve4)
C_LONGINT($vs5;$ve5)
C_LONGINT($vs6;$ve6)
C_TEXT($tmpTxt)
C_LONGINT($innerLoopMax)
Repeat
$targetVarSize:=$targetVarSize+(1024*128)
$innerLoopMax:=Int($targetVarSize/Length($textToAdd))
MESSAGE(" $targetVarSize = "+String($targetVarSize/1024)+" Kb")
If (Not($skip1))
$vs1:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
For ($j;1;$innerLoopMax)
$result:=$result+$textToAdd
End for
End for
$ve1:=Milliseconds
If ((($ve1-$vs1)/$outerLoopMax)>600)
$skip1:=True
End if
End if
If (Not($skip3))
$vs3:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
$tmpTxt:=""
For ($j;1;$innerLoopMax)
$tmpTxt:=$tmpTxt+$textToAdd
If (Length($tmpTxt)>2048)
$result:=$result+$tmpTxt
$tmpTxt:=""
End if
End for
$result:=$result+$tmpTxt
End for
$ve3:=Milliseconds
If ((($ve3-$vs3)/$outerLoopMax)>600)
$skip3:=True
End if
End if
If (Not($skip4))
$vs4:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
$tmpTxt:=""
For ($j;1;$innerLoopMax)
$tmpTxt:=$tmpTxt+$textToAdd
If (Length($tmpTxt)>4096)
$result:=$result+$tmpTxt
$tmpTxt:=""
End if
End for
$result:=$result+$tmpTxt
End for
$ve4:=Milliseconds
If ((($ve4-$vs4)/$outerLoopMax)>600)
$skip4:=True
End if
End if
$vs2:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
SET BLOB SIZE($blob;0)
For ($j;1;$innerLoopMax)
TEXT TO BLOB($textToAdd;$blob;UTF8 text without
length;*)
End for
$result:=BLOB to text($blob;UTF8 text without length)
End for
$ve2:=Milliseconds
$vs5:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
$tmpTxt:=""
SET BLOB SIZE($blob;0)
For ($j;1;$innerLoopMax)
$tmpTxt:=$tmpTxt+$textToAdd
If (Length($tmpTxt)>2048)
TEXT TO BLOB($tmpTxt;$blob;UTF8 text without
length;*)
$tmpTxt:=""
End if
End for
TEXT TO BLOB($tmpTxt;$blob;UTF8 text without length;*)
$result:=BLOB to text($blob;UTF8 text without length)
End for
$ve5:=Milliseconds
$vs6:=Milliseconds
For ($i;1;$outerLoopMax)
$result:=""
$tmpTxt:=""
SET BLOB SIZE($blob;0)
For ($j;1;$innerLoopMax)
$tmpTxt:=$tmpTxt+$textToAdd
If (Length($tmpTxt)>4096)
TEXT TO BLOB($tmpTxt;$blob;UTF8 text without
length;*)
$tmpTxt:=""
End if
End for
TEXT TO BLOB($tmpTxt;$blob;UTF8 text without length;*)
$result:=BLOB to text($blob;UTF8 text without length)
End for
$ve6:=Milliseconds
APPEND TO ARRAY($sizes;$targetVarSize)
APPEND TO ARRAY($r1;Round(($ve1-$vs1)/$outerLoopMax;2))
APPEND TO ARRAY($r2;Round(($ve2-$vs2)/$outerLoopMax;2))
APPEND TO ARRAY($r3;Round(($ve3-$vs3)/$outerLoopMax;2))
APPEND TO ARRAY($r4;Round(($ve4-$vs4)/$outerLoopMax;2))
APPEND TO ARRAY($r5;Round(($ve5-$vs5)/$outerLoopMax;2))
APPEND TO ARRAY($r6;Round(($ve6-$vs6)/$outerLoopMax;2))
Until ($targetVarSize>($maxSize))
C_TEXT($msg)
$msg:="size"
$msg:=$msg+",text"
$msg:=$msg+",text 2048"
$msg:=$msg+",text 4096"
$msg:=$msg+",blob"
$msg:=$msg+",blob 2048"
$msg:=$msg+",blob 4096"
$msg:=$msg+"\r"
For ($i;1;Size of array($sizes))
$msg:=$msg+String(Int($sizes{$i}/1024))+" Kb"
$msg:=$msg+","+String($r1{$i}) // text simple
$msg:=$msg+","+String($r3{$i}) // text 2048 buffer
$msg:=$msg+","+String($r4{$i}) // text 4096 buffer
$msg:=$msg+","+String($r2{$i}) // blob simple
$msg:=$msg+","+String($r5{$i}) // blob 2048 buffer
$msg:=$msg+","+String($r6{$i}) // blob 4096 buffer
$msg:=$msg+"\r"
End for
SET TEXT TO PASTEBOARD($msg)
> On Sep 17, 2018, at 4:42 PM, Arnaud de Montard via 4D_Tech
> <[email protected]> wrote:
>
>
>> Le 17 sept. 2018 à 16:15, Bart Davis via 4D_Tech <[email protected]
>> <mailto:[email protected]>> a écrit :
>>
>> No need to write to a file. Appending text to a text variable is very slow,
>> but using TEXT TO BLOB($textToAdd;$blob;UTF8 text without length;*) is very
>> fast. Give that a try and see the huge speed difference.
>
> You can also append in a text array, then implode that array using that blob
> technique.
> <http://forums.4d.com/Post/FR/15873353/2/17457534#17457534
> <http://forums.4d.com/Post/FR/15873353/2/17457534#17457534>>
> Huge amount of text can be hold this way, very fast.
>
> About the 2Gb limit of text, I noticed the limit in number of chars is about
> the half.
>
> --
> Arnaud de Montard
>
>
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive: http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub: mailto:[email protected]
> **********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive: http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:[email protected]
**********************************************************************
