Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

Just for the record, my 160x speed improvement came with pretty naive code. I loop to a record, build the text I need for that row, append the block to the BLOB, clear the text and hit the next row. All-in-all, only a couple of lines of extra code. After what everyone has contributed and

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

> Le 18 sept. 2018 à 18:50, Dani Beaubien a > écrit : > > This has been a very interesting discussion. I decided to test the various > techniques using the code I pasted below. > [...] Thanks for testing :-) Some thoughts… - I had surprises with tests using a single string of the same

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

Thanks to John for adding writing to disk to Dani's test set. You make some good points about documents in the modern world. You can probably append to a document all day long and never see any change in speed. Years ago, the legend was that you didn't have to buffer before calling *SEND PACKET*

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

Dani, Thanks for taking the time to test this and post your results. I looked at some document generation routines where I was caching some text before writing it out with SEND PACKET (to avoid lots of calls to SEND PACKET with small amounts of text). I changed to a blob cache and it seemed to

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

Dani, Thank you, for a very large value of "thank you." I woke up this morning all excited to write some tests, and you had already done the work. It's really great that you took the time to do the tests, to share your conclusions and your test method. Thank you! In my case, I'm not so kind or

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

This has been a very interesting discussion. I decided to test the various techniques using the code I pasted below. Essentially 6 different ways of building up a large text variable. 1) simplest method, add text directly to the variable 2) use a 2048 buffer, add text to a buffer, once that hits

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

> Le 17 sept. 2018 à 16:15, Bart Davis via 4D_Tech <4d_tech@lists.4d.com > > a écrit : > > No need to write to a file. Appending text to a text variable is very slow, > but using TEXT TO BLOB($textToAdd;$blob;UTF8 text without length;*) is very > fast. Give

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

> Bernd, > > Thanks for the report, it's interesting and believable. In this case, I > have to get the data into memory to send it... I guess I could write to a > file, load that and then send it. Interesting concept, thanks! No need to write to a file. Appending text to a text variable is very

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

David, (not on v17) I've played around with large text, mostly for testing purposes. large texts (>150meg) - very easily cause massive slow downs when trying to append - can easily crash/hang 4D. While the *limit* on text maybe 2gig (+/-) my experience is that you can not get anywhere near that

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Bernd, Thanks for the report, it's interesting and believable. In this case, I have to get the data into memory to send it... I guess I could write to a file, load that and then send it. Interesting concept, thanks! On Mon, Sep 17, 2018 at 4:53 PM Bernd Fröhlich via 4D_Tech <

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

David Adams: > I'm checking for the simplest, least difficult default as that should work in > the vast majority of cases. Here are my two Eurocents: it does not matter how you collect the data, but concatenating the text in memory vs. writing to a file makes a HUGE difference when you have

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

On Sun, Sep 16, 2018 at 9:09 PM Olivier Flury via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Not really an answer to your question but a more general observation: > > Very often, when I have to do an export function towards other systems, I > end up in doing a bulk export. > > Either because I am

AW: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Not really an answer to your question but a more general observation: Very often, when I have to do an export function towards other systems, I end up in doing a bulk export. Either because I am lazy or the other side is lazy or both sides are lazy. It's easy, just send everything. But

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Hello people, thanks for all of the suggestions and feedback. Several of you pointed out that my thinking was in the fuzzy-to-nonsense range. I hate you all. So much. Not really, the comments were completely on the mark. I'm trying to find a decent default option that will work over a wide range

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Following up on Jim’s and other responses I would venture to say that one good solution memory and performance wise would be: - loop thru records using NEXT RECORD, or GOTO SELECTED RECORD - preallocate a BLOB to build your text/json; say 5MB is your limit, or maybe 50Mb, or 500Mb, you pick your

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

On Sep 15, 2018, at 12:07 AM, David Adams via 4D_Tech <4d_tech@lists.4d.com> wrote: > > Short version: > I need to load some fields from records into a big text thingy. > > The code runs on the server-side only. > > I'm keen to preserve RAM. [snip] > TL;DR version > I'm working in V17 and

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Hey David, You know, you say that RAM is the primary limiting factor but then also say you're only interested in actual timing tests. Umm. Seems like the coding version of the the building contractor talking to a client: You can have it fast You can have it cheap You can have it done well Pick

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

With a little up front work, you can create a nice wrapper for SELECTION TO ARRAY/SELECTION RANGE TO ARRAY where you just pass an array of field pointers. You can use a process 2D array of each type to dynamically allocate/deallocate the necessary arrays and just manipulate the pointers. Some

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Thanks Keith, that all sounds pretty right to me too. And, yes, I'm chunking the pushes into batches to avoid a truly massive text object, when necessary. I don't know about with SQL Server and don't remember about MySQL, but with Postgres it's important to avoid single row inserts where possible.

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Hi GOTO SELECTED RECORD (or any ORDA equivalent of navigating record by record) will be slower than S2A but will take less RAM. That's based on actual experience. Building up a single big text thingy will have RAM (and eventually potentially performance) issues if using straight string/text

Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

Short version: I need to load some fields from records into a big text thingy. The code runs on the server-side only. I'm keen to preserve RAM. What are the trade-offs in V17 between *GOTO SELECTED* record and *SELECTION TO ARRAY*? I've been using *SELECTION TO ARRAY*, but it's hard to read,