Thanks Keith, that all sounds pretty right to me too. And, yes, I'm chunking the pushes into batches to avoid a truly massive text object, when necessary. I don't know about with SQL Server and don't remember about MySQL, but with Postgres it's important to avoid single row inserts where possible. Postgres is absolutely fanatical about integrity, so each operation is automatically wrapped in a transaction. Pushing rows one-by-one can, therefore, be surprisingly slow. So I bundle multiple rows into a mutli-value insert, and then several of those into a transaction. It goes pretty quickly and for smaller pushes, it probably doesn't matter how I build up the text, it just isn't that big. (Note for the PG people in the crowd, I'm using an UPSERT, so COPY TO isn't an option.)
Regarding the ever-resizing text block, I don't know how that behaves today. I used to know...and knew the sort of crazy stuff you could do to improve performance. Like, pre-size a giant block, keep track of the last byte written and then write into bytes by hand. So, you didn't resize the block as you went along. You might need a final trim or an expansion...but you pretty much kept the block as a set size and then write into it by direct character reference, etc. It's a lot more work and I have no idea how modern versions of 4D on modern versions of Mac/Win handle all of this. I have a notion in the back of my mind that all of that kind of byte tracking stuff isn't too useful any more. I notice that both 4D and Postgres don't have VARCHAR types internally, it's all variable length text. Haven't testing anything. Oh, we don't have any double-byte data to worry about. Thanks again! ********************************************************************** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:[email protected] **********************************************************************
