Re: Long running reports to disk intermittently disrupted and fail

[Arnaud de Montard via 4D_Tech]

> Le 20 avr. 2020 à 08:06, Chip Scheide via 4D_Tech <4d_tech@lists.4d.com> a 
> écrit :
> 
> :)
> that was my point.
> 
> if the file is created on the client, 
> new record
> document to [table]blob
> save record

Hi Chip, 
the problem with that technique is that [table] will probably inflate/deflate: 
save, process, delete… Even if the documents have to be processed later, I 
think I'd move them to a folder on server rather than in the data. 

-- 
Arnaud de Montard 



**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Chip Scheide via 4D_Tech]

:)
that was my point.

if the file is created on the client, 
new record
document to [table]blob
save record

blob is now on the server.
deamon running on server looks for new records in [table]
when found, it extracts it/them.

alternatively
after saving the new record on the client
execute on server(extract_file_method)
 
> 
> That is more complicated than just calling an "execute on server" method.

Hell is other people 
 Jean-Paul Sartre
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Bernd Fröhlich via 4D_Tech]

Chip Scheide:

>> Now I create the document as before, do a document to blob on the 
>> client and call an "execute on server"-method that sends the blob to 
>> the server where it gets written to the directory with blob to 

> why not just save the blob in a (temporary) record?
> the blob is auto transmitted to the server via 4D.
> 
> and you can then (on the server) do a blob to document if necessary

That is more complicated than just calling an "execute on server" method.
Maybe I was not clear: with the "execute on server" method attribute set, the 
parameters (in this case the blob) is also transferred to the server 
automatically.
No need for records.

Greetings from Germany,
Bernd Fröhlich
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Re: Long running reports to disk intermittently disrupted and fail

[Chip Scheide via 4D_Tech]

why not just save the blob in a (temporary) record?
the blob is auto transmitted to the server via 4D.

and you can then (on the server) do a blob to document if necessary

> Now I create the document as before, do a document to blob on the 
> client and call an "execute on server"-method that sends the blob to 
> the server where it gets written to the directory with blob to 
> document.

Hell is other people 
 Jean-Paul Sartre
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

On Apr 17, 2020, at 12:15 PM, John J Foster  wrote:

Hey Kirk and Miyako,

I like the idea of having a copy at  months end and then moving it.

This client is quite picky about what I can do and what I can’t. Been working 
with them since 1994-1995 I think.

Anyway thanks for the ideas.

Appreciate,
John…

> You can always write the report to a temp file somewhere on the server.
> Create a directory in the database folder, for example. Once the file is
> created you can blob it up and move it back to the client. It will still be
> faster I bet.
> 
> Or considering the fact it's so time sensitive and can't be recreated
> saving a copy on the server might be an actual good idea.

> 
> 
> 
> also, don't forget to compress the BLOB. 
> text files compact pretty well.
> 
>> On Apr 17, 2020, at 7:17, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com 
>> > wrote:
>> Once the file is created you can blob it up and move it back to the client. 
>> It will still befaster I bet.
> 


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Arnaud,

No worries not on a MacOS.

John…



> Unfortunately on macOS it's not enough, I sometimes have the shared volume 
> when I ask "are you here", a while after it's gone. I use an applescript that 
> is called each time 4D sees a volume is gone to mount it, but same problem 
> can occur. I feel like fighting with windmills. I hope a more recent MacOS 
> will be better, something's wrong between macOS and smb shared volumes. 
> Everything's fine when I copy another way (ftp, for example.)
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Re: Long running reports to disk intermittently disrupted and fail

[Bernd Fröhlich via 4D_Tech]

Arnaud de Montard:

> Unfortunately on macOS it's not enough, I sometimes have the shared volume 
> when I ask "are you here", a while after it's gone. I use an applescript that 
> is called each time 4D sees a volume is gone to mount it, but same problem 
> can occur. I feel like fighting with windmills. I hope a more recent MacOS 
> will be better, something's wrong between macOS and smb shared volumes. 
> Everything's fine when I copy another way (ftp, for example.)


I switched some related code here recently.
Used to generate a document on the client and copy it to a mounted share which 
was on the server.
Now I create the document as before, do a document to blob on the client and 
call an "execute on server"-method that sends the blob to the server where it 
gets written to the directory with blob to document.
Works much better and has the added benefit that it now works for WAN-Clients 
without the need to have the volume mounted.

Greetings from Germany,
Bernd Fröhlich
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Arnaud de Montard via 4D_Tech]

> Le 16 avr. 2020 à 22:33, John J Foster via 4D_Tech <4d_tech@lists.4d.com> a 
> écrit :
> 
> Hi Arnaud,
> 
> Yes that path would have to tested and if not found then a report or email 
> sent alerting them.

Unfortunately on macOS it's not enough, I sometimes have the shared volume when 
I ask "are you here", a while after it's gone. I use an applescript that is 
called each time 4D sees a volume is gone to mount it, but same problem can 
occur. I feel like fighting with windmills. I hope a more recent MacOS will be 
better, something's wrong between macOS and smb shared volumes. Everything's 
fine when I copy another way (ftp, for example.)

-- 
Arnaud de Montard 





**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Keisuke Miyako via 4D_Tech]

also, don't forget to compress the BLOB. 
text files compact pretty well.

> On Apr 17, 2020, at 7:17, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> Once the file is created you can blob it up and move it back to the client. 
> It will still befaster I bet.

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Kirk Brooks via 4D_Tech]

John,
On Thu, Apr 16, 2020 at 1:39 PM John J Foster via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Right now there’s a specific computer where these reports are run from at
> the end of each month. And yes they need to run at the end of the last day
> of the month and before any new data has been added due to the nature of
> the required field and printed reports.
>

You can always write the report to a temp file somewhere on the server.
Create a directory in the database folder, for example. Once the file is
created you can blob it up and move it back to the client. It will still be
faster I bet.

Or considering the fact it's so time sensitive and can't be recreated
saving a copy on the server might be an actual good idea.

-- 
Kirk Brooks
San Francisco, CA
==
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hi Randy,

Good to know.

Since I don’t have access, remote or otherwise, I’ll have to teach the client 
how to use that.

Anyway, it doe sound like it’s doable. And worst case the document is still 
created it just might not be moved if the drive cannot be found for there’s an 
error during the attempt to move.

Appreciate,
John…


> You should be able to, but the path may surprise you.
> 
> I have a client who has a networked Drobo and it has volumes called FTP and 
> Backup.  However, to access the drive, we have to use:
> 
> /Volumes/Drobo5N/drb153401a00827/1/FTP/
> 
> Go figure.  If you drag the volume into a terminal window, you’ll get the 
> path you have to use.
> 
> 
> Randy Kaempen
> Intellex Corporation
> 

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Randy Kaempen via 4D_Tech]

John,

> On Apr 16, 2020, at 3:47 PM, John J Foster via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Agreed!
> 
> I will try and change them if I can figure how to test the Server to see if 
> it can see the network drive.
> 
> I would imagine that it can but never assume…

You should be able to, but the path may surprise you.

I have a client who has a networked Drobo and it has volumes called FTP and 
Backup.  However, to access the drive, we have to use:

/Volumes/Drobo5N/drb153401a00827/1/FTP/

Go figure.  If you drag the volume into a terminal window, you’ll get the path 
you have to use.


Randy Kaempen
Intellex Corporation

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Randy,

Agreed!

I will try and change them if I can figure how to test the Server to see if it 
can see the network drive.

I would imagine that it can but never assume…

I’ll see if I can find out.

Appreciate,
John…


> On Apr 16, 2020, at 12:00 PM, 4d_tech-requ...@lists.4d.com wrote:
> 
>> On Apr 16, 2020, at 8:41 AM, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com 
>> > wrote:
>> 
>> John,
>> 
>> Better yet - re-write the file generation to run on the server. In my
>> experience when a report like that takes so long to run it's due to moving
>> lots of data back and forth. Move the code that actually creates the file
>> to a method that can execute on the server.
> 
> 
> I agree.  Definitely run the process on the server.  Big amounts of data 
> process _much_ faster there.  Make the one file from the server and then move 
> it where you need it.
> 
> 
> Randy Kaempen
> Intellex Corporation

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Jeff,

Been a while since I had to use Process Monitor as part of a debugging process.

I don't have access to the Windows Sever that runs our 4D Server App. But I 
could ask my contact what the setup is. I pretty much have to try and debug 
with the non-IT folks.

Anyway good idea.

Thanks,
John…



> If it's on a Windows server, check out Process Monitor (procmon). It runs in 
> the background and gives you a log of every filesystem activity. It's pretty 
> amazing the stuff it reveals... but it should be able to track down who/when 
> deleted your file:
> 
> https://docs.microsoft.com/en-us/sysinternals/downloads/procmon 
> 
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Kirk,

> I agree with Jim

Ditto!

Yes I have been exploring the idea of moving the code to run on the Server. 
It’s easy s I only need to set a property. But then I still need the Server to 
be able to see the paths to where the text files need to be moved to. I have no 
access to their Server so I can really test it so that’s a challenge.

Right now there’s a specific computer where these reports are run from at the 
end of each month. And yes they need to run at the end of the last day of the 
month and before any new data has been added due to the nature of the required 
field and printed reports.

Appreciate,
John...

> On Apr 16, 2020, at 12:00 PM, 4d_tech-requ...@lists.4d.com wrote:
> I agree with Jim: separate the creation of the file from moving the file to
> the network. A lot has changed in the last 20 years or so with the way
> network access is managed and old code like this that looks at a network
> address as equivalent to a local disk are problematic. We ran into this on
> another project where the 4D app is doing a lot of file manipulation on a
> networked drive.
> 
> Better yet - re-write the file generation to run on the server. In my
> experience when a report like that takes so long to run it's due to moving
> lots of data back and forth. Move the code that actually creates the file
> to a method that can execute on the server.

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hi Arnaud,

Yes that path would have to tested and if not found then a report or email sent 
alerting them.

John…

>> Le 15 avr. 2020 à 23:35, John J Foster via 4D_Tech <4d_tech@lists.4d.com 
>> > a écrit :
>> 
>> I do think writing the file locally and then moving it makes a lot of sense. 
>> Right now they choose the path to save files to but it would b easy enough 
>> to create it locally move doc into the path.
> 
> I do like this and strongly recommend it. But don't expect it guarantees the 
> target shared folder will be here when you move files from local to shared, 
> you just reduce the time you need it to be here. Shared volumes on a 4D 
> server are my personal nightmare, specially on mac. 
> 
> -- 
> Arnaud de Montard 

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Randy Kaempen via 4D_Tech]

> On Apr 16, 2020, at 8:41 AM, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> John,
> 
> Better yet - re-write the file generation to run on the server. In my
> experience when a report like that takes so long to run it's due to moving
> lots of data back and forth. Move the code that actually creates the file
> to a method that can execute on the server.


I agree.  Definitely run the process on the server.  Big amounts of data 
process _much_ faster there.  Make the one file from the server and then move 
it where you need it.


Randy Kaempen
Intellex Corporation

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Kirk Brooks via 4D_Tech]

John,
I agree with Jim: separate the creation of the file from moving the file to
the network. A lot has changed in the last 20 years or so with the way
network access is managed and old code like this that looks at a network
address as equivalent to a local disk are problematic. We ran into this on
another project where the 4D app is doing a lot of file manipulation on a
networked drive.

Better yet - re-write the file generation to run on the server. In my
experience when a report like that takes so long to run it's due to moving
lots of data back and forth. Move the code that actually creates the file
to a method that can execute on the server.

On Wed, Apr 15, 2020 at 10:10 AM John J Foster via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hey All,
>
> When it rains it pours…
>
> 4D v17.2 64bit Windows
> Windows 10 Pro
>
> We have some reports that are run in a batch overnight. Have to because
> they take that long to run all of them. Anyway, since we’ve upgraded from
> Windows 7 to Windows 10 there have been I’m guessing timeouts on some of
> these larger reports.
>
> The report is running along and then we get the error: "executing the
> method “EXP_CumulativeClaims” at the lilne number 63. The .txt document
> cannot be written. File not found.”
>
> This is line #63:
> SEND PACKET($fileRef;$packet)
>
> It varies when it quits. It’s usually saved to a network drive and IT says
> nothing is happening to disrupt the connection. And there are other reports
> where this happens as well.
>
> The text file report size - when it works - is large:
>
> Reserve file: 831,801kb
> Claims file: 26,959kb
>
> So these reports were created way back in 2004. The number of records has
> grown dramatically over the years. Reserves is up beyond 2.3 million
> records.
>
> Is it possible that this is a client-server timeout issue? If so, how to
> fix?
>
> Clients IT claims the network drive is always available. Is there a way to
> “know” for sure and test each time?
>
> Is there a way to capture debug info which might reveal the cause? Would
> logging using 4D’s component help?
>
> Would it be helpful and speedier to redesign the reports so they are not
> sending out each row but maybe save each row to a text variable, say 100 or
> 1000 rows and then send packet?
>
> We are open to all ideas.
>
> Appreciate,
> John…
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **



-- 
Kirk Brooks
San Francisco, CA
==
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Arnaud de Montard via 4D_Tech]

> Le 15 avr. 2020 à 23:35, John J Foster via 4D_Tech <4d_tech@lists.4d.com> a 
> écrit :
> 
> I do think writing the file locally and then moving it makes a lot of sense. 
> Right now they choose the path to save files to but it would b easy enough to 
> create it locally move doc into the path.

I do like this and strongly recommend it. But don't expect it guarantees the 
target shared folder will be here when you move files from local to shared, you 
just reduce the time you need it to be here. Shared volumes on a 4D server are 
my personal nightmare, specially on mac. 

-- 
Arnaud de Montard 




**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Jim,

> If your machine is on an SSD…

That makes sense to my very simple mind! 

I don’t think in all my years of 4D dating back to beta Silver Surfer days I 
have ever needed a routine where buffering using a BLOB was required. That’s 
amazing! Anyway...

Thanks again for your advice and for the code!

Makes it so much easier to think it through (see and build upon).

Appreciate,
John…


> On Apr 15, 2020, at 3:19 PM, James Crate  wrote:
> 
> On Apr 15, 2020, at 4:35 PM, John J Foster via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
>> 
>> I do believe that creating it locally and moving to the network drive can be 
>> more efficient. I’m not sure how much speed different there would be.
>> 
>>> using TEXT TO BLOB with the offset parameter
>> 
>> I’m investigating this command and seeing if I can understand how to use it.
> 
> If your machine is on an SSD, there will probably be little difference 
> between writing each row with SEND PACKET or buffering using a BLOB. If you 
> want to buffer using a blob, the general strategy is like this:
> 
> // Calculate rough target size of blob plus a little extra. If 2.3M rows was 
> 830MB, you are at roughly 2700 rows per MB, so maybe calculate 1MB per 2500 
> rows
> $blobSize:=(number of records / 2500)*(1024*1024)
> SET BLOB SIZE($buffer;$blobSize)
> $offset:=0
> For ($i;1;number of records)
>  // build line into $line
>  // make sure we have enough space in the blob
>  If (Blob size($buffer)<($offset+Length($line))
>$blobSize:=$blobSize+(Length($line)*2000)// increase by enough that we 
> don’t have to do it often
>SET BLOB SIZE($buffer;$blobSize)
>  End if
>  TEXT TO BLOB($line;$buffer;UTF8 text without length;$offset)
> End for
> // Truncate blob to correct length
> SET BLOB SIZE($buffer;$offset) // blobs are zero-indexed
> 
> 
> Jim Crate
> 

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[James Crate via 4D_Tech]

On Apr 15, 2020, at 4:35 PM, John J Foster via 4D_Tech <4d_tech@lists.4d.com> 
wrote:
> 
> I do believe that creating it locally and moving to the network drive can be 
> more efficient. I’m not sure how much speed different there would be.
> 
>> using TEXT TO BLOB with the offset parameter
> 
> I’m investigating this command and seeing if I can understand how to use it.

If your machine is on an SSD, there will probably be little difference between 
writing each row with SEND PACKET or buffering using a BLOB. If you want to 
buffer using a blob, the general strategy is like this:

// Calculate rough target size of blob plus a little extra. If 2.3M rows was 
830MB, you are at roughly 2700 rows per MB, so maybe calculate 1MB per 2500 rows
$blobSize:=(number of records / 2500)*(1024*1024)
SET BLOB SIZE($buffer;$blobSize)
$offset:=0
For ($i;1;number of records)
  // build line into $line
  // make sure we have enough space in the blob
  If (Blob size($buffer)<($offset+Length($line))
$blobSize:=$blobSize+(Length($line)*2000)// increase by enough that we 
don’t have to do it often
SET BLOB SIZE($buffer;$blobSize)
  End if
  TEXT TO BLOB($line;$buffer;UTF8 text without length;$offset)
End for
// Truncate blob to correct length
SET BLOB SIZE($buffer;$offset) // blobs are zero-indexed


Jim Crate

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Neil,

That’s interesting. This client, although in 4D v17.2 (yes unicode), hates 
upgrading. So I will have to code in v17 efficiency.

I think I’d be find with the slowness if I can figure out why the routine 
either stalls or timeout or suddenly cannot find the disk.

I do think writing the file locally and then moving it makes a lot of sense. 
Right now they choose the path to save files to but it would b easy enough to 
create it locally move doc into the path.

Thanks for the ideas!

John…


> On Apr 15, 2020, at 1:48 PM, Neil Dennis  wrote:
> 
>> The fastest way to build the output is probably with a pre-sized blob, and 
>> using TEXT TO BLOB with the offset parameter. Pushing elements into a 
>> collection and then using collection.join() is also pretty fast, but again 
>> you’d have to test to see how large a text variable can actually be, and 
>> whether the limit was large enough both interpreted and compiled. If you 
>> don’t have enough RAM for a 1GB blob, smaller blobs may work with SEND 
>> PACKET.
> 
> I ran some speed tests using 4D v18 on Windows 10. The results were 
> interesting… in times past it was much faster to run a report using an 
> internal buffer of some sort such as a text or blob variable. My resent tests 
> showed it was faster to use SEND PACKET without buffering anything.
> 
> Again I was testing with 4D v18 and results may vary.
> 
> I’m assuming you are using unicode on your app? I think you can still do 
> that, but I’m not sure. If you are not in unicode you should switch.
> 
> With unicode on a text can be very large. For imports I just use Document to 
> Text. When exporting, large text vars can slow down as the resizing of them 
> becomes slow. You could try filling a blob, but like I said SEND PACKT seemed 
> to me to be the fastest option.
> 
> As was mentioned be sure to build your document local in the temp folder then 
> move it to its home on the network. Writing across the network to create a 
> file is slow and prone to error.
> 
> Neil

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[Neil Dennis via 4D_Tech]

> The fastest way to build the output is probably with a pre-sized blob, and 
> using TEXT TO BLOB with the offset parameter. Pushing elements into a 
> collection and then using collection.join() is also pretty fast, but again 
> you’d have to test to see how large a text variable can actually be, and 
> whether the limit was large enough both interpreted and compiled. If you 
> don’t have enough RAM for a 1GB blob, smaller blobs may work with SEND PACKET.

I ran some speed tests using 4D v18 on Windows 10. The results were 
interesting… in times past it was much faster to run a report using an internal 
buffer of some sort such as a text or blob variable. My resent tests showed it 
was faster to use SEND PACKET without buffering anything.

Again I was testing with 4D v18 and results may vary.

I’m assuming you are using unicode on your app? I think you can still do that, 
but I’m not sure. If you are not in unicode you should switch.

With unicode on a text can be very large. For imports I just use Document to 
Text. When exporting, large text vars can slow down as the resizing of them 
becomes slow. You could try filling a blob, but like I said SEND PACKT seemed 
to me to be the fastest option.

As was mentioned be sure to build your document local in the temp folder then 
move it to its home on the network. Writing across the network to create a file 
is slow and prone to error.

Neil


--
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[John J Foster via 4D_Tech]

Hey Jim,

Yea you could be right!

I do believe that creating it locally and moving to the network drive can be 
more efficient. I’m not sure how much speed different there would be.

> using TEXT TO BLOB with the offset parameter

I’m investigating this command and seeing if I can understand how to use it.

I’ve only used/seen this command in Tony’s R. BB_Decrypt/BB_Encrypt methods.

Nothing in the KB that I can find with a real world example.

Anyway, I think that’s a road I have to go down.

Appreciate,
John…


> On Apr 15, 2020, at 1:25 PM, James Crate  wrote:
> 
> In the time it takes to troubleshoot this, you could probably rewrite the 
> report to be much faster and more reliable. 
> 
> I would start by writing the file to a local temp file, and then copying that 
> file to the network drive when finished. This reduces the impact of network 
> issues both for completing the operation and the network share, and makes it 
> much more likely you only get a completed version on the network share for 
> external consumption.
> 
> The fastest way to build the output is probably with a pre-sized blob, and 
> using TEXT TO BLOB with the offset parameter. Pushing elements into a 
> collection and then using collection.join() is also pretty fast, but again 
> you’d have to test to see how large a text variable can actually be, and 
> whether the limit was large enough both interpreted and compiled. If you 
> don’t have enough RAM for a 1GB blob, smaller blobs may work with SEND PACKET.
> 
> Jim Crate

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Long running reports to disk intermittently disrupted and fail

[James Crate via 4D_Tech]

On Apr 15, 2020, at 2:20 PM, John J Foster via 4D_Tech <4d_tech@lists.4d.com> 
wrote:
> 
> 4D v17.2 64bit Windows
> Windows 10 Pro
> 
> We have some reports that are run in a batch overnight. Have to because they 
> take that long to run all of them. Anyway, since we’ve upgraded from Windows 
> 7 to Windows 10 there have been I’m guessing timeouts on some of these larger 
> reports.
> 
> The report is running along and then we get the error: "executing the method 
> “EXP_CumulativeClaims” at the lilne number 63. The .txt document cannot be 
> written. File not found.”
> [snip]
> 
> The text file report size - when it works - is large:
> 
> Reserve file: 831,801kb
> Claims file: 26,959kb
> [snip]
> 
> Would it be helpful and speedier to redesign the reports so they are not 
> sending out each row but maybe save each row to a text variable, say 100 or 
> 1000 rows and then send packet?
> 
> We are open to all ideas.

In the time it takes to troubleshoot this, you could probably rewrite the 
report to be much faster and more reliable. 

I would start by writing the file to a local temp file, and then copying that 
file to the network drive when finished. This reduces the impact of network 
issues both for completing the operation and the network share, and makes it 
much more likely you only get a completed version on the network share for 
external consumption.

The fastest way to build the output is probably with a pre-sized blob, and 
using TEXT TO BLOB with the offset parameter. Pushing elements into a 
collection and then using collection.join() is also pretty fast, but again 
you’d have to test to see how large a text variable can actually be, and 
whether the limit was large enough both interpreted and compiled. If you don’t 
have enough RAM for a 1GB blob, smaller blobs may work with SEND PACKET.

Jim Crate

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**