Re: AW: Question about storing data outside of datafile
I use a record. document to blob then Blob to document - client to server, or server to client. An alternative would be to use FTP and launch this with LEP, but I see that as just another piece to have to manage. Variable to variable - I don't like (see various comments by David Adams) Never tried the resource folder. On Fri, 17 Feb 2017 09:39:19 +0100, flury olivier via 4D_Tech wrote: > Hi Maurice, > > That sounds reasonable. And how do you move the BLOB from the Server > to the Client? > > I see several possibilities: > > - Keep the BLOB in a variable and move the variable from the Server > to the Client. > - Store the BLOB in a field (Record), the Client can access the > Record by e.g. a query. > - Move the Document (not a BLOB) from the Server to the Client by use > of the Resources folder. > > I see advantages/disadvantages for each of the solutions. > > What works well for you? > > Olivier > > -Ursprüngliche Nachricht- > Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von > Maurice Inzirillo - AJAR via 4D_Tech > Gesendet: Freitag, 17. Februar 2017 08:23 > An: 4D iNug Technical <4d_tech@lists.4d.com> > Cc: Maurice Inzirillo - AJAR <maurice.inziri...@ajar.ch> > Betreff: Re: Question about storing data outside of datafile > > Hello Olivier, > > Blob to document And Document to blob with a method set to be > executed on server is an easy and reliable solution ! > > > > Best regards, > > Maurice Inzirillo > -- > AJAR S.A. > > 4D v16: http://www.ajar.ch/ > Wakanda Wednesday Lab : http://www.ajar.ch/wwl > twitter: ajar_info > Tél : +41 (0)323422684 > > > > >> On 17 Feb 2017, at 07:45, flury olivier via 4D_Tech >> <4d_tech@lists.4d.com> wrote: >> >> I am currently thinking of rewriting an existing document management >> system. What is the "best" way to move files from the clients to the >> server and back from the server to the client? >> >> VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a >> kind of caching table (store the last x used files)? ... >> >> Gladly appreciate all sharing of experience. >> >> -Ursprüngliche Nachricht----- >> Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von >> G-Mail via 4D_Tech >> Gesendet: Donnerstag, 16. Februar 2017 23:55 >> An: 4D iNug Technical <4d_tech@lists.4d.com> >> Cc: G-Mail <jody.be...@gmail.com> >> Betreff: Re: Question about storing data outside of datafile >> >>> On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech >>> <4d_tech@lists.4d.com> wrote: >>> >>> Would you like to exchange your know-how? I did similar system, and >>> there are some tricky parts in case of C/S. >> >> >> There is lots of decisions we made for our purposes, and there are >> different ways to do things for sure. We had to meet various >> security regulations therefore we changed some of the ways we did >> things over the 20 or so years that we had built it. Of course new >> technologies came along too that permitted us to change that up. >> >> Is there something specific that you are wanting to discuss? >> >> >> >> Jody Bevan >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http:
Re: Question about storing data outside of datafile
one caveat:
OS X 10.6 and 4D v12
I have not tested on newer systems
I have had problems with very large files (500 meg and over).
Also - beside to "clean" your file names before trying to save them,
this is especially true if your system(s) run ion a mixed platform
environment. Windows can be very picky about file names.
here is some code to 'clean' file names.
(I hope any of the called other utility methods are self explanatory)
//Project Method: utl_File_Cleanup_Filenames
// $1 - text - file NAME (only) to check for bad characters
//looks for, and changes "bad" characters (determined by windows
limits) to a dash ("-")
//RETURNS - text - altered filename
//• Created 6-15- 09 by Chip
//• Updated 1/11/17 by Chip - replaced char 202 with char 160 for
non-breaking space
//difference between ascii and unicode, also check for trailing
period
C_TEXT($1;$Source;$Bad_Characters;$0)
C_TEXT($Replacement_Character)
C_LONGINT($Bad_Char_Count;$i)
$Source:=$1
$Bad_Characters:="!$~[]/:;*?<>|#^'&()"+<>x_Quote+<>x_Cr+<>x_Tab+\
Char(Character code("•"))+Char(92)+Char(160) //char 92 = \ ,
char(160)= 'hard' non breaking space"
$Replacement_Character:="-"
$Bad_Char_Count:=Length($Bad_Characters)
For ($i;1;$Bad_Char_Count)
$Source:=Replace
string($Source;$Bad_Characters[[$i]];$Replacement_Character)
End for
//windows file names can not end with a period
$Source:=utl_text_Strip_Characters ($Source;"R";".") //removes
trailing '.'
$0:=utl_text_Strip_Characters ($Source) //remove leading and trailing
spaces
//end
On Fri, 17 Feb 2017 08:22:53 +0100, Maurice Inzirillo - AJAR via
4D_Tech wrote:
> Hello Olivier,
>
> Blob to document And Document to blob with a method set to be
> executed on server is an easy and reliable solution !
>
>
>
> Best regards,
>
> Maurice Inzirillo
> --
> AJAR S.A.
>
> 4D v16: http://www.ajar.ch/
> Wakanda Wednesday Lab : http://www.ajar.ch/wwl
> twitter: ajar_info
> Tél : +41 (0)323422684
>
>
>
>
>> On 17 Feb 2017, at 07:45, flury olivier via 4D_Tech
>> <4d_tech@lists.4d.com> wrote:
>>
>> I am currently thinking of rewriting an existing document management
>> system. What is the "best" way to move files from the clients to the
>> server and back from the server to the client?
>>
>> VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a
>> kind of caching table (store the last x used files)? ...
>>
>> Gladly appreciate all sharing of experience.
>>
>> -Ursprüngliche Nachricht-
>> Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von
>> G-Mail via 4D_Tech
>> Gesendet: Donnerstag, 16. Februar 2017 23:55
>> An: 4D iNug Technical <4d_tech@lists.4d.com>
>> Cc: G-Mail <jody.be...@gmail.com>
>> Betreff: Re: Question about storing data outside of datafile
>>
>>> On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech
>>> <4d_tech@lists.4d.com> wrote:
>>>
>>> Would you like to exchange your know-how? I did similar system, and
>>> there are some tricky parts in case of C/S.
>>
>>
>> There is lots of decisions we made for our purposes, and there are
>> different ways to do things for sure. We had to meet various
>> security regulations therefore we changed some of the ways we did
>> things over the 20 or so years that we had built it. Of course new
>> technologies came along too that permitted us to change that up.
>>
>> Is there something specific that you are wanting to discuss?
>>
>>
>>
>> Jody Bevan
>>
>> **
>> 4D Internet Users Group (4D iNUG)
>> FAQ: http://lists.4d.com/faqnug.html
>> Archive: http://lists.4d.com/archives.html
>> Options: http://lists.4d.com/mailman/options/4d_tech
>> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
>> **
>>
>> **
>> 4D Internet Users Group (4D iNUG)
>> FAQ: http://lists.4d.com/faqnug.html
>> Archive: http://lists.4d.com/archives.html
>> Options: http://lists.4d.com/mailman/options/4d_tech
>> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
>> **
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ: http://lists.4d.com/faqnug.html
> Archive: http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
> **
---
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing
**
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
Re: Question about storing data outside of datafile
When I wrote a document management system, I used FTP to move files to correct locations, which included creating directory etc. FTP is agnostic. No problem with bob to server etc. Another approach might be to create another table that holds document in a blob. On server have a process that runs every x minutes that moves these documents to correct location and then marks record available again Regards Chuck Chuck Miller Voice: (617) 739-0306 Informed Solutions, Inc. Fax: (617) 232-1064 mailto:cjmillerinformed-solutions.com Brookline, MA 02446 USA Registered 4D Developer Providers of 4D and Sybase connectivity http://www.informed-solutions.com This message and any attached documents contain information which may be confidential, subject to privilege or exempt from disclosure under applicable law. These materials are intended only for the use of the intended recipient. If you are not the intended recipient of this transmission, you are hereby notified that any distribution, disclosure, printing, copying, storage, modification or the taking of any action in reliance upon this transmission is strictly prohibited. Delivery of this message to any person other than the intended recipient shall not compromise or waive such confidentiality, privilege or exemption from disclosure as to this communication. > On Feb 17, 2017, at 5:53 AM, Peter Bozek via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > On Thu, Feb 16, 2017 at 11:55 PM, G-Mail via 4D_Tech <4d_tech@lists.4d.com> > wrote: > >> There is lots of decisions we made for our purposes, and there are >> different ways to do things for sure. We had to meet various security >> regulations therefore we changed some of the ways we did things over the 20 >> or so years that we had built it. Of course new technologies came along too >> that permitted us to change that up. >> >> Is there something specific that you are wanting to discuss? >> > > What I do is not pass the file to client when record is loaded, it is moved > to client only if needed. For example, if the file is an image, only > thumbnail is part of record and moved to client, the file is transferred > only if user wants to see image in original quality. There is a latency (I > saw a 100MB inserted) that requires a user feedback and error handling. > Then there are more complex cases, when user wants to edit file in external > app etc. > > -- > > Peter Bozek > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
On Thu, Feb 16, 2017 at 11:55 PM, G-Mail via 4D_Tech <4d_tech@lists.4d.com> wrote: > There is lots of decisions we made for our purposes, and there are > different ways to do things for sure. We had to meet various security > regulations therefore we changed some of the ways we did things over the 20 > or so years that we had built it. Of course new technologies came along too > that permitted us to change that up. > > Is there something specific that you are wanting to discuss? > What I do is not pass the file to client when record is loaded, it is moved to client only if needed. For example, if the file is an image, only thumbnail is part of record and moved to client, the file is transferred only if user wants to see image in original quality. There is a latency (I saw a 100MB inserted) that requires a user feedback and error handling. Then there are more complex cases, when user wants to edit file in external app etc. -- Peter Bozek ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
And if you want to track where you store your files, you can save the path In my case, I’m using a UUID as a file name, because I don’t care really about the file name of such file and in a 4D table “document” I store the uuid, the relative path, a time stamp, etc. In top of this storing mechanism I’m using a Path generator which is defining a hierarchical path based on a max number of files to be stored in each folder. Best regards, Maurice Inzirillo -- AJAR S.A. 4D v16: http://www.ajar.ch/ Wakanda Wednesday Lab : http://www.ajar.ch/wwl twitter: ajar_info Tél : +41 (0)323422684 > On 17 Feb 2017, at 09:39, flury olivier via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > Hi Maurice, > > That sounds reasonable. And how do you move the BLOB from the Server to the > Client? > > I see several possibilities: > > - Keep the BLOB in a variable and move the variable from the Server to the > Client. > - Store the BLOB in a field (Record), the Client can access the Record by > e.g. a query. > - Move the Document (not a BLOB) from the Server to the Client by use of the > Resources folder. > > I see advantages/disadvantages for each of the solutions. > > What works well for you? > > Olivier > > -Ursprüngliche Nachricht- > Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von Maurice > Inzirillo - AJAR via 4D_Tech > Gesendet: Freitag, 17. Februar 2017 08:23 > An: 4D iNug Technical <4d_tech@lists.4d.com> > Cc: Maurice Inzirillo - AJAR <maurice.inziri...@ajar.ch> > Betreff: Re: Question about storing data outside of datafile > > Hello Olivier, > > Blob to document And Document to blob with a method set to be executed on > server is an easy and reliable solution ! > > > > Best regards, > > Maurice Inzirillo > -- > AJAR S.A. > > 4D v16: http://www.ajar.ch/ > Wakanda Wednesday Lab : http://www.ajar.ch/wwl > twitter: ajar_info > Tél : +41 (0)323422684 > > > > >> On 17 Feb 2017, at 07:45, flury olivier via 4D_Tech <4d_tech@lists.4d.com> >> wrote: >> >> I am currently thinking of rewriting an existing document management system. >> What is the "best" way to move files from the clients to the server and back >> from the server to the client? >> >> VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a kind of >> caching table (store the last x used files)? ... >> >> Gladly appreciate all sharing of experience. >> >> -Ursprüngliche Nachricht----- >> Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von >> G-Mail via 4D_Tech >> Gesendet: Donnerstag, 16. Februar 2017 23:55 >> An: 4D iNug Technical <4d_tech@lists.4d.com> >> Cc: G-Mail <jody.be...@gmail.com> >> Betreff: Re: Question about storing data outside of datafile >> >>> On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech <4d_tech@lists.4d.com> >>> wrote: >>> >>> Would you like to exchange your know-how? I did similar system, and >>> there are some tricky parts in case of C/S. >> >> >> There is lots of decisions we made for our purposes, and there are different >> ways to do things for sure. We had to meet various security regulations >> therefore we changed some of the ways we did things over the 20 or so years >> that we had built it. Of course new technologies came along too that >> permitted us to change that up. >> >> Is there something specific that you are wanting to discuss? >> >> >> >> Jody Bevan >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ:
Re: Question about storing data outside of datafile
No no no the blob is sent automagically by 4D. Nothing to do. Your parameters are sent by 4D to the server to be executed. Best regards, Maurice Inzirillo -- AJAR S.A. 4D v16: http://www.ajar.ch/ Wakanda Wednesday Lab : http://www.ajar.ch/wwl twitter: ajar_info Tél : +41 (0)323422684 > On 17 Feb 2017, at 09:39, flury olivier via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > Hi Maurice, > > That sounds reasonable. And how do you move the BLOB from the Server to the > Client? > > I see several possibilities: > > - Keep the BLOB in a variable and move the variable from the Server to the > Client. > - Store the BLOB in a field (Record), the Client can access the Record by > e.g. a query. > - Move the Document (not a BLOB) from the Server to the Client by use of the > Resources folder. > > I see advantages/disadvantages for each of the solutions. > > What works well for you? > > Olivier > > -Ursprüngliche Nachricht- > Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von Maurice > Inzirillo - AJAR via 4D_Tech > Gesendet: Freitag, 17. Februar 2017 08:23 > An: 4D iNug Technical <4d_tech@lists.4d.com> > Cc: Maurice Inzirillo - AJAR <maurice.inziri...@ajar.ch> > Betreff: Re: Question about storing data outside of datafile > > Hello Olivier, > > Blob to document And Document to blob with a method set to be executed on > server is an easy and reliable solution ! > > > > Best regards, > > Maurice Inzirillo > -- > AJAR S.A. > > 4D v16: http://www.ajar.ch/ > Wakanda Wednesday Lab : http://www.ajar.ch/wwl > twitter: ajar_info > Tél : +41 (0)323422684 > > > > >> On 17 Feb 2017, at 07:45, flury olivier via 4D_Tech <4d_tech@lists.4d.com> >> wrote: >> >> I am currently thinking of rewriting an existing document management system. >> What is the "best" way to move files from the clients to the server and back >> from the server to the client? >> >> VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a kind of >> caching table (store the last x used files)? ... >> >> Gladly appreciate all sharing of experience. >> >> -Ursprüngliche Nachricht----- >> Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von >> G-Mail via 4D_Tech >> Gesendet: Donnerstag, 16. Februar 2017 23:55 >> An: 4D iNug Technical <4d_tech@lists.4d.com> >> Cc: G-Mail <jody.be...@gmail.com> >> Betreff: Re: Question about storing data outside of datafile >> >>> On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech <4d_tech@lists.4d.com> >>> wrote: >>> >>> Would you like to exchange your know-how? I did similar system, and >>> there are some tricky parts in case of C/S. >> >> >> There is lots of decisions we made for our purposes, and there are different >> ways to do things for sure. We had to meet various security regulations >> therefore we changed some of the ways we did things over the 20 or so years >> that we had built it. Of course new technologies came along too that >> permitted us to change that up. >> >> Is there something specific that you are wanting to discuss? >> >> >> >> Jody Bevan >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** >> >> ** >> 4D Internet Users Group (4D iNUG) >> FAQ: http://lists.4d.com/faqnug.html >> Archive: http://lists.4d.com/archives.html >> Options: http://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
AW: Question about storing data outside of datafile
Hi Maurice, That sounds reasonable. And how do you move the BLOB from the Server to the Client? I see several possibilities: - Keep the BLOB in a variable and move the variable from the Server to the Client. - Store the BLOB in a field (Record), the Client can access the Record by e.g. a query. - Move the Document (not a BLOB) from the Server to the Client by use of the Resources folder. I see advantages/disadvantages for each of the solutions. What works well for you? Olivier -Ursprüngliche Nachricht- Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von Maurice Inzirillo - AJAR via 4D_Tech Gesendet: Freitag, 17. Februar 2017 08:23 An: 4D iNug Technical <4d_tech@lists.4d.com> Cc: Maurice Inzirillo - AJAR <maurice.inziri...@ajar.ch> Betreff: Re: Question about storing data outside of datafile Hello Olivier, Blob to document And Document to blob with a method set to be executed on server is an easy and reliable solution ! Best regards, Maurice Inzirillo -- AJAR S.A. 4D v16: http://www.ajar.ch/ Wakanda Wednesday Lab : http://www.ajar.ch/wwl twitter: ajar_info Tél : +41 (0)323422684 > On 17 Feb 2017, at 07:45, flury olivier via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > I am currently thinking of rewriting an existing document management system. > What is the "best" way to move files from the clients to the server and back > from the server to the client? > > VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a kind of > caching table (store the last x used files)? ... > > Gladly appreciate all sharing of experience. > > -Ursprüngliche Nachricht- > Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von > G-Mail via 4D_Tech > Gesendet: Donnerstag, 16. Februar 2017 23:55 > An: 4D iNug Technical <4d_tech@lists.4d.com> > Cc: G-Mail <jody.be...@gmail.com> > Betreff: Re: Question about storing data outside of datafile > >> On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech <4d_tech@lists.4d.com> >> wrote: >> >> Would you like to exchange your know-how? I did similar system, and >> there are some tricky parts in case of C/S. > > > There is lots of decisions we made for our purposes, and there are different > ways to do things for sure. We had to meet various security regulations > therefore we changed some of the ways we did things over the 20 or so years > that we had built it. Of course new technologies came along too that > permitted us to change that up. > > Is there something specific that you are wanting to discuss? > > > > Jody Bevan > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
On Fri, Feb 17, 2017 at 5:45 PM, flury olivier via 4D_Tech < 4d_tech@lists.4d.com> wrote: > I am currently thinking of rewriting an existing document management > system. What is the "best" way to move files from the clients to the server > and back from the server to the client? > > VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a kind > of caching table (store the last x used files)? ... > Not sure if this will map well to your case, but the following strategy has worked well for me. For reference, this a database with zillions of scanned PDFs. Here's how it works. * There's a documents table with a field for the stored document. * The data is stored outside of the data file automatically by 4D. * When a document is needed, it's loaded by 4D. Works great. The data file is small and documents aren't loaded when they aren't needed. Since I'm not 100% certain what/how 4D would deal with the external documents if something went wrong with the data file, I thought about taking over document management server-side. It looked like that meant giving up a lot of the speed and ease of the built-in system. (I'd like to hear from others that have gone down this road and liked where they ended up.) Instead, I leave the 4D system alone and have a redundant backup. The documents are all exported to an external file hierarchy regularly so that we have a complete mirror of the internal documents. Even if the data file got trashed, we've got a recoverable file hierarchy. (It would take some work, but it would be doable as each document is stored in a folder for its 'owner' record.) This external file dump can be backed up safely using regular OS tools, reused for a Web site, etc. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
AW: Question about storing data outside of datafile
I am currently thinking of rewriting an existing document management system. What is the "best" way to move files from the clients to the server and back from the server to the client? VARIABLE TO VARIABLE? Execute on Client / Execute on Server? Using a kind of caching table (store the last x used files)? ... Gladly appreciate all sharing of experience. -Ursprüngliche Nachricht- Von: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] Im Auftrag von G-Mail via 4D_Tech Gesendet: Donnerstag, 16. Februar 2017 23:55 An: 4D iNug Technical <4d_tech@lists.4d.com> Cc: G-Mail <jody.be...@gmail.com> Betreff: Re: Question about storing data outside of datafile > On 02/16/2017, at 11:09 AM, Peter Bozek via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > Would you like to exchange your know-how? I did similar system, and > there are some tricky parts in case of C/S. There is lots of decisions we made for our purposes, and there are different ways to do things for sure. We had to meet various security regulations therefore we changed some of the ways we did things over the 20 or so years that we had built it. Of course new technologies came along too that permitted us to change that up. Is there something specific that you are wanting to discuss? Jody Bevan ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
I use a separate process to move the file(s) from client to server. Because i work inside transactions a lot. copy file to blob (document to blob) save record daemon on server watches for files (it is actually kicked awake when a file is transferred) ; extracts them (blob to document), and saves on the server in a specified directory > On Thu, Feb 16, 2017 at 4:55 PM, G-Mail via 4D_Tech <4d_tech@lists.4d.com> > wrote: > >> We programmed in the management of the files outside of the database. We >> started doing this with 4D version 3.5 so we had to. We continued doing >> this since it worked very well. There was no need for us to change it, so >> we never really investigated having 4D manage this. Too many other things >> to work on. >> >> > Would you like to exchange your know-how? I did similar system, and there > are some tricky parts in case of C/S. Hell is other people Jean-Paul Sartre ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
On Thu, Feb 16, 2017 at 4:55 PM, G-Mail via 4D_Tech <4d_tech@lists.4d.com> wrote: > We programmed in the management of the files outside of the database. We > started doing this with 4D version 3.5 so we had to. We continued doing > this since it worked very well. There was no need for us to change it, so > we never really investigated having 4D manage this. Too many other things > to work on. > > Would you like to exchange your know-how? I did similar system, and there are some tricky parts in case of C/S. -- Peter Bozek ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Question about storing data outside of datafile
Kirk, Not sure this will help in your situation but this is what we do. This applies to OSX only. I store all documents (hundreds of thousands) in their native format outside the data file in a directory on the same drive as the 4D datafile. I store the files by client / patient in their own folder with several subfolders. I use Execute on Server to retrieve / save the documents. I store all the relevant metadata including the relative path of the document in the 4D datafile. I like to keep the 4D datafile as small as reasonable. e.g. for repairs etc. Now for the backup, I have a separate RAID drive attached to the 4D Server. I use Carbon Copy Cloner to clone the documents folder every 15 minutes. At night, the 4D Server runs the backup to a separate directory on this second drive. Once the 4D backup is complete, I use Retrospect to backup encrypted to a removable drive that goes offsite and to the cloud. Just another option. Mitch ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
Chuck: At each site we stored terabytes of files (PDFs, pictures, etc therefore hundreds of thousands of documents) outside the data file for 24 years at a few hundred sites. As you know medical so many regulations on security and logging of changes to the files. 1. We automatically created new folders each month and within the month each day. 2. The backup was adjusted each month to only backup the current months files and the rest of the year (usually empty). 3. All backups encrypted appropriately. 4. Multiple backups kept of all previous years of data. 5. If a document was modified, it would be moved to the current month’s folder. Doing the above greatly reduced the size of the backup required each night. Security 1. No on the network had access to the server where the files were stored. The backup device was attached directly to it. Only 4D had access to manipulate the files in anyway. 2. All access to the files was audited (even a view) both within our application, and the log files of the server they were stored on. 3. When the files were delivered to 4D Client for viewing they were encrypted and decrypted upon viewing. This provided security over the network of the documents. This is how we set things up to keep the data file size down within 4D. This greatly speeded up the maintenance of the 4D Data file which got done each quarter - more or less depending on the site. By creating new directories the speed of the storage Server was faster in delivering the documents. Also faster in working on that server directly and going into a folder. If ever we needed to have programatic access to run through the files to manipulate for some reason we could easily keep track of where we where in the process. Of course you may already have thought of all of this and there is some other reason this would not work for you. Jody Bevan > On 02/15/2017, at 11:59 AM, Charles Miller via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > On Wed, Feb 15, 2017 at 9:43 AM, Randy Engle via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> However, we found that storing blobs in external file had detrimental >> effects in certain cases. >> If you've got 1000s and 1000s of blobs, these are all stored as individual >> files on disk. >> When copying these files (e.g. doing a manual backup), it takes vastly >> longer than if the blobs are stored in the data file. >> > > Did you let 4D decide where to place these files or did you control them. > Could you not in either case add a folder for 4D to include in backups. > > I have one customer where there are more than 500,000 pictures and drawings > stored in data file. It would be great to store them outside, but for > security reasons (outside access) and moving files between different > locations make this impossible > > Right now we have two copies of data file running on two different servers > and data gets moved between when changes are made. We also have another 5 > servers as well as many single user versions. > > Regards > > Chuck > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
RE: Question about storing data outside of datafile
Hi Chuck, Yes, we let 4D decide where to place the files. Yes, we include this folder in the backup. The detriment came if we need to do a manual copy of the folder holding the blobs. Mucho Tiempo! Randy Engle XC2 Software LLC Did you let 4D decide where to place these files or did you control them. Could you not in either case add a folder for 4D to include in backups. I have one customer where there are more than 500,000 pictures and drawings stored in data file. It would be great to store them outside, but for security reasons (outside access) and moving files between different locations make this impossible Right now we have two copies of data file running on two different servers and data gets moved between when changes are made. We also have another 5 servers as well as many single user versions. Regards Chuck -- - Chuck Miller Voice: (617) 739-0306 Fax: (617) 232-1064 Informed Solutions, Inc. Brookline, MA 02446 USA Registered 4D Developer Providers of 4D, Sybase & SQL Sever connectivity http://www.informed-solutions.com - This message and any attached documents contain information which may be confidential, subject to privilege or exempt from disclosure under applicable law. These materials are intended only for the use of the intended recipient. If you are not the intended recipient of this transmission, you are hereby notified that any distribution, disclosure, printing, copying, storage, modification or the taking of any action in reliance upon this transmission is strictly prohibited. Delivery of this message to any person other than the intended recipient shall not compromise or waive such confidentiality, privilege or exemption from disclosure as to this communication. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Question about storing data outside of datafile
On Wed, Feb 15, 2017 at 9:43 AM, Randy Engle via 4D_Tech < 4d_tech@lists.4d.com> wrote: > However, we found that storing blobs in external file had detrimental > effects in certain cases. > If you've got 1000s and 1000s of blobs, these are all stored as individual > files on disk. > When copying these files (e.g. doing a manual backup), it takes vastly > longer than if the blobs are stored in the data file. > Did you let 4D decide where to place these files or did you control them. Could you not in either case add a folder for 4D to include in backups. I have one customer where there are more than 500,000 pictures and drawings stored in data file. It would be great to store them outside, but for security reasons (outside access) and moving files between different locations make this impossible Right now we have two copies of data file running on two different servers and data gets moved between when changes are made. We also have another 5 servers as well as many single user versions. Regards Chuck -- - Chuck Miller Voice: (617) 739-0306 Fax: (617) 232-1064 Informed Solutions, Inc. Brookline, MA 02446 USA Registered 4D Developer Providers of 4D, Sybase & SQL Sever connectivity http://www.informed-solutions.com - This message and any attached documents contain information which may be confidential, subject to privilege or exempt from disclosure under applicable law. These materials are intended only for the use of the intended recipient. If you are not the intended recipient of this transmission, you are hereby notified that any distribution, disclosure, printing, copying, storage, modification or the taking of any action in reliance upon this transmission is strictly prohibited. Delivery of this message to any person other than the intended recipient shall not compromise or waive such confidentiality, privilege or exemption from disclosure as to this communication. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
RE: Question about storing data outside of datafile
Hi Kirk, My understanding is that the external data will not be loaded, until it is "needed", e.g. Record is loaded (e.g input form) or programmatically loaded. However, we found that storing blobs in external file had detrimental effects in certain cases. If you've got 1000s and 1000s of blobs, these are all stored as individual files on disk. When copying these files (e.g. doing a manual backup), it takes vastly longer than if the blobs are stored in the data file. We put everything back into the data file. Copying the files was a nightmare. FYI This is on Windows Might work differently on Mac Also, we previously stored all BLOBs in a separate related table, so we were sure they weren't loaded until specifically requested programmatically. However, this was quite awhile back, but we just never changed it. Randy Engle XC2 Software LLC I haven't used the 'store outside of datafile' option (text, object, picture fields) very often. I'm wondering about how 4D manages loading the records across the network. I know that if I query a table from a client and then load or display the selection of records the record data gets pulled down from the server to the client. Thus it's generally faster to use EOS to do the searching on the server and only have to load the specific records required. I'm wondering if 4D is optimized to not download the externally stored data unless it's required? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
