That's a v. good suggestion. On Tue, Feb 7, 2012 at 5:00 PM, Florentine, George <[email protected]> wrote: > One little trick we've found useful on these kind of loosely coupled file > transfer architectures is to transmit a small file AFTER the big file > transfer is complete to indicate that file transfer of the big file(s) is > complete. Lots of ftp servers will create the file in the target directory > and start filling it with bytes. This partially transferred file is then > visible to other applications that are monitoring the directory. Lots of > opportunities for race conditions between your code that is monitoring the > directory for new files to ingest and the (s)ftp server process transferring > the contents of the file, so checking on just the existence of the transfer > file is not a good approach. > > thx, > > g > --- > > > George Florentine > > VP, Engineering > > +1 (303) 542-2173 | Office > +1 (303) 669-8628 | Cell > +1 (303) 544-0522 | Fax > > [email protected] > > http://www.flatironssolutions.com > > > > > > > > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Demian Hess > Sent: Tuesday, February 07, 2012 1:59 PM > To: General MarkLogic Developer Discussion > Subject: Re: [MarkLogic Dev General] Can app server handle file uploads >2GB? > > I think FTP is the best choice for this. > > In regard to monitoring a directory: is there an xdmp function for > getting a listing of files and directories from the file system? I > didn't see that in the API documents. > > Thanks, Demian > > On Tue, Feb 7, 2012 at 2:04 PM, Geert Josten <[email protected]> wrote: >> Hi Demian, >> >> My guess at it is that MarkLogic would try to maintain the binary file as >> request param within memory, which is limited to some size for good >> reason. With 10+ concurrent requests, all uploading 2 Gb, memory would >> build up very rapidly. I'm guessing the limits are related to the limits >> within xdmp:document-get, which does not allow reading files larger than >> 16Mb or 64Mb (depending on 32/64-bit install). >> >> I think I wouldn't advice to use form upload to send large files across. >> Other protocols like (S)FTP are better suited for that. You could allow >> such files to be uploaded to file locations through a separate (S)FTP >> server on the same host. You can let MarkLogic monitor that directory >> using a Scheduled Task, and perhaps leveraging Information Studio flows. >> In these processes you can use different function to load such large >> files, though I am not sure they are able to handle 2+Gb files.. >> >> Kind regards, >> Geert >> >>> -----Oorspronkelijk bericht----- >>> Van: [email protected] [mailto:general- >>> [email protected]] Namens Demian Hess >>> Verzonden: dinsdag 7 februari 2012 19:45 >>> Aan: [email protected] >>> Onderwerp: [MarkLogic Dev General] Can app server handle file uploads >>>2GB? >>> >>> We would like to store large binary files (many EPS and high >>> resolution PDFs) in MarkLogic. It would be easiest for users to create >>> zips with all their files and then upload via an HTML form >>> (enctype=multipart/form-data), but the size of the zip archives would >>> be more than 2GB. Indeed, some individual files can get to 2GB by >>> themselves. >>> >>> I tried POSTing a 2GB file to an app server and got a "413 Request >>> Entity Too Large" error. I don't see any settings in MarkLogic to >>> control max size of uploads for HTTP app servers. >>> >>> How efficient would ML be at handling large posts of 100MB and larger? >>> Would it buffer everything in memory or would it stream to a temporary >>> file on disk? Would ML 5.0 be more efficient than 4.0 at handling >>> large file uploads? >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://developer.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general > > > > -- > Demian Hess > > Avalon Consulting, LLC > 527 Maple Avenue East, Suite 200, Vienna, VA 22180 > > Mobile: 301-943-8307 > Fax: 845-367-5496 > [email protected] > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general
-- Demian Hess Avalon Consulting, LLC 527 Maple Avenue East, Suite 200, Vienna, VA 22180 Mobile: 301-943-8307 Fax: 845-367-5496 [email protected] _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
