I'll echo Blakeley's comment about the loading binary documents chapter. Working with the server's local filesystem can be done with xdmp:filesystem-directory, xdmp:filesystem-file, etc …
--Dave On Feb 7, 2012, at 3:58 PM, Demian Hess wrote: > I think FTP is the best choice for this. > > In regard to monitoring a directory: is there an xdmp function for > getting a listing of files and directories from the file system? I > didn't see that in the API documents. > > Thanks, Demian > > On Tue, Feb 7, 2012 at 2:04 PM, Geert Josten <[email protected]> wrote: >> Hi Demian, >> >> My guess at it is that MarkLogic would try to maintain the binary file as >> request param within memory, which is limited to some size for good >> reason. With 10+ concurrent requests, all uploading 2 Gb, memory would >> build up very rapidly. I'm guessing the limits are related to the limits >> within xdmp:document-get, which does not allow reading files larger than >> 16Mb or 64Mb (depending on 32/64-bit install). >> >> I think I wouldn't advice to use form upload to send large files across. >> Other protocols like (S)FTP are better suited for that. You could allow >> such files to be uploaded to file locations through a separate (S)FTP >> server on the same host. You can let MarkLogic monitor that directory >> using a Scheduled Task, and perhaps leveraging Information Studio flows. >> In these processes you can use different function to load such large >> files, though I am not sure they are able to handle 2+Gb files.. >> >> Kind regards, >> Geert >> >>> -----Oorspronkelijk bericht----- >>> Van: [email protected] [mailto:general- >>> [email protected]] Namens Demian Hess >>> Verzonden: dinsdag 7 februari 2012 19:45 >>> Aan: [email protected] >>> Onderwerp: [MarkLogic Dev General] Can app server handle file uploads >>> 2GB? >>> >>> We would like to store large binary files (many EPS and high >>> resolution PDFs) in MarkLogic. It would be easiest for users to create >>> zips with all their files and then upload via an HTML form >>> (enctype=multipart/form-data), but the size of the zip archives would >>> be more than 2GB. Indeed, some individual files can get to 2GB by >>> themselves. >>> >>> I tried POSTing a 2GB file to an app server and got a "413 Request >>> Entity Too Large" error. I don't see any settings in MarkLogic to >>> control max size of uploads for HTTP app servers. >>> >>> How efficient would ML be at handling large posts of 100MB and larger? >>> Would it buffer everything in memory or would it stream to a temporary >>> file on disk? Would ML 5.0 be more efficient than 4.0 at handling >>> large file uploads? >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://developer.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general > > > > -- > Demian Hess > > Avalon Consulting, LLC > 527 Maple Avenue East, Suite 200, Vienna, VA 22180 > > Mobile: 301-943-8307 > Fax: 845-367-5496 > [email protected] > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
