@Scott: I can't afford to have this datastream managed by Fedora, because it's several megabytes large and there are over a million of them.
@Rich: the http://user@pass:/resource syntax works with cURL too, that's why I gave it a shot even though I don't know what Fedora actually uses to connect to remote servers. Your source link is very interesting though. I'll give it a look. I wonder how this hasn't been brought up before. Accessing resources through authentication seems like quite a common task to me, and I hoped I could do it without hacking the Fedora code. Thanks sc Stefano Cossu Director of Application Services, Collections The Art Institute of Chicago 116 S. Michigan Ave. Chicago, IL 60603 312-499-4026 On 5/9/13 3:45 PM, [email protected] wrote: > Send Fedora-commons-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Fedora-commons-users digest..." > > > Today's Topics: > > 1. Re: [NEWBIE] Accessing external resource with HTTP > authentication (Benjamin Armintor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 9 May 2013 16:45:16 -0400 > From: Benjamin Armintor <[email protected]> > Subject: Re: [fcrepo-user] [NEWBIE] Accessing external resource with > HTTP authentication > To: "Support and info exchange list for Fedora users." > <[email protected]> > Message-ID: > <cadqq8tpfxo8va7ub2ncb6xvbjzcbf1s7unu7hftcqhpntwb...@mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Yes, in that sense it would be straightforward- though this question of > where the credentials would be stored was one of things that derailed the > feature in the first place! > > > On Thu, May 9, 2013 at 4:40 PM, Rich d'Rich <[email protected]> wrote: > >> AFAIK the username:password@ syntax is a browser artefact that the Java >> HTTP access libraries (apache commons httpclient) that Fedora uses doesn't >> support. >> >> This also means that you can't do a server-server import where the source >> Fedora server requires authentication, and it causes problems with >> disseminators. >> >> However, looking at the code, most of the "wiring" is there: >> >> https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/access/DefaultAccess.java >> >> it just needs getDatastreamDissemination (around line 1145) to extract a >> username and password from somewhere and put it into ContentManagerParams. >> Ideally, there would be a configured table of known external servers and >> credentials that could be kept secure so passwords aren't bandied about. >> >> I may be wrong though and there's already a way to do this :-) >> >> >> On 10 May 2013 06:36, Scott Prater <[email protected]> wrote: >> >>> Stefano -- >>> >>> Are you ingesting the datastreams as managed datastreams, or as redirect >>> or external datastreams? >>> >>> If the former, once Fedora ingests the FOXML, the object is referred to >>> by its internal Fedora URI, and no source URLs or passwords are exposed >>> in any object export. >>> >>> If the datastreams are managed, then you may want to take a compromise >>> approach: fetch them to the local machine using curl or some such tool, >>> then ingest the local file. Once it's ingested, you can delete the >>> local file. >>> >>> Managed datastreams are usually preferred to external or redirect >>> datastreams; there are use cases for external and redirect datastreams >>> (which is why they exist), but the normal case is to store datastreams >>> as managed. >>> >>> -- Scott >>> >>> On 05/09/2013 01:08 PM, Benjamin Armintor wrote: >>>> Stefano- >>>> I remember some conversation a couple of years ago about supporting >>>> BASIC auth in services, but as far as I know they didn't go anywhere. >>>> Maybe another committer remembers something? In any case, I don't see >>>> why storing the credentials like that wouldn't work, if you can accept >>>> the plain-text issues you cite. >>>> >>>> As far as certs, Im afraid you're on your own. I will warn you that >>>> Java errs on the side of verification unless you instruct it not to, so >>>> invalid certs will cause other problems. >>>> >>>> - Ben >>>> >>>> >>>> On Thu, May 9, 2013 at 12:32 PM, Stefano Cossu <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi there, >>>> I'm starting to tinker with Fedora and trying to write a CMA >>> workflow. >>>> I'm building a digital object that should grab an image datastream >>> from >>>> an HTTPS server which requires basic authentication. >>>> I tried inserting the authentication data in the URL for the >>> datastream, >>>> but now I have 2 problems: >>>> 1) Username and password are stored in plain text in the FOXML, >>> visible >>>> by everyone who looks up that record in Fedora, as well as all over >>> the >>>> logs. >>>> 2) I still can't connect to the server this way. The server's >>>> certificate is expired, I don't know if that plays a role. >>>> >>>> Fedora throws this error: >>>> >>>> ERROR 2013-05-09 11:04:28.618 [http-8080-1] (BaseRestResource) >>>> Unexpected error fulfilling REST API request >>>> org.fcrepo.server.errors.HttpServiceNotFoundException: >>>> [DefaultExternalContentManager] returned an error. The underlying >>> error >>>> was a org.fcrepo.server.errors.GeneralException T >>>> he message was "Error getting >>>> https://username:password@imageserver/myHugePicture"; . >>>> at >>>> >>> org.fcrepo.server.storage.DefaultExternalContentManager.getExternalContent(DefaultExternalContentManager.java:152) >>>> ~[fcrepo-server-3.6.2.jar:na] >>>> at >>>> >>> org.fcrepo.server.access.DefaultAccess.getDatastreamDissemination(DefaultAccess.java:1148) >>>> ~[fcrepo-server-3.6.2.jar:na] >>>> at >>>> >>> org.fcrepo.server.rest.DatastreamResource.getDatastream(DatastreamResource.java:247) >>>> ~[fcrepo-server-3.6.2.jar:na] >>>> [...] >>>> >>>> And the image server's Apache error log: >>>> >>>> Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection to >>>> child 0 established (server imageserver:443) >>>> [Thu May 09 11:04:25 2013] [info] Seeding PRNG with 144 bytes of >>> entropy >>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] SSL library >>> error >>>> 1 in handshake (server imageserver:443) >>>> [Thu May 09 11:04:25 2013] [info] SSL Library Error: 336151608 >>>> error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal >>> error >>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection >>> closed >>>> to child 0 with abortive shutdown (server imageserver:443) >>>> [...] >>>> >>>> Of course, I can always use a redirect datastream and let the client >>>> deal with authentication and SSL, but I'd like to hide the source >>> URI if >>>> possible. >>>> >>>> >>>> Below is the FOXML representation of my object: >>>> >>>> <foxml:digitalObject VERSION="1.1" PID="test:dervPub_obj" >>>> xsi:schemaLocation="info:fedora/fedora-system:def/foxml# >>>> http://www.fedora.info/definitions/1/0/foxml1-1.xsd";> >>>> <foxml:objectProperties> >>>> <foxml:property >>>> NAME="info:fedora/fedora-system:def/model#state" VALUE="Active"/> >>>> <foxml:property >>>> NAME="info:fedora/fedora-system:def/model#label" VALUE="Disseminator >>>> object"/> >>>> <foxml:property >>>> NAME="info:fedora/fedora-system:def/model#ownerId" >>> VALUE="fedoraAdmin"/> >>>> <foxml:property >>>> NAME="info:fedora/fedora-system:def/model#createdDate" >>>> VALUE="2013-05-09T15:37:41.708Z"/> >>>> <foxml:property >>>> NAME="info:fedora/fedora-system:def/view#lastModifiedDate" >>>> VALUE="2013-05-09T15:37:41.892Z"/> >>>> </foxml:objectProperties> >>>> <foxml:datastream ID="AUDIT" STATE="A" CONTROL_GROUP="X" >>>> VERSIONABLE="false"> >>>> <foxml:datastreamVersion ID="AUDIT.0" LABEL="Audit Trail >>> for >>>> this object" CREATED="2013-05-09T15:37:41.708Z" MIMETYPE="text/xml" >>>> FORMAT_URI="info:fedora/fedora-system:format/xml.fedora.audit"> >>>> <foxml:xmlContent> >>>> <audit:auditTrail> >>>> <audit:record ID="AUDREC1"> >>>> <audit:process type="Fedora API-M"/> >>>> <audit:action>addDatastream</audit:action> >>>> <audit:componentID>SOURCE_IMG</audit:componentID> >>>> <audit:responsibility>fedoraAdmin</audit:responsibility> >>>> <audit:date>2013-05-09T15:37:41.892Z</audit:date> >>>> <audit:justification/> >>>> </audit:record> >>>> </audit:auditTrail> >>>> </foxml:xmlContent> >>>> </foxml:datastreamVersion> >>>> </foxml:datastream> >>>> <foxml:datastream ID="DC" STATE="A" CONTROL_GROUP="X" >>>> VERSIONABLE="true"> >>>> <foxml:datastreamVersion ID="DC1.0" LABEL="Dublin Core >>> Record >>>> for this object" CREATED="2013-05-09T15:37:41.708Z" >>> MIMETYPE="text/xml" >>>> FORMAT_URI="http://www.openarchives.org/OAI/2.0/oai_dc/"; >>> SIZE="388"> >>>> <foxml:xmlContent> >>>> <oai_dc:dc >>>> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >>>> http://www.openarchives.org/OAI/2.0/oai_dc.xsd";> >>>> <dc:title>Disseminator object</dc:title> >>>> <dc:identifier>test:dervPub_obj</dc:identifier> >>>> </oai_dc:dc> >>>> </foxml:xmlContent> >>>> </foxml:datastreamVersion> >>>> </foxml:datastream> >>>> <foxml:datastream ID="RELS-EXT" STATE="A" CONTROL_GROUP="X" >>>> VERSIONABLE="false"> >>>> <foxml:datastreamVersion ID="RELS-EXT.0" >>> LABEL="Relationships" >>>> CREATED="2013-05-09T15:37:41.837Z" MIMETYPE="application/rdf+xml" >>>> FORMAT_URI="info:fedora/fedora-system:FedoraRELSExt-1.0" SIZE="273"> >>>> <foxml:xmlContent> >>>> <rdf:RDF> >>>> <rdf:Description >>>> rdf:about="info:fedora/test:dervPub_obj"> >>>> <hasModel >>>> rdf:resource="info:fedora/test:dervPub_CModel"/> >>>> </rdf:Description> >>>> </rdf:RDF> >>>> </foxml:xmlContent> >>>> </foxml:datastreamVersion> >>>> </foxml:datastream> >>>> <foxml:datastream ID="SOURCE_IMG" STATE="A" CONTROL_GROUP="E" >>>> VERSIONABLE="true"> >>>> <foxml:datastreamVersion ID="SOURCE_IMG.0" LABEL="full >>> sized >>>> image" CREATED="2013-05-09T15:37:41.892Z" MIMETYPE="image/jpeg"> >>>> <foxml:contentLocation TYPE="URL" >>>> REF="https://username:password@imageserver/myHugePicture"/> >>>> </foxml:datastreamVersion> >>>> </foxml:datastream> >>>> >>>> I would really appreciate your help. >>>> >>>> >>>> >>>> >>> ------------------------------------------------------------------------------ >>>> Learn Graph Databases - Download FREE O'Reilly Book >>>> "Graph Databases" is the definitive new guide to graph databases and >>>> their applications. This 200-page book is written by three acclaimed >>>> leaders in the field. The early access version is available now. >>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>> _______________________________________________ >>>> Fedora-commons-users mailing list >>>> [email protected] >>>> <mailto:[email protected]> >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>> >>>> >>>> >>>> >>>> >>> ------------------------------------------------------------------------------ >>>> Learn Graph Databases - Download FREE O'Reilly Book >>>> "Graph Databases" is the definitive new guide to graph databases and >>>> their applications. This 200-page book is written by three acclaimed >>>> leaders in the field. The early access version is available now. >>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>> >>>> >>>> >>>> _______________________________________________ >>>> Fedora-commons-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>> >>> >>> -- >>> Scott Prater >>> Shared Development Group >>> General Library System >>> University of Wisconsin - Madison >>> [email protected] >>> 5-5415 >>> >>> >>> ------------------------------------------------------------------------------ >>> Learn Graph Databases - Download FREE O'Reilly Book >>> "Graph Databases" is the definitive new guide to graph databases and >>> their applications. This 200-page book is written by three acclaimed >>> leaders in the field. The early access version is available now. >>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>> _______________________________________________ >>> Fedora-commons-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>> >> >> >> ------------------------------------------------------------------------------ >> Learn Graph Databases - Download FREE O'Reilly Book >> "Graph Databases" is the definitive new guide to graph databases and >> their applications. This 200-page book is written by three acclaimed >> leaders in the field. The early access version is available now. >> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >> _______________________________________________ >> Fedora-commons-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >> >> > -------------- next part -------------- > An HTML attachment was scrubbed... > > ------------------------------ > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and > their applications. This 200-page book is written by three acclaimed > leaders in the field. The early access version is available now. > Download your free book today! http://p.sf.net/sfu/neotech_d2d_may > > ------------------------------ > > _______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > > End of Fedora-commons-users Digest, Vol 75, Issue 7 > *************************************************** ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
