Re: content negotiating for package formats
On 21/01/2011 08:46, Ian Stuart wrote: On 20/01/11 18:11, Julie Allinson wrote: I might be talking nonsense here, but is this something that could support 'graceful' behaviour ... one thing I noticed in testing was that EPrints will accept a METS package with epdcx but the deposit fails if there is any other metadata instead of or in addition to the epdcx embedded in the METS doc. I'm not criticising EPrints or advocating METS but it struck me that if there is a package that could be deposited knowing that it will succeed and that you could stuff all kinds of things into it which the repository will either know what to do with and do that (unpack etc.) or simply accept and store? ... so for the EPrints case, you might need a new export plug, but in the meantime you could still be making deposits. EPrints does, indeed, expect epcdx in the section of the METS data, however I have also found that EPints is much more relaxed about the structure of METS/epcdx that DPspace is I've yet to find a Fedora volunteer to try imports with ;-) Is that a hint? ... we could probably help with Fedora at York. It is certainly possible to write an Importer for EPrints that will accept whatever format you care to specify... and my experience is that this is easier in EPrints than DSpace (but then again, I'm a Perl Monkey :chuckle: ) Whilst on the subject of epcdx: I am swinging away from it now - there are just so many things it doesn't do well, or misses out all together. Perhaps this is an opportunity to get people from L&T, Data, Article, and various other data-store types together, and try to come up with an extensible core schema that can be both cross-platform as well as cross-type? I don't think this is for SWORD to fix ... I agree epdcx hasn't had the takeup it might have, is problematic and isn't being actively maintained. The only other reliable alternative currently is DC. Is there any work happening in JISC/UKOLN at the moment around this area? Julie -- Julie Allinson Digital Library Manager University Library& Archives, J.B. Morrell Library University of York, Heslington, York, YO10 5DD, UK tel: ++44 (0) 1904 324083 skype: j.allinson web: http://tinyurl.com/dcd6a5 disclaimer: http://www.york.ac.uk/docs/disclaimer/email.htm --
Re: content negotiating for package formats
On 20/01/11 18:11, Julie Allinson wrote: I might be talking nonsense here, but is this something that could support 'graceful' behaviour ... one thing I noticed in testing was that EPrints will accept a METS package with epdcx but the deposit fails if there is any other metadata instead of or in addition to the epdcx embedded in the METS doc. I'm not criticising EPrints or advocating METS but it struck me that if there is a package that could be deposited knowing that it will succeed and that you could stuff all kinds of things into it which the repository will either know what to do with and do that (unpack etc.) or simply accept and store? ... so for the EPrints case, you might need a new export plug, but in the meantime you could still be making deposits. EPrints does, indeed, expect epcdx in the section of the METS data, however I have also found that EPints is much more relaxed about the structure of METS/epcdx that DPspace is I've yet to find a Fedora volunteer to try imports with ;-) It is certainly possible to write an Importer for EPrints that will accept whatever format you care to specify... and my experience is that this is easier in EPrints than DSpace (but then again, I'm a Perl Monkey :chuckle: ) Whilst on the subject of epcdx: I am swinging away from it now - there are just so many things it doesn't do well, or misses out all together. Perhaps this is an opportunity to get people from L&T, Data, Article, and various other data-store types together, and try to come up with an extensible core schema that can be both cross-platform as well as cross-type? -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
On 19/01/2011 17:26, Richard Jones wrote: Hi Ed, On 19/01/11 13:27, Ed Summers wrote: On Wed, Jan 19, 2011 at 6:28 AM, Richard Jones wrote: We've had a few discussions in the past about "supporting" some formats, and they always end up pretty divisive. So SWORD is aiming to be totally agnostic on the point, but it does need to provide the client and server a mechanism to negotiate over what format they are interchanging. If we can achieve that, that will be relatively useful in an interoperability setting, I think, particularly as many SWORD servers (particularly repositories) are able to create dissemination packages in a large number of formats (see EPrints export plugins for example). Would it be too restrictive to require SWORD collections to only support one package format? This would mean that there MUST be one and only one sword:acceptPackaging per app:collection in the service document. I think this would simplify matters significantly for implementors since: 1) there would no longer be any need for the q attribute on sword:acceptPackaging, and the requirement to interpret them. 2) X-Packaging header registration, and the need for clients to send it would go away 2) a client could only retrieve a package in the format it was deposited in. Does anyone really have an appetite for dynamically rewriting package formats as part of an HTTP request cycle? I think that would be overly restrictive, while the above gains are relatively minimal in the scheme of things. Also, as per my earlier comment about export plugins in EPrints, you could easily imagine throwing a known package format into the repository, and then asking it to give you it back in a variety of formats that you don't know how to generate yourself. I guess a better question is: do we have many SWORD implementations that support POSTing multiple package flavors to the same collection? Also, I am -1 on SWORD requiring a standard package format. I think it would be fine to list some preferences, and light-weight, community driven mechanisms for identifying them, but that's as far as SWORD should go IMHO. It's been a long standing complaint against SWORD that despite it being an "interoperability" standard, you can't even deposit the same package into DSpace, EPrints and Fedora, let alone other implementations that weren't funded as part of the original project. From both a practical point of view and a community perception point of view this has to be addressed. We have tried to work around the "standard" package format issue by adopting an Atom Multipart [1] deposit of an Atom Entry with optional embedded metadata plus a binary payload which may either be a single file or if given the Content-Type of application/zip a plain old zip file with no prescribed internal content. This ought to be satisfactory because it is not only about the most simple format you could come up with, but it also leverages the existing semantics of AtomPub, so adding it is of minimal effort. http://tools.ietf.org/html/draft-gregorio-atompub-multipart-04 I might be talking nonsense here, but is this something that could support 'graceful' behaviour ... one thing I noticed in testing was that EPrints will accept a METS package with epdcx but the deposit fails if there is any other metadata instead of or in addition to the epdcx embedded in the METS doc. I'm not criticising EPrints or advocating METS but it struck me that if there is a package that could be deposited knowing that it will succeed and that you could stuff all kinds of things into it which the repository will either know what to do with and do that (unpack etc.) or simply accept and store? ... so for the EPrints case, you might need a new export plug, but in the meantime you could still be making deposits. Julie Cheers, Richard -- Julie Allinson Digital Library Manager University Library& Archives, J.B. Morrell Library University of York, Heslington, York, YO10 5DD, UK tel: ++44 (0) 1904 324083 skype: j.allinson web: http://tinyurl.com/dcd6a5 disclaimer: http://www.york.ac.uk/docs/disclaimer/email.htm --
Re: content negotiating for package formats
On 20 Jan 2011, at 09:00, Ian Stuart wrote: > On 19/01/11 17:26, Richard Jones wrote: >> Also, as per my earlier comment about export plugins in EPrints, you >> could easily imagine throwing a known package format into the >> repository, and then asking it to give you it back in a variety of >> formats that you don't know how to generate yourself. > H.. interesting! > I see a future development for the OA-RJ broker there!! With K-Int we produced something called the Content Transcoder which was a cloud service for doing just this using common e-learning packaging formats (SCORM, IMS-CP, IMS-CC etc) http://purl.oclc.org/NET/transcoder Adding a SWORD endpoint to it might make for a nice example. > > >> It's been a long standing complaint against SWORD that despite it being >> an "interoperability" standard, you can't even deposit the same package >> into DSpace, EPrints and Fedora, let alone other implementations that >> weren't funded as part of the original project. From both a practical >> point of view and a community perception point of view this has to be >> addressed. > This is an issue that the OA-RJ Project is addressing with the Broker work. > Our initial work is to produce a single importable package, and the importers > that go with them, so that the Broker can pass on an Item to a number of > target repositories. > > (There is a future development idea which would be to allow each target > repository to identify the package format it wanted, and the Broker would > transfer in that package but that is a *much* slower mechanism when > dealing with multi-institutional papers) > > -- > > Ian Stuart. > Developer: Open Access Repository Junction and OpenDepot.org > Bibliographics and Multimedia Service Delivery team, > EDINA, > The University of Edinburgh. > > http://edina.ac.uk/ > > This email was sent via the University of Edinburgh. > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > smime.p7s Description: S/MIME cryptographic signature
Re: content negotiating for package formats
On 19/01/11 17:26, Richard Jones wrote: Also, as per my earlier comment about export plugins in EPrints, you could easily imagine throwing a known package format into the repository, and then asking it to give you it back in a variety of formats that you don't know how to generate yourself. H.. interesting! I see a future development for the OA-RJ broker there!! It's been a long standing complaint against SWORD that despite it being an "interoperability" standard, you can't even deposit the same package into DSpace, EPrints and Fedora, let alone other implementations that weren't funded as part of the original project. From both a practical point of view and a community perception point of view this has to be addressed. This is an issue that the OA-RJ Project is addressing with the Broker work. Our initial work is to produce a single importable package, and the importers that go with them, so that the Broker can pass on an Item to a number of target repositories. (There is a future development idea which would be to allow each target repository to identify the package format it wanted, and the Broker would transfer in that package but that is a *much* slower mechanism when dealing with multi-institutional papers) -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
Hi Ed, On 19/01/11 13:27, Ed Summers wrote: On Wed, Jan 19, 2011 at 6:28 AM, Richard Jones wrote: We've had a few discussions in the past about "supporting" some formats, and they always end up pretty divisive. So SWORD is aiming to be totally agnostic on the point, but it does need to provide the client and server a mechanism to negotiate over what format they are interchanging. If we can achieve that, that will be relatively useful in an interoperability setting, I think, particularly as many SWORD servers (particularly repositories) are able to create dissemination packages in a large number of formats (see EPrints export plugins for example). Would it be too restrictive to require SWORD collections to only support one package format? This would mean that there MUST be one and only one sword:acceptPackaging per app:collection in the service document. I think this would simplify matters significantly for implementors since: 1) there would no longer be any need for the q attribute on sword:acceptPackaging, and the requirement to interpret them. 2) X-Packaging header registration, and the need for clients to send it would go away 2) a client could only retrieve a package in the format it was deposited in. Does anyone really have an appetite for dynamically rewriting package formats as part of an HTTP request cycle? I think that would be overly restrictive, while the above gains are relatively minimal in the scheme of things. Also, as per my earlier comment about export plugins in EPrints, you could easily imagine throwing a known package format into the repository, and then asking it to give you it back in a variety of formats that you don't know how to generate yourself. I guess a better question is: do we have many SWORD implementations that support POSTing multiple package flavors to the same collection? Also, I am -1 on SWORD requiring a standard package format. I think it would be fine to list some preferences, and light-weight, community driven mechanisms for identifying them, but that's as far as SWORD should go IMHO. It's been a long standing complaint against SWORD that despite it being an "interoperability" standard, you can't even deposit the same package into DSpace, EPrints and Fedora, let alone other implementations that weren't funded as part of the original project. From both a practical point of view and a community perception point of view this has to be addressed. We have tried to work around the "standard" package format issue by adopting an Atom Multipart [1] deposit of an Atom Entry with optional embedded metadata plus a binary payload which may either be a single file or if given the Content-Type of application/zip a plain old zip file with no prescribed internal content. This ought to be satisfactory because it is not only about the most simple format you could come up with, but it also leverages the existing semantics of AtomPub, so adding it is of minimal effort. http://tools.ietf.org/html/draft-gregorio-atompub-multipart-04 Cheers, Richard
Re: content negotiating for package formats
I'm sorry, but I don't follow this... On 19 January 2011 13:27, Ed Summers wrote: > > Would it be too restrictive to require SWORD collections to only > support one package format? This would mean that there MUST be > one and only one sword:acceptPackaging per app:collection in the > service document. I think this would simplify matters significantly > for implementors since: > It might be too restrictive if you have mixed content in a collection. But in most cases, probably not. However... Also, I am -1 on SWORD requiring a standard package format. I think it > would be fine to list some preferences, and light-weight, community > driven mechanisms for identifying them, but that's as far as SWORD > should go IMHO. > > If we don't standardise on a packaging format (for at least a subset of content that we are interested in), and only allow having one format per collection, then we can forget any possibility of SWORD being used for interoperability, and possibly even for having deposit tools that work with a range of different repositories. I would also be very concerned about the ability to evolve a repository / service if we don't provide for content negotiation. Without the ability to support more than one package format for a collection, then the only way roll out new packaging formats to cope missing data, open up new features, etc. is to abruptly cut off support of the old format. That could cause all sorts of problems with existing clients that may be trying to use SWORD. G
Re: content negotiating for package formats
> I don't think you can ever get away from a degree of content negotiation, > but it doesn't necessarily need to be as complex as the scenarios outlined > depending on what agreements you can have for common formats in common > cases. I agree with Graham and Scott. Please can we step back and clearly and completely define the scope of the issue. My understanding is as follows: There are two dimensions of content type -- packaging (eg zip, bagit, tar, mets,...) and metadata (dc, mets, ore, ead, ...) and it is not feasible to specify all of the combinations as unique keys. These combinations are used to both deposit and retrieve content packages. Client to Server (Deposit): * There is a recommended packaging format (Zip + DC in Atom) * There is a header to inform the server which packaging format is actually being sent, as well as the metadata format * There is information available as to the types of format the server will accept (in the Svc Desc) Server to Client (Retrieve): * There SHOULD be the same recommended packaging format. If a client can construct it, then it can likely read it back again. Requiring a different format just doubles the work. * There is a header to inform the client which packaging format is being sent, as well as the metadata format * There is information available as to the types of format the server can send (in Svc Desc) which may not be the same as the set of formats it will accept for deposit. * There SHOULD be a header to request a particular format. Now, call me a heretic, but I think Packaging is the wrong way round. The outermost layer is the top level content type, and hence should be in the Accept and Content-Type headers. If you download a zip file containing 5 plain text files, Content-Type is application/zip. So what is really needed is Accept-Metadata and Content-Metadata, to request the format of the metadata in the response package. If the server can't deliver the combination, then it won't do that. The same way that if you ask for Accept-Language: fr and the server can't deliver French then it won't. I disagree with Ed that there MUST be only a single format. That's never going to fly with current workflows, and would result in the "interoperability" of OAI-PMH's use of simple DC ... to wit, that there is syntactic interoperability, but the semantics are completely worthless due to people stuffing random content into fields because they can't say what they really mean. The Linked Data effort has shown us that even in an open world of infinite vocabularies, people *will* self-organize into support of useful relationships, and the same should apply here. Rob
Re: content negotiating for package formats
On 19 Jan 2011, at 13:27, Ed Summers wrote: > On Wed, Jan 19, 2011 at 6:28 AM, Richard Jones > wrote: >> We've had a few discussions in the past about "supporting" some formats, and >> they always end up pretty divisive. So SWORD is aiming to be totally >> agnostic on the point, but it does need to provide the client and server a >> mechanism to negotiate over what format they are interchanging. If we can >> achieve that, that will be relatively useful in an interoperability setting, >> I think, particularly as many SWORD servers (particularly repositories) are >> able to create dissemination packages in a large number of formats (see >> EPrints export plugins for example). > > Would it be too restrictive to require SWORD collections to only > support one package format? This would mean that there MUST be > one and only one sword:acceptPackaging per app:collection in the > service document. I think this would simplify matters significantly > for implementors since: > > 1) there would no longer be any need for the q attribute on > sword:acceptPackaging, and the requirement to interpret them. > 2) X-Packaging header registration, and the need for clients to send > it would go away > 2) a client could only retrieve a package in the format it was > deposited in. Does anyone really have an appetite for dynamically > rewriting package formats as part of an HTTP request cycle? > > I guess a better question is: do we have many SWORD implementations > that support POSTing multiple package flavors to the same collection? > > Also, I am -1 on SWORD requiring a standard package format. I think it > would be fine to list some preferences, and light-weight, community > driven mechanisms for identifying them, but that's as far as SWORD > should go IMHO. Agreed; there is a distinction between what the spec must address, and what the community of implementers has to agree on outside the spec to make it work. > > //Ed smime.p7s Description: S/MIME cryptographic signature
Re: content negotiating for package formats
On Wed, Jan 19, 2011 at 6:28 AM, Richard Jones wrote: > We've had a few discussions in the past about "supporting" some formats, and > they always end up pretty divisive. So SWORD is aiming to be totally > agnostic on the point, but it does need to provide the client and server a > mechanism to negotiate over what format they are interchanging. If we can > achieve that, that will be relatively useful in an interoperability setting, > I think, particularly as many SWORD servers (particularly repositories) are > able to create dissemination packages in a large number of formats (see > EPrints export plugins for example). Would it be too restrictive to require SWORD collections to only support one package format? This would mean that there MUST be one and only one sword:acceptPackaging per app:collection in the service document. I think this would simplify matters significantly for implementors since: 1) there would no longer be any need for the q attribute on sword:acceptPackaging, and the requirement to interpret them. 2) X-Packaging header registration, and the need for clients to send it would go away 2) a client could only retrieve a package in the format it was deposited in. Does anyone really have an appetite for dynamically rewriting package formats as part of an HTTP request cycle? I guess a better question is: do we have many SWORD implementations that support POSTing multiple package flavors to the same collection? Also, I am -1 on SWORD requiring a standard package format. I think it would be fine to list some preferences, and light-weight, community driven mechanisms for identifying them, but that's as far as SWORD should go IMHO. //Ed
Re: content negotiating for package formats
On 19/01/11 11:28, Richard Jones wrote: I think we're talking slightly cross-purposes here. The problem with the content negotiation is in when the client /retrieves/ its content from the sword server, not when it deposits it. From a deposit point of view we have the acceptPackaging field in the service document, which tells the client what formats the server will support the deposit of, and there is the (X-)Packaging header which tells the server what the client has given them. The default (standard?) format that we're recommending for SWORD is a plain old zip file, optionally augmented with dc embedded in an atom entry sent along with it, which is very much in-line with the AtomPub spec (and also fits your criteria above :) ) The problem is when the client is trying to retrieve content from the server, and wants to ask for it in a particular format via content negotiation. At the moment, there is no provision in HTTP Accept- headers to ask the server for a METS package. To a certain extent, yes however that also assumes that SWORD is only ever a transport mechanism to transfer from Client-to-Server. To my mind, SWORD is an agnostic transport mechanism: carrying messages from one "machine" to another "machine", so messages should work bilaterally if the desktop application requests a transfer from the Learning Object Repository, which way is the "Deposit"? Hence my suggestion that the whole content negotiation thing should be manageable from both ends. -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
On 19 January 2011 09:49, Scott Wilson wrote: > I think its time to take a step backwards here. > > The "packaging problem" identified by the SWORD project was not that SWORD > or AtomPub have a problem with POSTing packaged content formats. > > The problem is that implementations of SWORD in the academic repositories > community use - needlessly, IMHO - diverse incompatible formats, especially > of metadata within a package. > > I don't see that adding any number of HTTP headers is going to improve > interoperability while this remains the case. If nothing else, I would > expect implementations to largely ignore any such headers sent by the client > and look inside the package to try and figure out what it is and if it can > support it. The headers just provide more opportunities for client error. > > I would suggest SWORD is completely agnostic on the subject of packaged > content formats, but that the SWORD implementation community make a > concerted effort to identify and support a common core of packaging and > metadata formats so that there is practical on-the-ground interoperability > with a reliable default format for client implementations to support > out-of-the-box. > > I want to agree on having a standard package, but there are issues with saying that. The most obvious is how that sits with the current use of SWORD. But there is also the situation that repositories are used in different ways, have different features (that reflect in their content) and contain a wide range of different materials. We could agree on and define a profile that works in the most general way for a set of use cases / content types. But that would still leave it up to us/others to define other profiles that would be used in those other scenarios. I don't think you can ever get away from a degree of content negotiation, but it doesn't necessarily need to be as complex as the scenarios outlined depending on what agreements you can have for common formats in common cases. G
Re: content negotiating for package formats
On 19/01/11 10:16, Scott Wilson wrote: On 19 Jan 2011, at 10:05, Ian Stuart wrote: On 19/01/11 09:49, Scott Wilson wrote: I would suggest SWORD is completely agnostic on the subject of packaged content formats, but that the SWORD implementation community make a concerted effort to identify and support a common core of packaging and metadata formats so that there is practical on-the-ground interoperability with a reliable default format for client implementations to support out-of-the-box. Bearing in mind that I have my tongue embedded firmly in my cheek here... Oh good! I have an excellent content package that will either work with binary data directly included or passed-by-reference, and I am working on Importers for DSpace& EPrints as we speak... as outputs of the OA-RJ Broker work. I suggest we standardise on a really crappy packaging format with almost zero features, combined with a totally inadequate metadata schema. At least that way it might actually work.* I think we're talking slightly cross-purposes here. The problem with the content negotiation is in when the client /retrieves/ its content from the sword server, not when it deposits it. From a deposit point of view we have the acceptPackaging field in the service document, which tells the client what formats the server will support the deposit of, and there is the (X-)Packaging header which tells the server what the client has given them. The default (standard?) format that we're recommending for SWORD is a plain old zip file, optionally augmented with dc embedded in an atom entry sent along with it, which is very much in-line with the AtomPub spec (and also fits your criteria above :) ) The problem is when the client is trying to retrieve content from the server, and wants to ask for it in a particular format via content negotiation. At the moment, there is no provision in HTTP Accept- headers to ask the server for a METS package. We've had a few discussions in the past about "supporting" some formats, and they always end up pretty divisive. So SWORD is aiming to be totally agnostic on the point, but it does need to provide the client and server a mechanism to negotiate over what format they are interchanging. If we can achieve that, that will be relatively useful in an interoperability setting, I think, particularly as many SWORD servers (particularly repositories) are able to create dissemination packages in a large number of formats (see EPrints export plugins for example). I don't want us to decide, therefore, on a package format, but on a mechanism by which a client and server can get together and negotiate one for themselves. Cheers, Richard On a serious note: yes - the transport mechanism should be agnostic to the content... unless one wants to define a content transfer mechanism rather than a transport mechanism :) -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. * http://en.wikipedia.org/wiki/Worse_is_better
Re: content negotiating for package formats
On 19/01/11 10:16, Scott Wilson wrote: I have an excellent content package that will either work with binary data directly included or passed-by-reference, and I am working on Importers for DSpace& EPrints as we speak... as outputs of the OA-RJ Broker work. I suggest we standardise on a really crappy packaging format with almost zero features, combined with a totally inadequate metadata schema. At least that way it might actually work.* Oh, so you've seen my work then ;-) -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
On 19 Jan 2011, at 10:05, Ian Stuart wrote: > On 19/01/11 09:49, Scott Wilson wrote: >> I would suggest SWORD is completely agnostic on the subject of >> packaged content formats, but that the SWORD implementation community >> make a concerted effort to identify and support a common core of >> packaging and metadata formats so that there is practical >> on-the-ground interoperability with a reliable default format for >> client implementations to support out-of-the-box. > Bearing in mind that I have my tongue embedded firmly in my cheek here... Oh good! > I have an excellent content package that will either work with binary data > directly included or passed-by-reference, and I am working on Importers for > DSpace & EPrints as we speak... as outputs of the OA-RJ Broker work. I suggest we standardise on a really crappy packaging format with almost zero features, combined with a totally inadequate metadata schema. At least that way it might actually work.* > On a serious note: yes - the transport mechanism should be agnostic to the > content... unless one wants to define a content transfer mechanism rather > than a transport mechanism :) > > -- > > Ian Stuart. > Developer: Open Access Repository Junction and OpenDepot.org > Bibliographics and Multimedia Service Delivery team, > EDINA, > The University of Edinburgh. > > http://edina.ac.uk/ > > This email was sent via the University of Edinburgh. > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > * http://en.wikipedia.org/wiki/Worse_is_better smime.p7s Description: S/MIME cryptographic signature
RE: content negotiating for package formats
Enjoying watching the discussion :) getting good now... keep it coming. /dff > -Original Message- > From: Scott Wilson [mailto:[email protected]] > Sent: 19 January 2011 09:50 > To: [email protected] > Subject: Re: content negotiating for package formats > > I think its time to take a step backwards here. > > The "packaging problem" identified by the SWORD project was not that > SWORD or AtomPub have a problem with POSTing packaged content formats. > > The problem is that implementations of SWORD in the academic > repositories community use - needlessly, IMHO - diverse incompatible > formats, especially of metadata within a package. > > I don't see that adding any number of HTTP headers is going to improve > interoperability while this remains the case. If nothing else, I would > expect implementations to largely ignore any such headers sent by the > client and look inside the package to try and figure out what it is and > if it can support it. The headers just provide more opportunities for > client error. > > I would suggest SWORD is completely agnostic on the subject of packaged > content formats, but that the SWORD implementation community make a > concerted effort to identify and support a common core of packaging and > metadata formats so that there is practical on-the-ground > interoperability with a reliable default format for client > implementations to support out-of-the-box. > > S > > On 19 Jan 2011, at 08:06, Richard Jones wrote: > > > Hi Ian, > > > > On 18/01/11 12:11, Ian Stuart wrote: > >> On 10/01/11 18:49, Richard Jones wrote: > >>> It's looking like a separate header is the way to do this, with the > >>> following couple of options immediately standing out: > >>> > >>> Accept-Features (or X-Accept-Features if it isn't sufficiently > official) > >>> X-Packaging > >>> X-Accept-Packaging (which I just made up for the purposes of this > >>> discussion) > >>> > >>> Some comments on these: > >>> > >>> Accept-Features > >>> Having looked at the document [1] (thanks Graham (K)) it looks like > it > >>> would give us the leeway that we need to describe requirements > while > >>> ensuring that Graham (T)'s concerns (which I share) about matching > up > >>> package format requirements with mimetypes would be dealt with. On > the > >>> other hand, this document is 12/13 years old and the header has not > made > >>> it into the HTTP content negotiation documentation and is > significantly > >>> different in format to all the other Accept- headers. It could also > be a > >>> substantial effort for servers to implement the full requirements > of > >>> this header. > >>> > >>> X-Packaging > >>> I'm against using this in this way as it is already used to alert > the > >>> server during POST as to the package format that is being supplied. > The > >>> format of the header for content negotiation would have to be > totally > >>> different to this usage: a list of package formats and q values for > >>> example, rather than a single definitive URI. I see scope for > confusion. > >>> > >>> X-Accept-Packaging > >>> Given my concerns about X-Packaging and the comments above about > >>> Accept-Feature, perhaps there is a middle ground that we can define > >>> which does something more minimal with just mimetypes, package > formats > >>> and q values in a way similar to having a mimetype that has added > >>> parameters. > >>> > >>> For example: > >>> Accept: application/zip; q=1.0, > application/atom+xml;type=entry;q=0.8 > >>> > >>> X-Accept-Packaging: application/zip;{package=METSDSpaceSIP};q=1.0, > >>> application/atom+xml;type=entry;{package=AtomSIP};q=0.8 > >>> > >>> Or some other suitably neat and unambiguous serialisation which is > in > >>> line with how the other Accept- headers work and also gives us the > >>> information we want in a totally definitive mimetype<->package > format > >>> way. This could be supplied alongside the usual Accept header so > that > >>> clients which can't generate the X-Accept-Packaging header can fall > back > >>> easily to the usual content negotiation route. > >> > >> I'm still unclear why there is a need to combine the content
Re: content negotiating for package formats
On 19/01/11 09:49, Scott Wilson wrote: I would suggest SWORD is completely agnostic on the subject of packaged content formats, but that the SWORD implementation community make a concerted effort to identify and support a common core of packaging and metadata formats so that there is practical on-the-ground interoperability with a reliable default format for client implementations to support out-of-the-box. Bearing in mind that I have my tongue embedded firmly in my cheek here... I have an excellent content package that will either work with binary data directly included or passed-by-reference, and I am working on Importers for DSpace & EPrints as we speak... as outputs of the OA-RJ Broker work. On a serious note: yes - the transport mechanism should be agnostic to the content... unless one wants to define a content transfer mechanism rather than a transport mechanism :) -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
I think its time to take a step backwards here.
The "packaging problem" identified by the SWORD project was not that SWORD or
AtomPub have a problem with POSTing packaged content formats.
The problem is that implementations of SWORD in the academic repositories
community use - needlessly, IMHO - diverse incompatible formats, especially of
metadata within a package.
I don't see that adding any number of HTTP headers is going to improve
interoperability while this remains the case. If nothing else, I would expect
implementations to largely ignore any such headers sent by the client and look
inside the package to try and figure out what it is and if it can support it.
The headers just provide more opportunities for client error.
I would suggest SWORD is completely agnostic on the subject of packaged content
formats, but that the SWORD implementation community make a concerted effort to
identify and support a common core of packaging and metadata formats so that
there is practical on-the-ground interoperability with a reliable default
format for client implementations to support out-of-the-box.
S
On 19 Jan 2011, at 08:06, Richard Jones wrote:
> Hi Ian,
>
> On 18/01/11 12:11, Ian Stuart wrote:
>> On 10/01/11 18:49, Richard Jones wrote:
>>> It's looking like a separate header is the way to do this, with the
>>> following couple of options immediately standing out:
>>>
>>> Accept-Features (or X-Accept-Features if it isn't sufficiently official)
>>> X-Packaging
>>> X-Accept-Packaging (which I just made up for the purposes of this
>>> discussion)
>>>
>>> Some comments on these:
>>>
>>> Accept-Features
>>> Having looked at the document [1] (thanks Graham (K)) it looks like it
>>> would give us the leeway that we need to describe requirements while
>>> ensuring that Graham (T)'s concerns (which I share) about matching up
>>> package format requirements with mimetypes would be dealt with. On the
>>> other hand, this document is 12/13 years old and the header has not made
>>> it into the HTTP content negotiation documentation and is significantly
>>> different in format to all the other Accept- headers. It could also be a
>>> substantial effort for servers to implement the full requirements of
>>> this header.
>>>
>>> X-Packaging
>>> I'm against using this in this way as it is already used to alert the
>>> server during POST as to the package format that is being supplied. The
>>> format of the header for content negotiation would have to be totally
>>> different to this usage: a list of package formats and q values for
>>> example, rather than a single definitive URI. I see scope for confusion.
>>>
>>> X-Accept-Packaging
>>> Given my concerns about X-Packaging and the comments above about
>>> Accept-Feature, perhaps there is a middle ground that we can define
>>> which does something more minimal with just mimetypes, package formats
>>> and q values in a way similar to having a mimetype that has added
>>> parameters.
>>>
>>> For example:
>>> Accept: application/zip; q=1.0, application/atom+xml;type=entry;q=0.8
>>>
>>> X-Accept-Packaging: application/zip;{package=METSDSpaceSIP};q=1.0,
>>> application/atom+xml;type=entry;{package=AtomSIP};q=0.8
>>>
>>> Or some other suitably neat and unambiguous serialisation which is in
>>> line with how the other Accept- headers work and also gives us the
>>> information we want in a totally definitive mimetype<->package format
>>> way. This could be supplied alongside the usual Accept header so that
>>> clients which can't generate the X-Accept-Packaging header can fall back
>>> easily to the usual content negotiation route.
>>
>> I'm still unclear why there is a need to combine the content type
>> ("application/zip; q=1.0") with the data encoding ("METSDSpaceSIP; q=1.0")
>>
>> Can't you say "(1) I only deal in .tgz content, and (2) you can package
>> whatevers within that content as 'Foo', 'Bar', or even
>> 'Acme::WhiteSpaceEncoded'"
>
> I think that the problem is that you can't guarantee that the list of content
> types and the list of packaging types are combinable in a meaningful way;
> Graham T's email had an example.
>
> So suppose a server can give you content type A with packaging formats X and
> Y, or content type B with packaging format Z:
>
> A + (X or Y)
> B + Z
>
> and your content negotiation header says:
>
> Accept: A; q=1.0, B; q=0.8
> Accept-Packaging: Z; q=1.0, X; q=1.0
>
> Which combination do you return?
>
> On the other hand, this is a general problem and even within the Media
> Feature syntax that Graham K describes in his RFC acknowledges this
> effectively limits the use of "q" values to top-level feature sets. So, you
> would be limited to content negotiating for:
>
> Accept-Media-Feature: A(X), B(Z), A(Y)
>
> for example; i.e. explicitly declaring your preference of the combination of
> content-type and packaging format.
>
> I've spent the last 3 or 4 days looking at the Media Feature stuff in detail,
> and I
Re: content negotiating for package formats
On 19 January 2011 08:44, Ian Stuart wrote: > On 19/01/11 08:06, Richard Jones wrote: > >> and your content negotiation header says: >> >> Accept: A; q=1.0, B; q=0.8 >> Accept-Packaging: Z; q=1.0, X; q=1.0 >> >> Which combination do you return? >> > I agree its not a clear-cut case. however it also has to be said that > the content negotiation header isn't helping by declaring two options to be > of equal "value" > > I would say: > A+Z, then A+X, then B+Z, then B+X > > FIFO-style > > (after all, if you have a preference for order, try: > > Accept-Packaging: X; q=1.0, Z; q=1.0 > > instead) > > > Even under the concept of a combined Accept model, what do you do when you > receive > > Accept-Media-Feature: A(Z); q=1.0, A(X); q=1.0, B(Z); q=0.8, B(X); q=0.8 > > It's complicated even beyond what you are suggesting. What if as the server, you consider your support of format A to only be partial - say q=0.6 equivalent? Or if your support of format B is also only q=0.8 equivalent? Basically, you end up with two sides negotiating with no [declared] perfect support in common. Is 'perfect' support on one side and poor support on the other better or worse than so/so support on either side? So what do you do when you receive the kind of negotiation that you mention? Answer: whatever you choose to do. You should use the quality indications to steer you to a format/packaging that is better supported by the client. But you can weight your choice with whatever factors you choose to use. Is it twice as fast/efficient for you to return packaging X? Then return A(X) - the client has declared that it doesn't care which of A(Z) or A(X) it gets, it's only said that it would prefer to get one of them instead of B(Z). Content negotiation isn't an exact discipline, because you are having to make generalisations. How do you express that in one metadata schema you only support 10% of the fields, but they are far more important than the 95% of fields you support in another? Or that there is such a vast difference in resources used that you don't care about a 5% loss in quality to take a JPEG rather than a TIF? All you can do is rank them as best you can for the general case, and that sometimes that might mean you get a format that isn't optimal for that particular case. G
Re: content negotiating for package formats
On 19/01/11 08:06, Richard Jones wrote: and your content negotiation header says: Accept: A; q=1.0, B; q=0.8 Accept-Packaging: Z; q=1.0, X; q=1.0 Which combination do you return? I agree its not a clear-cut case. however it also has to be said that the content negotiation header isn't helping by declaring two options to be of equal "value" I would say: A+Z, then A+X, then B+Z, then B+X FIFO-style (after all, if you have a preference for order, try: Accept-Packaging: X; q=1.0, Z; q=1.0 instead) Even under the concept of a combined Accept model, what do you do when you receive Accept-Media-Feature: A(Z); q=1.0, A(X); q=1.0, B(Z); q=0.8, B(X); q=0.8 -- Ian Stuart. Developer: Open Access Repository Junction and OpenDepot.org Bibliographics and Multimedia Service Delivery team, EDINA, The University of Edinburgh. http://edina.ac.uk/ This email was sent via the University of Edinburgh. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: content negotiating for package formats
Hi Ian,
On 18/01/11 12:11, Ian Stuart wrote:
On 10/01/11 18:49, Richard Jones wrote:
It's looking like a separate header is the way to do this, with the
following couple of options immediately standing out:
Accept-Features (or X-Accept-Features if it isn't sufficiently official)
X-Packaging
X-Accept-Packaging (which I just made up for the purposes of this
discussion)
Some comments on these:
Accept-Features
Having looked at the document [1] (thanks Graham (K)) it looks like it
would give us the leeway that we need to describe requirements while
ensuring that Graham (T)'s concerns (which I share) about matching up
package format requirements with mimetypes would be dealt with. On the
other hand, this document is 12/13 years old and the header has not made
it into the HTTP content negotiation documentation and is significantly
different in format to all the other Accept- headers. It could also be a
substantial effort for servers to implement the full requirements of
this header.
X-Packaging
I'm against using this in this way as it is already used to alert the
server during POST as to the package format that is being supplied. The
format of the header for content negotiation would have to be totally
different to this usage: a list of package formats and q values for
example, rather than a single definitive URI. I see scope for confusion.
X-Accept-Packaging
Given my concerns about X-Packaging and the comments above about
Accept-Feature, perhaps there is a middle ground that we can define
which does something more minimal with just mimetypes, package formats
and q values in a way similar to having a mimetype that has added
parameters.
For example:
Accept: application/zip; q=1.0, application/atom+xml;type=entry;q=0.8
X-Accept-Packaging: application/zip;{package=METSDSpaceSIP};q=1.0,
application/atom+xml;type=entry;{package=AtomSIP};q=0.8
Or some other suitably neat and unambiguous serialisation which is in
line with how the other Accept- headers work and also gives us the
information we want in a totally definitive mimetype<->package format
way. This could be supplied alongside the usual Accept header so that
clients which can't generate the X-Accept-Packaging header can fall back
easily to the usual content negotiation route.
I'm still unclear why there is a need to combine the content type
("application/zip; q=1.0") with the data encoding ("METSDSpaceSIP; q=1.0")
Can't you say "(1) I only deal in .tgz content, and (2) you can package
whatevers within that content as 'Foo', 'Bar', or even
'Acme::WhiteSpaceEncoded'"
I think that the problem is that you can't guarantee that the list of
content types and the list of packaging types are combinable in a
meaningful way; Graham T's email had an example.
So suppose a server can give you content type A with packaging formats X
and Y, or content type B with packaging format Z:
A + (X or Y)
B + Z
and your content negotiation header says:
Accept: A; q=1.0, B; q=0.8
Accept-Packaging: Z; q=1.0, X; q=1.0
Which combination do you return?
On the other hand, this is a general problem and even within the Media
Feature syntax that Graham K describes in his RFC acknowledges this
effectively limits the use of "q" values to top-level feature sets. So,
you would be limited to content negotiating for:
Accept-Media-Feature: A(X), B(Z), A(Y)
for example; i.e. explicitly declaring your preference of the
combination of content-type and packaging format.
I've spent the last 3 or 4 days looking at the Media Feature stuff in
detail, and I have to confess it does feel like a sledgehammer to crack
a nut. At the moment I'm playing with specifying restricted version of
it to see if we can get the effect that we want without the huge
overhead of a full implementation.
As a consequence, I'm still open to Ian's suggested approach here,
provided that we can decide a) what the new HTTP header should be
called, and b) what the rules for resolving content negotiation
ambiguities as shown above should be.
Cheers,
Richard
Re: content negotiating for package formats
On 10/01/11 18:49, Richard Jones wrote:
It's looking like a separate header is the way to do this, with the
following couple of options immediately standing out:
Accept-Features (or X-Accept-Features if it isn't sufficiently official)
X-Packaging
X-Accept-Packaging (which I just made up for the purposes of this
discussion)
Some comments on these:
Accept-Features
Having looked at the document [1] (thanks Graham (K)) it looks like it
would give us the leeway that we need to describe requirements while
ensuring that Graham (T)'s concerns (which I share) about matching up
package format requirements with mimetypes would be dealt with. On the
other hand, this document is 12/13 years old and the header has not made
it into the HTTP content negotiation documentation and is significantly
different in format to all the other Accept- headers. It could also be a
substantial effort for servers to implement the full requirements of
this header.
X-Packaging
I'm against using this in this way as it is already used to alert the
server during POST as to the package format that is being supplied. The
format of the header for content negotiation would have to be totally
different to this usage: a list of package formats and q values for
example, rather than a single definitive URI. I see scope for confusion.
X-Accept-Packaging
Given my concerns about X-Packaging and the comments above about
Accept-Feature, perhaps there is a middle ground that we can define
which does something more minimal with just mimetypes, package formats
and q values in a way similar to having a mimetype that has added
parameters.
For example:
Accept: application/zip; q=1.0, application/atom+xml;type=entry;q=0.8
X-Accept-Packaging: application/zip;{package=METSDSpaceSIP};q=1.0,
application/atom+xml;type=entry;{package=AtomSIP};q=0.8
Or some other suitably neat and unambiguous serialisation which is in
line with how the other Accept- headers work and also gives us the
information we want in a totally definitive mimetype<->package format
way. This could be supplied alongside the usual Accept header so that
clients which can't generate the X-Accept-Packaging header can fall back
easily to the usual content negotiation route.
I'm still unclear why there is a need to combine the content type
("application/zip; q=1.0") with the data encoding ("METSDSpaceSIP; q=1.0")
Can't you say "(1) I only deal in .tgz content, and (2) you can package
whatevers within that content as 'Foo', 'Bar', or even
'Acme::WhiteSpaceEncoded'"
--
Ian Stuart.
Developer: Open Access Repository Junction and OpenDepot.org
Bibliographics and Multimedia Service Delivery team,
EDINA,
The University of Edinburgh.
http://edina.ac.uk/
This email was sent via the University of Edinburgh.
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
Re: content negotiating for package formats
Hi Graham, I'm also wondering about your combination of content-type and internal packaging format. The media feature description framework [2] was intended to capture this kind of combination of features in a more structured fashion. Thus, I would imagine something like: Accept-media-features: (| (& type="application/zip" package=METSDSpaceSIP);q=1.0 (& type="application/atom+xml" atomtype=entry package=AtomSIP);q=0.8 ) This would require IANA registration of the new header field, and a new media features called "package" and "atomtype", per [4]. Feature "type" is already registered [3]. [2] http://tools.ietf.org/html/rfc2533 [3] http://tools.ietf.org/html/rfc2913 [4] http://tools.ietf.org/html/rfc2506 This seems like the "proper" way to do it, and just comes with the same caveats as above. What is the status of this RFC? Are we safe to go ahead and use it? (incidentally, we wouldn't need to register atomtype, as the param type=entry is part of the mimetype already). The RFCs are "proposed standard", which means they are at the first step on the standards track. Some of them have been used in the Internet fax work. But that said, I think that to date there is little implementation experience. The expression format is based on LDAP search filters, and is easy to parse. Do you have a ref to any code that can already do this? I looked into the Python LDAP bindings but these search filters are all passed around as strings and I don't see any good ways of getting programmatic access to the media feature sets for free. Cheers, Richard I remember having some discussions several years ago with someone from the team that was standardizing SIP who were looking at using this, and they were concerned that the full-blown media feature matching was too complex. What we discussed at the time was to define a profile that restricted the form of feature expression to union of conjunction (aka disjunctive normal form) - which is also adopted imn the example I gave previously. I'm not sure what came of those discussions, but I think this would be a reasonable path, which keeps the early implementations simple but does leave a possibility of introducing more complex matching patterns later if required (subject to migration constraints). (It's roughly equivalent to an ALC description logic, but I didn't know about that at the time; the full feature expression matching is basically a structural subsumption reasoner.) I'd be interested in the rest of the advisory panel's and project team's opinions on this, as this may well define the way that I work for the duration of the project. Initially, I think we need to be clear about the requirements. (BTW, if we get some implementation experience, I may attempt to progress the specs to Draft Standard status.) #g -- Richard Jones wrote: Hi Folks, Thanks, this is really great stuff. On 10/01/11 16:05, Robert D. Sanderson wrote: On 7 January 2011 17:36, Richard Jones wrote: 2/ Define an extension to the application/zip mimetype which allows us to specify the package format as a parameter. So we could, for example, specify a parameter "swordpackage" which can take the URI of a package format, and construct mimetypes like application/zip;swordpackage=uri:METSDSpaceSIP (see http://tools.ietf.org/html/draft-ietf-atompub-typeparam-00) The questions here are: is this a legitimate extension/approach, would this break anything else on the web in general, and is it naive to assume that all packages have the top level mimetype of application/zip? First of all, no, it's not legitimate to invent new parameters for existing mime types. RFC 2048, Section 2.2.3 ... the names, values, and meanings of any parameters must be fully specified when a media type is registered in the IETF tree ... http://www.faqs.org/rfcs/rfc2048.html So it's not legal to create a parameter swordpackage and attach it to the existing application/zip. Ok, it sounds like this option then is simply out altogether. More generally, the HTTP specification defines the accept headers as: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html Note that the extension parameter here is for the header, not the mimetype. The BNF allows the accept-extension rule ONLY after the mandatory q value in accept-params. I have to confess to having overlooked the accept-extension rule, as there wasn't an example of usage in that document. Can you give us an example as to how that is used? Which means using Accept-Encoding in this way is potentially problematic, but Accept does have provision that would make this use legitimate. Like mime types, content encodings also have a registry. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html section 3.5 Basically, there are two routes to avoid breaking the rules, neither easy: 1. Register new Mime Types for every packaging format. 2. Use an x- header and eventually write an RFC to standardize it. We looked at this in both SRU (eg wh
