Chunked File Upload
Status: DRAFT
Created: 20. January 2013
Author: shgupta
JIRA: SLING-2707
References: - http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
Updated: -
Use Cases:
1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload.
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point. It doesn't require client to restart the complete upload process.
Approach
Sling provides an endpoint which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling endpoint upon receiving the last chunk, stitches all chunks into a single file. The file inputstream is passed to inner layers for further processing.
In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point.
Protocol Specification
Advertising chunk upload support in sling
Sling advertise its support for chunk upload by adding POST and PATCH methods to the listing of allowed methods in the "Allow" OPTIONS response header defined in HTTP/1.1.
[request]
OPTIONS /content/dam/dam-folder/catalog.pdf.chunk HTTP/1.1
[response]
HTTP/1.1 200 OK
Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH
Accept-Patch: append/binary-patch, application/json-patch, application/json-patch
Accept-Patch used to specify the patch document formats accepted by the server. Chunk upload supports three append/binary-patch, application/json-patch, application/json-patch formats.
Upload chunk using POST
Upload of first and intermediate chunks
Client uses POST method to upload binary chunk of file.
[request]
POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Host: localhost:4502
If-Match: "e0023aa4e"
Content-Type:
Date: <date>
Content-Length: $requestlen
[response]
HTTP/1.1 201 CREATED
Content-Length: 0
ETag: "e0023aa4f"
Location: /content/dam/dam-folder/catalog.pdf.chunk
Connection: close
Server: localhost:4502
The location header provides path to retrieve chunk upload attributes.
Concurrent request from multiple client requests will corrupt the resource. Clients should use a conditional request such that the request will fail if the resource has been updated since the client last accessed the resource. For example, the client can use a strong ETag in an If-Match header on the chunk request.
Upload of last chunk
Client appends "/last" to upload url to signify that it would be last that needs to be uploaded. The last chunk upload request can optionally be multipart request if client requires to send additional parameters to sling.
[request]
POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res/last HTTP/1.1
Host: localhost:4502
Date: <date>
Content-type: multipart/form-data, boundary=AaB03x
--AaB03x
content-disposition: form-data; name="fileSize"
210000000
--AaB03x
content-disposition: form-data; name="file"; fileName="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
$binarydata
--AaB03x--
[response]
HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close
The location header provides location of fully uploaded asset.The response contains the $md5_hash of merged binary.
Upload chunk using PATCH
Sling support three patch formats to upload chunks on sling.
- append/binary-patch: Use this content type if request contain only binary chunk data. If request has additional parameters that need to be passed, then use application/json-patch or multipart/form-data-patch content type.
[request]
PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/octet-stream-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
$binarydata
[response]
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502
- application/json-patch: JSON format patch document to pass upload attributes along with binary chunk
[request]
PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/json-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
[
{
"jcr:data": "$binarydata",
"fileName": "catalog.pdf",
" field1": "$field1"
}
]
[response]
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502
- multipart/form-data-patch: multipart/form-data-patch format to pass upload attributes along with binary chunk
[request]
PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: multipart/form-data-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
--AaB03x
content-disposition: form-data; name="field1"
$field1
--AaB03x
content-disposition: form-data; name="field2"
$field2
--AaB03x
content-disposition: form-data; name="userfile"; filename="$filename"
Content-Type: $mimetype
Content-Transfer-Encoding: binary
$binarydata
--AaB03x--
[response]
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502
h3. Query upload
h4. Query to retrieve last successful chunk upload
Returns the number of last chunk upload and bytes uploaded successfully on sling.
\[request\]
{code:title= Get last successful chunk upload request}
GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: <date>
Content-Length: 0
[response]
HTTP/1.1 200/OK
Connection: close
{
"chunkNumber": 8
"bytesuploaded": 19872
}
If no broken/discontinued upload sling returns 404 not found response.
[response]
Abort chunked upload
[request]
DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: <date>
[response]
HTTP/1.1 200/OK
Connection: close
