Chunked File Upload
Status: DRAFT
Created: 20. January 2013
Author: shgupta
JIRA: SLING-2707
References: - http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
Updated: -
Use Cases:
1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload.
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point. It doesn't require client to restart the complete upload process.
Approach
Sling provides an extension to SlingPostServlet which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk has "Offset" attribute which identify chunk position in complete file. SlingPostServletupon receiving the last chunk, stitches all chunks into a single file and save them to the final destination.
In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point.
Content Model
Chunks are stored within actual path in sling:chunkMixin mixin node type. The content model to store chunk is defined as follows:
[sling:chunkNode] > nt:hierarchyNode
primaryitem jcr:data
- sling:offset (long) mandatory
- jcr:data (binary) mandatory
[sling:chunkMixin]
mixin
- sling:fileLength (long)
- sling:chunksLength (long)
+ * (sling:chunkNode) multiple
The typical nt:file node under chunked upload would look like
/content/dam/folder/catalog.pdf [nt:file]
+ jcr:content [nt:resource] [sling:chunkMixin]
- jcr:data = "" until completed
- sling:fileLength = 982145 - sling:chunksLength = 30000 + chunk_0-9999 [sling:chunkNode]
- sling:offset =0
- jcr:data [binary data]
+ chunk_10000-19999 [sling:chunkNode]
+ ....
Protocol Specification
Upload chunk using POST
Chunk upload request
Client uses POST method to parent path to upload binary chunk of file. Client passes $filename@Offset and $filename@Length as multipart request parameters. "Offset" indicate chunk's data offset in complete file. "Length" is optional parameter and indicates length of complete file. If "Length" parameter is known, Sling automatically calculates if request is last chunk request and according stitches all chunks and store it into final destination.
[request]
POST /content/dam/folder HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Length"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
1000
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Offset"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
400
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf"; filename="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
$binary-data
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe--
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Mon, 06 May 2013 14:42:22 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /content/dam/folder</title>
</head>
<body>
<h1>Content modified /content/dam/folder</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/content/dam/folder" id="Location">/content/dam/folder</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/" id="ParentLocation">/</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/content/dam/folder</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:lastModified");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:mimeType");
<br/>created("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799/jcr:data");<br/>
modified("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799/sling:offset");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/sling:chunksLength");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/content/dam/folder">Modified Resource</a></p>
<p><a href="" class="code-quote">"/">Parent of Modified Resource</a></p>
</body>
</html>
The response shows that chunk is stored in tmp location. The location tag contains unique chunk upload id which need to be passed on subsequent request.
Intermediate chunk upload
Client require to additionally pass chunk upload id.
[request]
POST /content/dam HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name=":chunkNumber"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
3
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name=":chunkUploadId"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name="*"; filename="catalog.pdf"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
$binarydata
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08">Modified Resource</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
Last chunk upload
To indicate the last chunk, client additionally pass ":lastChunk" to true in request parameter. In last chunk upload response, the changelog has all the modification made in actual path repository. It would the same response as if client has uploaded file in single shot to SlingPostServlet
[request]
POST /content/dam HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe
Content-Disposition: form-data; name=":lastChunk"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
true
--XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe
Content-Disposition: form-data; name=":chunkNumber"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
6
--XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe
Content-Disposition: form-data; name=":chunkUploadId"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08
--XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe
Content-Disposition: form-data; name="*"; filename="catalog.pdf"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
$binarydata
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /content/dam</title>
</head>
<body>
<h1>Content modified /content/dam</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/content/dam" id="Location">/content/dam</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/content" id="ParentLocation">/content</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/content/dam</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>created("/content/dam/catalog.pdf");<br/>created("/content/dam/catalog.pdf/jcr:content");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:lastModified");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:mimeType");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:data");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/content/dam">Modified Resource</a></p>
<p><a href="" class="code-quote">"/content">Parent of Modified Resource</a></p>
</body>
</html
The location header provides location of fully uploaded asset.The response contains the $md5_hash of merged binary.
Query Sling about the interrupted chunk upload status
Client sent get request on upload to retrieve chunk upload status.
[request]
GET /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08.json HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: application/json;charset=UTF-8
Content-Length: 155
Date: Thu, 28 Feb 2013 14:14:25 GMT
{"chunksUploaded":5,"bytesUploaded":1600,"jcr:createdBy":"admin","jcr:created":"Thu Feb 28 2013 19:44:24 GMT+0530","jcr:primaryType":"sling:OrderedFolder"}
The above response means that 5 continuous chunks with size 1600 (index 0 to index 1599) are successfully uploaded to server. Client should start with :chunkNumber 6 and slice file from offset 1600.
If no broken/discontinued upload sling returns 404 not found response.
[response]
HTTP/1.1 404 Not Found
Connection: Close
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=utf-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<!DOCTYPE HTML PUBLIC "->
<html>
<head><title>404 No resource found</title></head>
<body>
<h1>No resource found</h1>
<p>Cannot serve request to /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 in /libs/sling/servlet/errorhandler/404.jsp</p>
<h3>Request Progress:</h3>
<pre>
0 (2013-02-28 19:44:25) TIMER_START{Request Processing}
0 (2013-02-28 19:44:25) COMMENT timer_end format is {<elapsed msec>,<timer name>} <optional message>
0 (2013-02-28 19:44:25) LOG Method=GET, PathInfo=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08
0 (2013-02-28 19:44:25) TIMER_START{ResourceResolution}
0 (2013-02-28 19:44:25) TIMER_END{0,ResourceResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 resolves to Resource=, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08]
0 (2013-02-28 19:44:25) LOG Resource Path Info: SlingRequestPathInfo: path='/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08', selectorString='null', extension='null', suffix='null'
0 (2013-02-28 19:44:25) TIMER_START{ServletResolution}
0 (2013-02-28 19:44:25) TIMER_START{resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])}
0 (2013-02-28 19:44:25) LOG {0}: no servlet found
0 (2013-02-28 19:44:25) TIMER_END{0,resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])} Using servlet org.apache.sling.servlets.get.DefaultGetServlet
0 (2013-02-28 19:44:25) TIMER_END{0,ServletResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 handled by Servlet=org.apache.sling.servlets.get.DefaultGetServlet
0 (2013-02-28 19:44:25) LOG Applying Requestfilters
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.bgservlets.impl.BackgroundServletStarterFilter
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.i18n.impl.I18NFilter
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.designimporter.CanvasPageDeleteRequestFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMRequestFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.adobe.granite.optout.impl.OptOutFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.theme.impl.ThemeResolverFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter
0 (2013-02-28 19:44:25) LOG RedirectFilter did not redirect (request extension does not match)
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.warp.TimeWarpFilter
0 (2013-02-28 19:44:25) LOG Applying Componentfilters
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMComponentFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMDebugFilter
0 (2013-02-28 19:44:25) TIMER_START{org.apache.sling.servlets.get.DefaultGetServlet#0}
0 (2013-02-28 19:44:25) TIMER_END{0,org.apache.sling.servlets.get.DefaultGetServlet#0}
0 (2013-02-28 19:44:25) LOG Applying Error filters
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter
0 (2013-02-28 19:44:25) TIMER_START{handleError:status=404}
0 (2013-02-28 19:44:25) TIMER_END{0,handleError:status=404} Using handler /libs/sling/servlet/errorhandler/404.jsp
0 (2013-02-28 19:44:25) LOG Found processor for post processing ProcessorConfiguration: {contentTypes=[text/html],order=-1, active=true, valid=true, processErrorResponse=true, pipeline=(generator=Config(type=htmlparser, config={}), transformers=(Config(type=linkchecker, config={}), Config(type=mobile, config=org.apache.sling.jcr.resource.JcrPropertyMap@42e95cbf), Config(type=mobiledebug, config=org.apache.sling.jcr.resource.JcrPropertyMap@66c821ea), Config(type=contentsync, config=org.apache.sling.jcr.resource.JcrPropertyMap@5d2abe5d), serializer=Config(type=htmlwriter, config={}))}
0 (2013-02-28 19:44:25) TIMER_END{0,Request Processing} Dumping SlingRequestProgressTracker Entries
</pre>
<hr>
<address>ApacheSling/2.2 (Day-Servlet-Engine/4.1.42, Java HotSpot(TM) 64-Bit Server VM 1.6.0_26, Windows 7 6.1 amd64)</address>
</body>
</html>
Abort chunk upload
Client will send delete" operation on upload id to SlingPostServlet to abort chunk upload.
[request]
POST /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Length: 207
Content-Type: multipart/form-data; boundary=tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Content-Disposition: form-data; name=":operation"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
delete
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_--
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>deleted("/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">Modified Resource<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
Implementation
Chunk upload is supported in "modify"operation of SlingPostServlet ( ie. In default operation) . No new operation introduced for it
Chunks Storage
Chunks are saved in temporary location identified by upload id in the following node structure.
[request]
GET /tmp/sling/chunks/f4bc84fbee4d4a608a87c1bf347ddded.infinity.json HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: application/json;charset=UTF-8
Content-Length: 155
Date: Thu, 28 Feb 2013 15:09:53 GMT
{"chunksUploaded":5,"jcr:createdBy":"admin","bytesUploaded":1600,"jcr:created":"Thu Feb 28 2013 20:39:53 GMT+0530","jcr:primaryType":"sling:OrderedFolder","1":{"size":200,":jcr:data":200,"jcr:primaryType":"nt:unstructured"},"2":{"size":300,":jcr:data":300,"jcr:primaryType":"nt:unstructured"},"3":{"size":300,":jcr:data":300,"jcr:primaryType":"nt:unstructured"},"4":{"size":400,":jcr:data":400,"jcr:primaryType":"nt:unstructured"},"5":{"size":400,":jcr:data":400,"jcr:primaryType":"nt:unstructured"}}
Chunk processing
- First/Intermediate chunks: Chunk is saved in temporary location in jcr. Ignores all upload semantics (@TypeHint, etc) and request parameters.
- Last chunk: Stitches all chunks. Process all upload semantics and request parameters and creates jcr node structure at the right location.
