Chunked File Upload
Status: DRAFT
Created: 20. January 2013
Author: shgupta
JIRA: SLING-2707
References: - http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
Updated: -
Use Cases:
1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload.
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point. It doesn't require client to restart the complete upload process.
Approach
Sling provides an extension to SlingPostServlet which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk has "Offset" attribute which identify chunk position in complete file. SlingPostServletupon receiving the last chunk, stitches all chunks into a single file and save them to the final destination.
In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point.
Content Model
Chunks are stored within actual path in sling:chunkMixin mixin node type. The content model to store chunk is defined as follows:
[sling:chunkNode] > nt:hierarchyNode
primaryitem jcr:data
- sling:offset (long) mandatory
- jcr:data (binary) mandatory
[sling:chunkMixin]
mixin
- sling:fileLength (long)
- sling:chunksLength (long)
+ * (sling:chunkNode) multiple
The typical nt:file node under chunked upload would look like
/content/dam/folder/catalog.pdf [nt:file]
+ jcr:content [nt:resource] [sling:chunkMixin]
- jcr:data = "" until completed
- sling:fileLength = 982145 - sling:chunksLength = 30000 + chunk_0-9999 [sling:chunkNode]
- sling:offset =0
- jcr:data [binary data]
+ chunk_10000-19999 [sling:chunkNode]
+ ....
Protocol Specification
Upload chunk using POST
Chunk upload request
Client uses POST method to parent path to upload binary chunk of file. Sling client passes $filename@Offset and $filename@Length as multipart request parameters. "Offset" indicate chunk's data offset in complete file. "Length" is optional parameter and indicates length of complete file. If "Length" parameter is known, Sling automatically calculates if request is last chunk request and according stitches all chunks and store it into final destination.
[request]
POST /content/dam/folder HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Length"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
1000
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Offset"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
400
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf"; filename="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
$binary-data
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe--
[response]
The response shows that chunk is stored within the actual path in sling:chunkMixin node type
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Mon, 06 May 2013 14:42:22 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /content/dam/folder</title>
</head>
<body>
<h1>Content modified /content/dam/folder</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/content/dam/folder" id="Location">/content/dam/folder</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/" id="ParentLocation">/</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/content/dam/folder</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:lastModified");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:mimeType");
<br/>created("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799/jcr:data");<br/>
modified("/content/dam/folder/catalog.pdf/jcr:content/chunk_400_799/sling:offset");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/sling:chunksLength");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/content/dam/folder">Modified Resource</a></p>
<p><a href="" class="code-quote">"/">Parent of Modified Resource</a></p>
</body>
</html>
Chunk Upload in streaming use case
[request]
POST /content/dam/folder HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=lMaKIb2KPscWvPV8B0fULKkKayVtcxugD8Lt
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--lMaKIb2KPscWvPV8B0fULKkKayVtcxugD8Lt
Content-Disposition: form-data; name="catalog.pdf@Completed"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
true
--lMaKIb2KPscWvPV8B0fULKkKayVtcxugD8Lt
Content-Disposition: form-data; name="catalog.pdf@Offset"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
400
--lMaKIb2KPscWvPV8B0fULKkKayVtcxugD8Lt
Content-Disposition: form-data; name="catalog.pdf"; filename="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
$binary_data
--lMaKIb2KPscWvPV8B0fULKkKayVtcxugD8Lt--
[response]
The response shows that merge chunks uploaded at final destination and deletion of chunk upload metadata.
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Mon, 06 May 2013 15:52:16 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /content/dam/folder</title>
</head>
<body>
<h1>Content modified /content/dam/folder</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/content/dam/folder" id="Location">/content/dam/folder</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/" id="ParentLocation">/</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/content/dam/folder</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:lastModified");<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:mimeType");
<br/>modified("/content/dam/folder/catalog.pdf/jcr:content/jcr:data");<br/>deleted("/content/dam/folder/catalog.pdf/jcr:content/chunk_0_199");
<br/>deleted("/content/dam/folder/catalog.pdf/jcr:content/chunk_200_399");<br/>deleted("/content/dam/folder/catalog.pdf/jcr:content/sling:chunksLength");
<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/content/dam/folder">Modified Resource</a></p>
<p><a href="" class="code-quote">"/">Parent of Modified Resource</a></p>
</body>
</html>
Query Sling about the interrupted chunk upload status
Client sent get request on upload to retrieve chunk upload status.
[request]
GET Authorization: Basic YWRtaW46YWRtaW4=
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
[response]
sling:chunksLength property indicate cumulative length of all chunks stored in sling. Sling client should resume upload from sling:chunksLength property's value offset
{"jcr:createdBy":"admin","jcr:created":"Mon May 06 2013 21:31:22 GMT+0530","jcr:primaryType":"nt:file","jcr:content":{"jcr:lastModifiedBy":"admin","jcr:uuid":"845e9cee-f963-4f72-b115-fa021859c809",
":jcr:data":0,"jcr:mixinTypes":["sling:chunkMixin"],"sling:chunksLength":200,"jcr:mimeType":"application/pdf","jcr:lastModified":"Mon May 06 2013 21:31:22 GMT+0530",
"jcr:primaryType":"nt:resource","sling:fileLength":1700,"chunk_0_199":{"jcr:createdBy":"admin",":jcr:data":200,"sling:offset":0,"jcr:created":"Mon May 06 2013 21:31:22 GMT+0530","jcr:primaryType":"sling:chunkNode"}}
If no broken/discontinued upload sling returns 404 not found response.
[response]
HTTP/1.1 404 Not Found
Connection: Close
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=utf-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<!DOCTYPE HTML PUBLIC "->
<html>
<head><title>404 No resource found</title></head>
<body>
<h1>No resource found</h1>
<p>Cannot serve request to /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 in /libs/sling/servlet/errorhandler/404.jsp</p>
<h3>Request Progress:</h3>
<pre>
0 (2013-02-28 19:44:25) TIMER_START{Request Processing}
0 (2013-02-28 19:44:25) COMMENT timer_end format is {<elapsed msec>,<timer name>} <optional message>
0 (2013-02-28 19:44:25) LOG Method=GET, PathInfo=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08
0 (2013-02-28 19:44:25) TIMER_START{ResourceResolution}
0 (2013-02-28 19:44:25) TIMER_END{0,ResourceResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 resolves to Resource=, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08]
0 (2013-02-28 19:44:25) LOG Resource Path Info: SlingRequestPathInfo: path='/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08', selectorString='null', extension='null', suffix='null'
0 (2013-02-28 19:44:25) TIMER_START{ServletResolution}
0 (2013-02-28 19:44:25) TIMER_START{resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])}
0 (2013-02-28 19:44:25) LOG {0}: no servlet found
0 (2013-02-28 19:44:25) TIMER_END{0,resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])} Using servlet org.apache.sling.servlets.get.DefaultGetServlet
0 (2013-02-28 19:44:25) TIMER_END{0,ServletResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 handled by Servlet=org.apache.sling.servlets.get.DefaultGetServlet
0 (2013-02-28 19:44:25) LOG Applying Requestfilters
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.bgservlets.impl.BackgroundServletStarterFilter
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.i18n.impl.I18NFilter
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.designimporter.CanvasPageDeleteRequestFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMRequestFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.adobe.granite.optout.impl.OptOutFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.theme.impl.ThemeResolverFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter
0 (2013-02-28 19:44:25) LOG RedirectFilter did not redirect (request extension does not match)
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.warp.TimeWarpFilter
0 (2013-02-28 19:44:25) LOG Applying Componentfilters
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMComponentFilter
0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMDebugFilter
0 (2013-02-28 19:44:25) TIMER_START{org.apache.sling.servlets.get.DefaultGetServlet#0}
0 (2013-02-28 19:44:25) TIMER_END{0,org.apache.sling.servlets.get.DefaultGetServlet#0}
0 (2013-02-28 19:44:25) LOG Applying Error filters
0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter
0 (2013-02-28 19:44:25) TIMER_START{handleError:status=404}
0 (2013-02-28 19:44:25) TIMER_END{0,handleError:status=404} Using handler /libs/sling/servlet/errorhandler/404.jsp
0 (2013-02-28 19:44:25) LOG Found processor for post processing ProcessorConfiguration: {contentTypes=[text/html],order=-1, active=true, valid=true, processErrorResponse=true, pipeline=(generator=Config(type=htmlparser, config={}), transformers=(Config(type=linkchecker, config={}), Config(type=mobile, config=org.apache.sling.jcr.resource.JcrPropertyMap@42e95cbf), Config(type=mobiledebug, config=org.apache.sling.jcr.resource.JcrPropertyMap@66c821ea), Config(type=contentsync, config=org.apache.sling.jcr.resource.JcrPropertyMap@5d2abe5d), serializer=Config(type=htmlwriter, config={}))}
0 (2013-02-28 19:44:25) TIMER_END{0,Request Processing} Dumping SlingRequestProgressTracker Entries
</pre>
<hr>
<address>ApacheSling/2.2 (Day-Servlet-Engine/4.1.42, Java HotSpot(TM) 64-Bit Server VM 1.6.0_26, Windows 7 6.1 amd64)</address>
</body>
</html>
Abort chunk upload
Client will send delete" operation on upload id to SlingPostServlet to abort chunk upload.
[request]
POST /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Length: 207
Content-Type: multipart/form-data; boundary=tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Content-Disposition: form-data; name=":operation"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
delete
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_--
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>deleted("/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">Modified Resource<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
Implementation
Chunk upload is supported in "modify"operation of SlingPostServlet ( ie. In default operation) . No new operation introduced for it
Chunks Storage
Chunks are saved in temporary location identified by upload id in the following node structure.
[request]
GET /tmp/sling/chunks/f4bc84fbee4d4a608a87c1bf347ddded.infinity.json HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
[response]
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: application/json;charset=UTF-8
Content-Length: 155
Date: Thu, 28 Feb 2013 15:09:53 GMT
{"chunksUploaded":5,"jcr:createdBy":"admin","bytesUploaded":1600,"jcr:created":"Thu Feb 28 2013 20:39:53 GMT+0530","jcr:primaryType":"sling:OrderedFolder","1":{"size":200,":jcr:data":200,"jcr:primaryType":"nt:unstructured"},"2":{"size":300,":jcr:data":300,"jcr:primaryType":"nt:unstructured"},"3":{"size":300,":jcr:data":300,"jcr:primaryType":"nt:unstructured"},"4":{"size":400,":jcr:data":400,"jcr:primaryType":"nt:unstructured"},"5":{"size":400,":jcr:data":400,"jcr:primaryType":"nt:unstructured"}}
Chunk processing
- First/Intermediate chunks: Chunk is saved in temporary location in jcr. Ignores all upload semantics (@TypeHint, etc) and request parameters.
- Last chunk: Stitches all chunks. Process all upload semantics and request parameters and creates jcr node structure at the right location.
