Chunked File Upload SupportPage edited by Shashank GuptaChanges (13)
Full ContentChunked File UploadStatus: DRAFT Use Cases:1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload. ApproachSling provides an extension to SlingPostServlet which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. SlingPostServletupon receiving the last chunk, stitches all chunks into a single file and creates node tree in repository. In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point. Protocol SpecificationUpload chunk using POSTFirst chunk uploadClient uses POST method to parent path to upload binary chunk of file. Client passes ":chunkNumber" and "fileName" as multipart request parameters. First chunk upload request POST /content/dam HTTP/1.1 Authorization: Basic YWRtaW46YWRtaW4= Transfer-Encoding: chunked Content-Type: multipart/form-data; boundary=v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1 (java 1.5) Host: localhost:4502 --v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6 Content-Disposition: form-data; name=":chunkNumber" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1 --v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6 Content-Disposition: form-data; name="*"; filename="catalog.pdf" Content-Type: application/octet-stream Content-Transfer-Encoding: binary $binarydata [response] First chunk upload response
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:24 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08">Modified Resource</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
The response shows that chunk is stored in tmp location. The location tag contains unique chunk upload id which need to be passed on subsequent request. Intermediate chunk uploadClient require to additionally pass chunk upload id. Intermediate chunk upload request POST /content/dam HTTP/1.1 Authorization: Basic YWRtaW46YWRtaW4= Transfer-Encoding: chunked Content-Type: multipart/form-data; boundary=F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1 (java 1.5) Host: localhost:4502 --F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r Content-Disposition: form-data; name=":chunkNumber" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3 --F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r Content-Disposition: form-data; name=":chunkUploadId" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 --F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r Content-Disposition: form-data; name="*"; filename="catalog.pdf" Content-Type: application/octet-stream Content-Transfer-Encoding: binary $binarydata [response] Intermediate chunk upload response
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08">Modified Resource</a></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
Last chunk uploadTo indicate the last chunk, client additionally pass ":lastChunk" to true in request parameter. In last chunk upload response, the changelog has all the modification made in actual path repository. It would the same response as if client has uploaded file in single shot to SlingPostServlet Last chunk upload request POST /content/dam HTTP/1.1 Authorization: Basic YWRtaW46YWRtaW4= Transfer-Encoding: chunked Content-Type: multipart/form-data; boundary=XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1 (java 1.5) Host: localhost:4502 --XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe Content-Disposition: form-data; name=":lastChunk" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit true --XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe Content-Disposition: form-data; name=":chunkNumber" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 6 --XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe Content-Disposition: form-data; name=":chunkUploadId" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 --XHnea0Y5RY92mJwpnKXHfl0dfhcqOCe Content-Disposition: form-data; name="*"; filename="catalog.pdf" Content-Type: application/octet-stream Content-Transfer-Encoding: binary $binarydata [response] Last chunk upload response
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /content/dam</title>
</head>
<body>
<h1>Content modified /content/dam</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><a href="" class="code-quote">"/content/dam" id="Location">/content/dam</a></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/content" id="ParentLocation">/content</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/content/dam</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>created("/content/dam/catalog.pdf");<br/>created("/content/dam/catalog.pdf/jcr:content");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:lastModified");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:mimeType");<br/>modified("/content/dam/catalog.pdf/jcr:content/jcr:data");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><a href="" class="code-quote">"/content/dam">Modified Resource</a></p>
<p><a href="" class="code-quote">"/content">Parent of Modified Resource</a></p>
</body>
</html
The location header provides location of fully uploaded asset.The response contains the $md5_hash of merged binary. Query Sling about the interrupted chunk upload statusClient sent get request on upload to retrieve chunk upload status. Query interrupted chunk upload request GET /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08.json HTTP/1.1 Authorization: Basic YWRtaW46YWRtaW4= Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1 (java 1.5) Host: localhost:4502 [response] Query interrupted chunk upload response
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: application/json;charset=UTF-8
Content-Length: 155
Date: Thu, 28 Feb 2013 14:14:25 GMT
{"chunksUploaded":5,"bytesUploaded":1600,"jcr:createdBy":"admin","jcr:created":"Thu Feb 28 2013 19:44:24 GMT+0530","jcr:primaryType":"sling:OrderedFolder"}
The above response means that 5 continuous chunks with size 1600 (index 0 to index 1599) are successfully uploaded to server. Client should start with :chunkNumber 6 and slice file from offset 1600. If no broken/discontinued upload sling returns 404 not found response.[response] Chunk upload not found response HTTP/1.1 404 Not Found Connection: Close Server: Day-Servlet-Engine/4.1.42 Content-Type: text/html;charset=utf-8 Date: Thu, 28 Feb 2013 14:14:25 GMT Transfer-Encoding: chunked <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html> <head><title>404 No resource found</title></head> <body> <h1>No resource found</h1> <p>Cannot serve request to /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 in /libs/sling/servlet/errorhandler/404.jsp</p> <h3>Request Progress:</h3> <pre> 0 (2013-02-28 19:44:25) TIMER_START{Request Processing} 0 (2013-02-28 19:44:25) COMMENT timer_end format is {<elapsed msec>,<timer name>} <optional message> 0 (2013-02-28 19:44:25) LOG Method=GET, PathInfo=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 0 (2013-02-28 19:44:25) TIMER_START{ResourceResolution} 0 (2013-02-28 19:44:25) TIMER_END{0,ResourceResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 resolves to Resource=, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08] 0 (2013-02-28 19:44:25) LOG Resource Path Info: SlingRequestPathInfo: path='/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08', selectorString='null', extension='null', suffix='null' 0 (2013-02-28 19:44:25) TIMER_START{ServletResolution} 0 (2013-02-28 19:44:25) TIMER_START{resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])} 0 (2013-02-28 19:44:25) LOG {0}: no servlet found 0 (2013-02-28 19:44:25) TIMER_END{0,resolveServlet(, type=sling:nonexisting, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08, resource=[NonExistingResource, path=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08])} Using servlet org.apache.sling.servlets.get.DefaultGetServlet 0 (2013-02-28 19:44:25) TIMER_END{0,ServletResolution} URI=/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 handled by Servlet=org.apache.sling.servlets.get.DefaultGetServlet 0 (2013-02-28 19:44:25) LOG Applying Requestfilters 0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.bgservlets.impl.BackgroundServletStarterFilter 0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.i18n.impl.I18NFilter 0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.designimporter.CanvasPageDeleteRequestFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMRequestFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.adobe.granite.optout.impl.OptOutFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.theme.impl.ThemeResolverFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet 0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter 0 (2013-02-28 19:44:25) LOG RedirectFilter did not redirect (request extension does not match) 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.warp.TimeWarpFilter 0 (2013-02-28 19:44:25) LOG Applying Componentfilters 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMComponentFilter 0 (2013-02-28 19:44:25) LOG Calling filter: com.day.cq.wcm.core.impl.WCMDebugFilter 0 (2013-02-28 19:44:25) TIMER_START{org.apache.sling.servlets.get.DefaultGetServlet#0} 0 (2013-02-28 19:44:25) TIMER_END{0,org.apache.sling.servlets.get.DefaultGetServlet#0} 0 (2013-02-28 19:44:25) LOG Applying Error filters 0 (2013-02-28 19:44:25) LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter 0 (2013-02-28 19:44:25) TIMER_START{handleError:status=404} 0 (2013-02-28 19:44:25) TIMER_END{0,handleError:status=404} Using handler /libs/sling/servlet/errorhandler/404.jsp 0 (2013-02-28 19:44:25) LOG Found processor for post processing ProcessorConfiguration: {contentTypes=[text/html],order=-1, active=true, valid=true, processErrorResponse=true, pipeline=(generator=Config(type=htmlparser, config={}), transformers=(Config(type=linkchecker, config={}), Config(type=mobile, config=org.apache.sling.jcr.resource.JcrPropertyMap@42e95cbf), Config(type=mobiledebug, config=org.apache.sling.jcr.resource.JcrPropertyMap@66c821ea), Config(type=contentsync, config=org.apache.sling.jcr.resource.JcrPropertyMap@5d2abe5d), serializer=Config(type=htmlwriter, config={}))} 0 (2013-02-28 19:44:25) TIMER_END{0,Request Processing} Dumping SlingRequestProgressTracker Entries </pre> <hr> <address>ApacheSling/2.2 (Day-Servlet-Engine/4.1.42, Java HotSpot(TM) 64-Bit Server VM 1.6.0_26, Windows 7 6.1 amd64)</address> </body> </html> Abort chunk uploadClient will send delete" operation on upload id to SlingPostServlet to abort chunk upload. Abort chunk upload request
POST /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Length: 207
Content-Type: multipart/form-data; boundary=tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_
Content-Disposition: form-data; name=":operation"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
delete
--tBnruF3YNdCs-besNxXn8zWO6bN1GL_--
[response] Abort chunk upload response
HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked
<html>
<head>
<title>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</title>
</head>
<body>
<h1>Content modified /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</h1>
<table>
<tbody>
<tr>
<td>Status</td>
<td><div id="Status">200</div></td>
</tr>
<tr>
<td>Message</td>
<td><div id="Message">OK</div></td>
</tr>
<tr>
<td>Location</td>
<td><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></td>
</tr>
<tr>
<td>Parent Location</td>
<td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
</tr>
<tr>
<td>Path</td>
<td><div id="Path">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</div></td>
</tr>
<tr>
<td>Referer</td>
<td><a href="" class="code-quote">"" id="Referer"></a></td>
</tr>
<tr>
<td>ChangeLog</td>
<td><div id="ChangeLog"><pre>deleted("/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08");<br/></pre></div></td>
</tr>
</tbody>
</table>
<p><a href="">Go Back</a></p>
<p><img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_o.gif" alt="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" title="invalid link: /tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" border="0">Modified Resource<img src="" class="code-quote">"/libs/cq/linkchecker/resources/linkcheck_c.gif" border="0"></p>
<p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
</body>
</html>
Change Notification Preferences
View Online
|
View Changes
|
Add Comment
|
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
- [CONF] Apache Sling > Chunked File Upload Support confluence
