[CONF] Apache Sling > Chunked File Upload Support

confluence Thu, 28 Feb 2013 06:28:27 -0800

     Chunked File Upload Support
    Page edited by             Shashank Gupta
    
        

                         Changes (3)
                                 
    

                    
    
            ...

            1 
--v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6 

            Content-Disposition: form-data; name="*"; filename="test340645987550795425chunkupload" filename="$filename" 

            Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary 

            ...

            </html> 
{code} 

            *The location tag contains unique chunk upload id which need to be passed on subsequent request.* 

             

            h4. Intermediate chunk upload  
Client require to additionally  pass chunk upload id  
\[request\] 
{code:title= Intermediate chunk upload request} 
POST /content/dam HTTP/1.1 
Authorization: Basic YWRtaW46YWRtaW4= 
Transfer-Encoding: chunked 
Content-Type: multipart/form-data; boundary=F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Connection: Keep-Alive 
User-Agent: Apache-HttpClient/4.1 (java 1.5) 
Host: localhost:4502 
 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name=":chunkNumber" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
3 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name=":chunkUploadId" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name="*"; filename="$filename" 
Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary 
 
$binarydata 
{code} 
 
\[response\] 
{code:title= Intermediate chunk upload response} 
HTTP/1.1 200 OK 
Connection: Keep-Alive 
Server: Day-Servlet-Engine/4.1.42 
Content-Type: text/html;charset=UTF-8 
Date: Thu, 28 Feb 2013 14:14:25 GMT 
Transfer-Encoding: chunked 
 
<html> 
<head> 
    <title>Content modified /tmp/sling/chunks</title> 
</head> 
    <body> 
    <h1>Content modified /tmp/sling/chunks</h1> 
    <table> 
        <tbody> 
            <tr> 
                <td>Status</td> 
                <td><div id="Status">200</div></td> 
            </tr> 
            <tr> 
                <td>Message</td> 
                <td><div id="Message">OK</div></td> 
            </tr> 
            <tr> 
                <td>Location</td> 
                <td><a href="" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td> 
            </tr> 
            <tr> 
                <td>Parent Location</td> 
                <td><a href="" id="ParentLocation">/tmp/sling/chunks</a></td> 
            </tr> 
            <tr> 
                <td>Path</td> 
                <td><div id="Path">/tmp/sling/chunks</div></td> 
            </tr> 
            <tr> 
                <td>Referer</td> 
                <td><a href="" id="Referer"></a></td> 
            </tr> 
            <tr> 
                <td>ChangeLog</td> 
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td> 
            </tr> 
        </tbody> 
    </table> 
    <p><a href="" Back</a></p> 
    <p><a href="" Resource</a></p> 
    <p><a href="" of Modified Resource</a></p> 
    </body> 
</html> 
{code} 
 
 

            h4. Upload of last chunk 
Client appends "/last" to upload url to signify that it would be last that needs to be uploaded. The last chunk upload request can optionally be multipart request if client requires to send additional parameters to sling. 

            ...

    
            
    
                            Full Content
                    
        

    1 Chunked File Upload
    2 Use Cases:
    3 Approach
    4 Protocol Specification

    4.1 Upload chunk using POST
    4.2 Upload chunk using PATCH
    4.3 Query to retrieve last successful chunk upload
    4.4 Abort chunked upload



Chunked File Upload

Status: DRAFT

Created: 20. January 2013

Author: shgupta

JIRA: SLING-2707

References: - http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Updated: -

Use Cases:
1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload.

2. Fault tolerant uploads - Sling provides capability to resume upload from failure point. It doesn't require client to restart the complete upload process. 

Approach
Sling provides an extension to SlingPostServlet which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. SlingPostServletupon receiving the last chunk, stitches all chunks into a single file and creates node tree in repository. 

In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point. 

Protocol Specification 


Upload chunk using POST 

First chunk upload
Client uses POST method to parent path to upload binary chunk of file. Client passes ":chunkNumber" and "fileName" as multipart request parameters.

[request]
First chunk upload request

POST /content/dam HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502

--v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6
Content-Disposition: form-data; name=":chunkNumber"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

1
--v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6
Content-Disposition: form-data; name="*"; filename="$filename"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

$binarydata



[response]
First chunk upload response

TTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:24 GMT
Transfer-Encoding: chunked

<html>
<head>
    <title>Content modified /tmp/sling/chunks</title>
</head>
    <body>
    <h1>Content modified /tmp/sling/chunks</h1>
    <table>
        <tbody>
            <tr>
                <td>Status</td>
                <td><div id="Status">200</div></td>
            </tr>
            <tr>
                <td>Message</td>
                <td><div id="Message">OK</div></td>
            </tr>
            <tr>
                <td>Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td>
            </tr>
            <tr>
                <td>Parent Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
            </tr>
            <tr>
                <td>Path</td>
                <td><div id="Path">/tmp/sling/chunks</div></td>
            </tr>
            <tr>
                <td>Referer</td>
                <td><a href="" class="code-quote">"" id="Referer"></a></td>
            </tr>
            <tr>
                <td>ChangeLog</td>
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td>
            </tr>
        </tbody>
    </table>
    <p><a href="">Go Back</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08">Modified Resource</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
    </body>
</html>


The location tag contains unique chunk upload id which need to be passed on subsequent request.

Intermediate chunk upload 
Client require to additionally  pass chunk upload id 

[request]
Intermediate chunk upload request

POST /content/dam HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502

--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name=":chunkNumber"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

3
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name=":chunkUploadId"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r
Content-Disposition: form-data; name="*"; filename="$filename"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

$binarydata



[response]
Intermediate chunk upload response

HTTP/1.1 200 OK
Connection: Keep-Alive
Server: Day-Servlet-Engine/4.1.42
Content-Type: text/html;charset=UTF-8
Date: Thu, 28 Feb 2013 14:14:25 GMT
Transfer-Encoding: chunked

<html>
<head>
    <title>Content modified /tmp/sling/chunks</title>
</head>
    <body>
    <h1>Content modified /tmp/sling/chunks</h1>
    <table>
        <tbody>
            <tr>
                <td>Status</td>
                <td><div id="Status">200</div></td>
            </tr>
            <tr>
                <td>Message</td>
                <td><div id="Message">OK</div></td>
            </tr>
            <tr>
                <td>Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td>
            </tr>
            <tr>
                <td>Parent Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
            </tr>
            <tr>
                <td>Path</td>
                <td><div id="Path">/tmp/sling/chunks</div></td>
            </tr>
            <tr>
                <td>Referer</td>
                <td><a href="" class="code-quote">"" id="Referer"></a></td>
            </tr>
            <tr>
                <td>ChangeLog</td>
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td>
            </tr>
        </tbody>
    </table>
    <p><a href="">Go Back</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08">Modified Resource</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
    </body>
</html>




Upload of last chunk
Client appends "/last" to upload url to signify that it would be last that needs to be uploaded. The last chunk upload request can optionally be multipart request if client requires to send additional parameters to sling.

[request]
Upload last chunk request

POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res/last HTTP/1.1
Host: localhost:4502
Date: <date>
Content-type: multipart/form-data, boundary=AaB03x

--AaB03x
content-disposition: form-data; name="fileSize"

210000000
--AaB03x
content-disposition: form-data; name="file"; fileName="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--



[response]
Upload last chunk response

HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close


The location header provides location of fully uploaded asset.The response contains the $md5_hash of merged binary. 

Upload chunk using PATCH
Sling support three patch formats to upload chunks on sling. All other semantics remain same as  Upload chunk using POST

	append/binary-patch: Use this content type if request contain only binary chunk data. If request has additional parameters that need to be passed, then use application/json-patch or multipart/form-data-patch content type.

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/octet-stream-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
$binarydata


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502


	application/json-patch: JSON format patch document to pass upload attributes along with binary chunk

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/json-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
[
  {
	"jcr:data": "$binarydata",
	"fileName": "catalog.pdf",
	" field1": "$field1"
  }
]


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502


	multipart/form-data-patch: multipart/form-data-patch format to pass upload attributes along with binary chunk

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: multipart/form-data-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen

--AaB03x
content-disposition: form-data; name="field1"

$field1
--AaB03x
content-disposition: form-data; name="field2"

$field2
--AaB03x
content-disposition: form-data; name="userfile"; filename="$filename"
Content-Type: $mimetype
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502





Query to retrieve last successful chunk upload
Returns the number of last chunk upload and bytes uploaded successfully on sling. 

[request]
Get last successful chunk upload request

GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: <date>
Content-Length: 0


[response]
Get last successful upload chunk response

HTTP/1.1 200/OK
Connection: close

{
    "chunkNumber": 8
    "bytesuploaded": 19872
}


If no broken/discontinued upload sling returns 404 not found response. 

[response]
Chunked upload not found response

HTTP/1.1 404/Not Found



Abort chunked upload
[request]
Abort chunked upload Request

DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: <date>


[response]
Abort chunked upload response

HTTP/1.1 200/OK
Connection: close




    
        
        
            Change Notification Preferences
        
        View Online
        |
        View Changes
                |
        Add Comment
 ...
 
--v0_zWJbImtVo0uK-bJWayWUiT3zTBIj6
 Content-Disposition: form-data; name="*"; filename="test340645987550795425chunkupload" filename="$filename"
 Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary
 ...
 </html> 
{code}
 *The location tag contains unique chunk upload id which need to be passed on subsequent request.*
+h4. Intermediate chunk upload  
Client require to additionally  pass chunk upload id  
\[request\] 
{code:title= Intermediate chunk upload request} 
POST /content/dam HTTP/1.1 
Authorization: Basic YWRtaW46YWRtaW4= 
Transfer-Encoding: chunked 
Content-Type: multipart/form-data; boundary=F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Connection: Keep-Alive 
User-Agent: Apache-HttpClient/4.1 (java 1.5) 
Host: localhost:4502 
 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name=":chunkNumber" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
3 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name=":chunkUploadId" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08 
--F1MYdmO-MYq-wLW2P2tWIu-7KUH3-r 
Content-Disposition: form-data; name="*"; filename="$filename" 
Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary 
 
$binarydata 
{code} 
 
\[response\] 
{code:title= Intermediate chunk upload response} 
HTTP/1.1 200 OK 
Connection: Keep-Alive 
Server: Day-Servlet-Engine/4.1.42 
Content-Type: text/html;charset=UTF-8 
Date: Thu, 28 Feb 2013 14:14:25 GMT 
Transfer-Encoding: chunked 
 
<html> 
<head> 
    <title>Content modified /tmp/sling/chunks</title> 
</head> 
    <body> 
    <h1>Content modified /tmp/sling/chunks</h1> 
    <table> 
        <tbody> 
            <tr> 
                <td>Status</td> 
                <td><div id="Status">200</div></td> 
            </tr> 
            <tr> 
                <td>Message</td> 
                <td><div id="Message">OK</div></td> 
            </tr> 
            <tr> 
                <td>Location</td> 
                <td><a href="" id="Location">/tmp/sling/chunks/21d36856b76045da8719ac932f88fe08</a></td> 
            </tr> 
            <tr> 
                <td>Parent Location</td> 
                <td><a href="" id="ParentLocation">/tmp/sling/chunks</a></td> 
            </tr> 
            <tr> 
                <td>Path</td> 
                <td><div id="Path">/tmp/sling/chunks</div></td> 
            </tr> 
            <tr> 
                <td>Referer</td> 
                <td><a href="" id="Referer"></a></td> 
            </tr> 
            <tr> 
                <td>ChangeLog</td> 
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td> 
            </tr> 
        </tbody> 
    </table> 
    <p><a href="" Back</a></p> 
    <p><a href="" Resource</a></p> 
    <p><a href="" of Modified Resource</a></p> 
    </body> 
</html> 
{code}
 h4. Upload of last chunk 
Client appends "/last" to upload url to signify that it would be last that needs to be uploaded. The last chunk upload request can optionally be multipart request if client requires to send additional parameters to sling.
 ...

[CONF] Apache Sling > Chunked File Upload Support

Chunked File Upload Support

Page edited by Shashank Gupta

Changes (3)

Full Content

Chunked File Upload

Use Cases:

Approach

Protocol Specification

Upload chunk using POST

First chunk upload

Intermediate chunk upload

Upload of last chunk

Upload chunk using PATCH

Query to retrieve last successful chunk upload

Abort chunked upload

Reply via email to