[CONF] Apache Sling > Chunked File Upload Support

confluence Thu, 28 Feb 2013 06:11:08 -0800

     Chunked File Upload Support
    Page edited by             Shashank Gupta
    
        

                         Changes (16)
                                 
    

                    
    
            ...

             
h1. Approach 

            Sling provides an endpoint which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling endpoint upon receiving the last chunk, stitches all chunks into a single file. The file inputstream is passed to inner layers for further processing. 

            Sling provides an extension to [SlingPostServlet|http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html] which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling [SlingPostServlet|http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html]upon receiving the last chunk, stitches all chunks into a single file and creates node tree in repository.  

             
In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point.  

            ...

            h1. Protocol Specification  
 

            h3. Advertising chunk upload support in sling 
Sling advertise its support for chunk upload by adding POST and PATCH methods to the listing of allowed methods in the "Allow" OPTIONS response header defined in HTTP/1.1. 
\[request\] 
{code:title= OPTION request} 
OPTIONS /content/dam/dam-folder/catalog.pdf.chunk HTTP/1.1 
{code} 
\[response\] 
{code:title= OPTION response} 
HTTP/1.1 200 OK 
Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH 
Accept-Patch: append/binary-patch, application/json-patch, application/json-patch 
{code} 
[Accept-Patch|http://greenbytes.de/tech/webdav/rfc5789.html#accept-patch] used to specify the patch document formats accepted by the server. Chunk upload supports three  append/binary-patch, application/json-patch, application/json-patch formats. 

             
h3. Upload chunk using POST  
 

            h4. Upload of first and intermediate chunks 

            h4. First chunk upload 

            Client uses POST method to parent path to upload binary chunk of file. Client passes ":chunkNumber" and "fileName" as multipart request parameters. 

            \[request\] 

            {code:title= Upload the first/intermediate chunk request} 
POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1 
Host: localhost:4502 
If-Match: "e0023aa4e" 

            {code:title= First chunk upload request} 
POST /content/dam 

            Content-Type: --ULPTM_fXARgFjsvsyvaO_W-2loSNQw 

            Date: <date> 
Content-Length: $requestlen 

            Content-Disposition: form-data; name=":chunkNumber" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
1 
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw 
Content-Disposition: form-data; name="*"; filename="$fileName" 
Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary 
$binarydata 

            {code} 
 
\[response\] 

            {code:title= First/Intermediate chunk upload response} 

            HTTP/1.1 201 CREATED 
Content-Length: 0 
ETag: "e0023aa4f" 
Location: /content/dam/dam-folder/catalog.pdf.chunk 
Connection: close 
Server: localhost:4502 

            HTTP/1.1 200 OK 
<html> 
<head> 
    <title>Content modified /tmp/sling/chunks</title> 
</head> 
    <body> 
    <h1>Content modified /tmp/sling/chunks</h1> 
    <table> 
        <tbody> 
            <tr> 
                <td>Status</td> 
                <td><div id="Status">200</div></td> 
            </tr> 
            <tr> 
                <td>Message</td> 
                <td><div id="Message">OK</div></td> 
            </tr> 
            <tr> 
                <td>Location</td> 
                <td><a href="" id="Location">/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c</a></td> 
            </tr> 
            <tr> 
                <td>Parent Location</td> 
                <td><a href="" id="ParentLocation">/tmp/sling/chunks</a></td> 
            </tr> 
            <tr> 
                <td>Path</td> 
                <td><div id="Path">/tmp/sling/chunks</div></td> 
            </tr> 
            <tr> 
                <td>Referer</td> 
                <td><a href="" id="Referer"></a></td> 
            </tr> 
            <tr> 
                <td>ChangeLog</td> 
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td> 
            </tr> 
        </tbody> 
    </table> 
    <p><a href="" Back</a></p> 
    <p><a href="" Resource</a></p> 
    <p><a href="" of Modified Resource</a></p> 
    </body> 
</html> 

            {code} 

            The location header provides path to retrieve chunk upload attributes. 
Concurrent request from multiple client requests  will corrupt the resource. Clients should use a conditional request such that the request will fail if the resource has been updated since the client last accessed the resource. For example, the client can use a strong [ETag|http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19] in an [If-Match|http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.24]  header on the chunk request. 

            The location tag contains unique chunk upload which need to be passed on subsequent request.  

             
h4. Upload of last chunk 

            ...

    
            
    
                            Full Content
                    
        

    1 Chunked File Upload
    2 Use Cases:
    3 Approach
    4 Protocol Specification

    4.1 Upload chunk using POST
    4.2 Upload chunk using PATCH
    4.3 Query to retrieve last successful chunk upload
    4.4 Abort chunked upload



Chunked File Upload

Status: DRAFT

Created: 20. January 2013

Author: shgupta

JIRA: SLING-2707

References: - http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Updated: -

Use Cases:
1. Large file upload - With high speed internet connections, advent of cloud and HD going mainstream, Sling support large files (> 2GB) upload.

2. Fault tolerant uploads - Sling provides capability to resume upload from failure point. It doesn't require client to restart the complete upload process. 

Approach
Sling provides an extension to SlingPostServlet which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling SlingPostServletupon receiving the last chunk, stitches all chunks into a single file and creates node tree in repository. 

In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point. 

Protocol Specification 


Upload chunk using POST 

First chunk upload
Client uses POST method to parent path to upload binary chunk of file. Client passes ":chunkNumber" and "fileName" as multipart request parameters.

[request]
First chunk upload request

POST /content/dam
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw
Content-Disposition: form-data; name=":chunkNumber"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

1
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw
Content-Disposition: form-data; name="*"; filename="$fileName"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
$binarydata



[response]
First chunk upload response

HTTP/1.1 200 OK
<html>
<head>
    <title>Content modified /tmp/sling/chunks</title>
</head>
    <body>
    <h1>Content modified /tmp/sling/chunks</h1>
    <table>
        <tbody>
            <tr>
                <td>Status</td>
                <td><div id="Status">200</div></td>
            </tr>
            <tr>
                <td>Message</td>
                <td><div id="Message">OK</div></td>
            </tr>
            <tr>
                <td>Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c" id="Location">/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c</a></td>
            </tr>
            <tr>
                <td>Parent Location</td>
                <td><a href="" class="code-quote">"/tmp/sling/chunks" id="ParentLocation">/tmp/sling/chunks</a></td>
            </tr>
            <tr>
                <td>Path</td>
                <td><div id="Path">/tmp/sling/chunks</div></td>
            </tr>
            <tr>
                <td>Referer</td>
                <td><a href="" class="code-quote">"" id="Referer"></a></td>
            </tr>
            <tr>
                <td>ChangeLog</td>
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td>
            </tr>
        </tbody>
    </table>
    <p><a href="">Go Back</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c">Modified Resource</a></p>
    <p><a href="" class="code-quote">"/tmp/sling/chunks">Parent of Modified Resource</a></p>
    </body>
</html>


The location tag contains unique chunk upload which need to be passed on subsequent request. 

Upload of last chunk
Client appends "/last" to upload url to signify that it would be last that needs to be uploaded. The last chunk upload request can optionally be multipart request if client requires to send additional parameters to sling.

[request]
Upload last chunk request

POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res/last HTTP/1.1
Host: localhost:4502
Date: <date>
Content-type: multipart/form-data, boundary=AaB03x

--AaB03x
content-disposition: form-data; name="fileSize"

210000000
--AaB03x
content-disposition: form-data; name="file"; fileName="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--



[response]
Upload last chunk response

HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close


The location header provides location of fully uploaded asset.The response contains the $md5_hash of merged binary. 

Upload chunk using PATCH
Sling support three patch formats to upload chunks on sling. All other semantics remain same as  Upload chunk using POST

	append/binary-patch: Use this content type if request contain only binary chunk data. If request has additional parameters that need to be passed, then use application/json-patch or multipart/form-data-patch content type.

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/octet-stream-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
$binarydata


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502


	application/json-patch: JSON format patch document to pass upload attributes along with binary chunk

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: application/json-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen
[
  {
	"jcr:data": "$binarydata",
	"fileName": "catalog.pdf",
	" field1": "$field1"
  }
]


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502


	multipart/form-data-patch: multipart/form-data-patch format to pass upload attributes along with binary chunk

[request]
Upload the first/intermediate chunk request

PATCH /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1
Content-Type: multipart/form-data-patch
If-Match: "e0023aa4e"
Content-Length: $requestlen

--AaB03x
content-disposition: form-data; name="field1"

$field1
--AaB03x
content-disposition: form-data; name="field2"

$field2
--AaB03x
content-disposition: form-data; name="userfile"; filename="$filename"
Content-Type: $mimetype
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--


[response]
First/Intermediate chunk upload response

HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: "e0023aa4f"
Connection: close
Server: localhost:4502





Query to retrieve last successful chunk upload
Returns the number of last chunk upload and bytes uploaded successfully on sling. 

[request]
Get last successful chunk upload request

GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: <date>
Content-Length: 0


[response]
Get last successful upload chunk response

HTTP/1.1 200/OK
Connection: close

{
    "chunkNumber": 8
    "bytesuploaded": 19872
}


If no broken/discontinued upload sling returns 404 not found response. 

[response]
Chunked upload not found response

HTTP/1.1 404/Not Found



Abort chunked upload
[request]
Abort chunked upload Request

DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: <date>


[response]
Abort chunked upload response

HTTP/1.1 200/OK
Connection: close




    
        
        
            Change Notification Preferences
        
        View Online
        |
        View Changes
                |
        Add Comment
 ...
 h1. Approach
-Sling provides an endpoint which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling endpoint upon receiving the last chunk, stitches all chunks into a single file. The file inputstream is passed to inner layers for further processing.
+Sling provides an extension to [SlingPostServlet|http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html] which accepts file chunks in accordance with a specified protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish from the rest chunks. Sling [SlingPostServlet|http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html]upon receiving the last chunk, stitches all chunks into a single file and creates node tree in repository.
 In case of upload failures, sling provides support to query the last chunk uploaded till failure point. Client resumes chunk upload from last failure point.
 ...
 h1. Protocol Specification
-h3. Advertising chunk upload support in sling 
Sling advertise its support for chunk upload by adding POST and PATCH methods to the listing of allowed methods in the "Allow" OPTIONS response header defined in HTTP/1.1. 
\[request\] 
{code:title= OPTION request} 
OPTIONS /content/dam/dam-folder/catalog.pdf.chunk HTTP/1.1 
{code} 
\[response\] 
{code:title= OPTION response} 
HTTP/1.1 200 OK 
Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH 
Accept-Patch: append/binary-patch, application/json-patch, application/json-patch 
{code} 
[Accept-Patch|http://greenbytes.de/tech/webdav/rfc5789.html#accept-patch] used to specify the patch document formats accepted by the server. Chunk upload supports three  append/binary-patch, application/json-patch, application/json-patch formats.
 h3. Upload chunk using POST
-h4. Upload of first and intermediate chunks
+h4. First chunk upload
 Client uses POST method to parent path to upload binary chunk of file. Client passes ":chunkNumber" and "fileName" as multipart request parameters.
 \[request\]
-{code:title= Upload the first/intermediate chunk request} 
POST /content/dam/dam-folder/catalog.pdf.chunk.<chunk_number>.res HTTP/1.1 
Host: localhost:4502 
If-Match: "e0023aa4e"
+{code:title= First chunk upload request} 
POST /content/dam
 Content-Type: --ULPTM_fXARgFjsvsyvaO_W-2loSNQw
-Date: <date> 
Content-Length: $requestlen
+Content-Disposition: form-data; name=":chunkNumber" 
Content-Type: text/plain; charset=UTF-8 
Content-Transfer-Encoding: 8bit 
 
1 
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw 
Content-Disposition: form-data; name="*"; filename="$fileName" 
Content-Type: application/octet-stream 
Content-Transfer-Encoding: binary 
$binarydata
 {code} 
 
\[response\]
 {code:title= First/Intermediate chunk upload response}
-HTTP/1.1 201 CREATED 
Content-Length: 0 
ETag: "e0023aa4f" 
Location: /content/dam/dam-folder/catalog.pdf.chunk 
Connection: close 
Server: localhost:4502
+HTTP/1.1 200 OK 
<html> 
<head> 
    <title>Content modified /tmp/sling/chunks</title> 
</head> 
    <body> 
    <h1>Content modified /tmp/sling/chunks</h1> 
    <table> 
        <tbody> 
            <tr> 
                <td>Status</td> 
                <td><div id="Status">200</div></td> 
            </tr> 
            <tr> 
                <td>Message</td> 
                <td><div id="Message">OK</div></td> 
            </tr> 
            <tr> 
                <td>Location</td> 
                <td><a href="" id="Location">/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c</a></td> 
            </tr> 
            <tr> 
                <td>Parent Location</td> 
                <td><a href="" id="ParentLocation">/tmp/sling/chunks</a></td> 
            </tr> 
            <tr> 
                <td>Path</td> 
                <td><div id="Path">/tmp/sling/chunks</div></td> 
            </tr> 
            <tr> 
                <td>Referer</td> 
                <td><a href="" id="Referer"></a></td> 
            </tr> 
            <tr> 
                <td>ChangeLog</td> 
                <td><div id="ChangeLog">&lt;pre&gt;&lt;/pre&gt;</div></td> 
            </tr> 
        </tbody> 
    </table> 
    <p><a href="" Back</a></p> 
    <p><a href="" Resource</a></p> 
    <p><a href="" of Modified Resource</a></p> 
    </body> 
</html>
 {code}
-The location header provides path to retrieve chunk upload attributes. 
Concurrent request from multiple client requests  will corrupt the resource. Clients should use a conditional request such that the request will fail if the resource has been updated since the client last accessed the resource. For example, the client can use a strong [ETag|http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19] in an [If-Match|http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.24]  header on the chunk request.
+The location tag contains unique chunk upload which need to be passed on subsequent request.
 h4. Upload of last chunk
 ...

[CONF] Apache Sling > Chunked File Upload Support

Chunked File Upload Support

Page edited by Shashank Gupta

Changes (16)

Full Content

Chunked File Upload

Use Cases:

Approach

Protocol Specification

Upload chunk using POST

First chunk upload

Upload of last chunk

Upload chunk using PATCH

Query to retrieve last successful chunk upload

Abort chunked upload

Reply via email to