[jira] [Commented] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079086#comment-17079086 ] Ian Boston commented on SLING-9060: --- For completeness, it was decided not to pursue this issue as the immediate need for it has passed. > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > Time Spent: 14h 20m > Remaining Estimate: 0h > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-9060. --- Resolution: Won't Fix > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > Time Spent: 14h 20m > Remaining Estimate: 0h > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reopened SLING-9060: --- Trying again to find a compromise. > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-9060. --- Resolution: Won't Fix API PR was voted -1 Closing as wont fix Will look for alternatives to address the problem. > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032861#comment-17032861 ] Ian Boston commented on SLING-9060: --- PRs https://github.com/apache/sling-org-apache-sling-api/pull/17 [https://github.com/apache/sling-org-apache-sling-servlets-get/pull/4] > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
[ https://issues.apache.org/jira/browse/SLING-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reassigned SLING-9060: - Assignee: Ian Boston > Improve Redirect handling for Resources that can be served from alternative > URLs > > > Key: SLING-9060 > URL: https://issues.apache.org/jira/browse/SLING-9060 > Project: Sling > Issue Type: Improvement > Components: API, Servlets >Affects Versions: API 2.22.0, Servlets Get 2.1.40 >Reporter: Ian Boston >Assignee: Ian Boston >Priority: Major > Fix For: Servlets Get 2.1.42, API 2.23.0 > > > Improve the mechanism used to redirect streamed binary requests so that it > has the context of the request and can be extended without provider API > changes. > > This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth > |https://markmail.org/message/msmxgulkmzx7czth] > PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SLING-9060) Improve Redirect handling for Resources that can be served from alternative URLs
Ian Boston created SLING-9060: - Summary: Improve Redirect handling for Resources that can be served from alternative URLs Key: SLING-9060 URL: https://issues.apache.org/jira/browse/SLING-9060 Project: Sling Issue Type: Improvement Components: API, Servlets Affects Versions: Servlets Get 2.1.40, API 2.22.0 Reporter: Ian Boston Fix For: API 2.23.0, Servlets Get 2.1.42 Improve the mechanism used to redirect streamed binary requests so that it has the context of the request and can be extended without provider API changes. This was discussed on list at [https://markmail.org/message/msmxgulkmzx7czth |https://markmail.org/message/msmxgulkmzx7czth] PR to follow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SLING-7510) UriProvider throws unchecked IllegalArgumentException that must be handled by consumers
[
https://issues.apache.org/jira/browse/SLING-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499929#comment-16499929
]
Ian Boston commented on SLING-7510:
---
The API was created being pragmatic rather than perfectionist.
It took that approach to minimise the number of new API classes it introduced.
This was done to satisfy community feedback from Sling and Oak. As such it made
compromises and tried to use java.* classes as far as possible.
The logic was as follows:
There are many views on the IllegalArgumentException and how it should be used.
eg [https://airbrake.io/blog/java/illegalargumentexception]. Most centre on
using the exception when the arguments are illegal from an application logic
point of view, but are not illegal from a JVM point of view (as is the case
with a NPE). The example given in that post was Books. A call to
Book.getPage(1492) where there are Book implementation has 1000 pages only,
should result in an IllegalArgumentException.
By the same argument, a call URIProvider.toURI(Resource resource,
URIProvider.Scope scope, URIProvider.Operation operation) where resource, scope
or operation and not legal for the implementation of the URIProvider should
throw an IllegalArgumentException.
It is unfortunate that Java decided to make an IllegalArgumentException
unchecked, and did not provide a checked version.
That was pragmatic logic for the API design at the time it was created.
The null check in the JcrNodeResource code is defensive implementation to catch
those bad developers that fail to add findbugs to their code and read the
JavaDoc. No one is perfect. Everyone makes mistakes.
--
In a perfectionists world all of the above is ugly.
A perfectionist might have create a URIHolder object with an isValid() method
and made the toURI call @Nonnull, but that would have added more API objects
and been against the community feedback in both Sling and Oak that surrounded
the introduction of this API.
An earlier version of the API did have more classes.
IIUC this API will be deprecated soon replaced by a completely new way of
handling Binaries that Alex is working on. Perhaps its better to close this
issue as wont fix and avoid investing more time in a API that will soon be
deprecated.
> UriProvider throws unchecked IllegalArgumentException that must be handled by
> consumers
> ---
>
> Key: SLING-7510
> URL: https://issues.apache.org/jira/browse/SLING-7510
> Project: Sling
> Issue Type: Improvement
> Components: API
>Affects Versions: API 2.16.4
>Reporter: Alexander Klimetschek
>Priority: Major
> Fix For: API 3.0.1
>
>
> h3. Status quo
> A consumer of the
> [UriProvider|https://github.com/apache/sling-org-apache-sling-api/blob/dfc41640031bc87ec271c648b22073e65f4f171a/src/main/java/org/apache/sling/api/resource/external/URIProvider.java#L45]
> currently is required to handle an unchecked {{IllegalArgumentException}},
> which is thrown when the provider is not able to handle the binary. Note that
> it is not supposed to ever return null per the javadoc. The
> [JcrNodeResource|https://github.com/apache/sling-org-apache-sling-jcr-resource/blob/0e2ebd0f1a5c7cb2044b2d754945eb0ee7641081/src/main/java/org/apache/sling/jcr/resource/internal/helper/jcr/JcrNodeResource.java#L233-L242]
> shows a typical consumer code (although it still does do a null check).
> For the use case of asking multiple providers and taking the first one that
> responds it's not an optimal pattern to rely on an unchecked exception for
> the expected failure case that one provider by design cannot handle a certain
> binary or request. Throwing an {{IllegalArgumentException}} if there is no
> problem with the argument passed from the client, but a limit or
> configuration setting of the provider, is misleading. Also, given there are
> multiple providers active, a client cannot know upfront which provider is the
> right one for a given binary and somehow prevent the "illegal argument" call
> in the first place.
> h3. Suggestion
> Often, {{null}} return values are used in such a case. The provider can log
> any possible useful information itself, on why it could not handle it, if
> needed. This would simplify the consumer code (no try/catch necessary) and
> remove unnecessary cost of exception handling for normal code paths.
> JcrNodeResource itself it uses a null return value to pass on the "could not
> retrieve anything" state to the upper layers.
> If the goal really is to use exceptions here, the API should add a
> {{@Nonnull}} annotation for the return value _and_ the expected failure
> exception should be a checked one such as a new {{UriProviderException}}.
> Then for any unexpected
[jira] [Commented] (SLING-7350) StreamRendererServlet redirects HEAD requests to external uri if available
[ https://issues.apache.org/jira/browse/SLING-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363637#comment-16363637 ] Ian Boston commented on SLING-7350: --- I have looked at this again and although the patch fixes the problem I am now not so certain. The http spec says GET and HEAD should be treated the same when deciding to perform a 302 redirect or not. see section 10.3.3 of page 62 of rfc2616. The reason for a HEAD is often for a browser to determine if something has changed. Changed from what ? Changed from the GET which was a redirect. So the GET after redirect had headers that defined the state of the final resource (etag, length, last modified). Those will have been provided by the service that served the content. Lets say that is AWS CloudFront. Only AWS CloudFront knows the headers it sent out. Sling has no knowledge of those headers so it cannot emit a valid HEAD response other than to redirect to CloudFront to provide that response. If Sling emits a HEAD response for HEAD and a redirect for GET, then the client will always interpret that as the GET is required and always perform the GET, defeating the purpose of the HEAD. If you have a different interpretation of more information please share it. If neither of us are certain about this, then we might need to ask a higher authority, perhaps even Roy Fielding or someone who really knows. - I appreciate this might break the client, but if it does, the client is probably already broken and has not implemented HEAD properly. I know some clients can't even follow redirects correctly, not modern browsers fortunately. Which client is having a problem with the HEAD redirect ? > StreamRendererServlet redirects HEAD requests to external uri if available > -- > > Key: SLING-7350 > URL: https://issues.apache.org/jira/browse/SLING-7350 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.30 >Reporter: Satya Deep Maheshwari >Priority: Major > Attachments: SLING-7350.diff > > > SLING-7140 added a feature to support redirects to external uri of a resource > if available. This handled by StreamRendererServlet. See [1]. As per the > current implementation, if there's a uri available, a redirect is done for a > http HEAD request as well. This can be problematic for clients which expect > the response as sent by the servlet as of now if the response is different > from the redirected HEAD request. > Would it be correct to handle the HEAD request *before* the redirect so that > its current behavior is retained? > [1] - > https://github.com/apache/sling-org-apache-sling-servlets-get/blob/8ba11a174996ca4b7237c54879d1db8f14796f4d/src/main/java/org/apache/sling/servlets/get/impl/helpers/StreamRendererServlet.java#L171-L173 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SLING-7350) StreamRendererServlet redirects HEAD requests to external uri if available
[ https://issues.apache.org/jira/browse/SLING-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309646#comment-16309646 ] Ian Boston commented on SLING-7350: --- Patch LGTM. Could you open a pull request please. > StreamRendererServlet redirects HEAD requests to external uri if available > -- > > Key: SLING-7350 > URL: https://issues.apache.org/jira/browse/SLING-7350 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.30 >Reporter: Satya Deep Maheshwari > Attachments: SLING-7350.diff > > > SLING-7140 added a feature to support redirects to external uri of a resource > if available. This handled by StreamRendererServlet. See [1]. As per the > current implementation, if there's a uri available, a redirect is done for a > http HEAD request as well. This can be problematic for clients which expect > the response as sent by the servlet as of now if the response is different > from the redirected HEAD request. > Would it be correct to handle the HEAD request *before* the redirect so that > its current behavior is retained? > [1] - > https://github.com/apache/sling-org-apache-sling-servlets-get/blob/8ba11a174996ca4b7237c54879d1db8f14796f4d/src/main/java/org/apache/sling/servlets/get/impl/helpers/StreamRendererServlet.java#L171-L173 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SLING-7156) SlingTldLocationsCache makes invalid assumptions about bundle entry URL's
[ https://issues.apache.org/jira/browse/SLING-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston closed SLING-7156. - Released in Scripting JSP 2.3.4 > SlingTldLocationsCache makes invalid assumptions about bundle entry URL's > - > > Key: SLING-7156 > URL: https://issues.apache.org/jira/browse/SLING-7156 > Project: Sling > Issue Type: Bug > Components: Scripting >Affects Versions: Scripting JSP 2.3.2 >Reporter: Karl Pauls >Assignee: Karl Pauls > Fix For: Scripting JSP 2.3.4 > > > The SlingTldLocationsCache tracks bundles and scans them for entries using > bundle.findEntries(). That is fine, however, it assumes that the returned > URLs will have a path that can be found again using > bundle.getEntry(url.getPath()). That is not guarantied to work and in this > case, unnecessary because it already has the url at this point to begin with. > The fix is simple, just use the URL it already has instead of getting the > path and using it to get the same url again using bundle.getEntry(). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston closed SLING-3317. - > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi >Assignee: Ian Boston > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-3317. --- Resolution: Fixed > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi >Assignee: Ian Boston > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270640#comment-16270640 ] Ian Boston commented on SLING-3317: --- Fixed by upgrading to 6.0.30, verified the fix in that version to be identical to the patch supplied and tested. Verified that the upgrade doesn't change any external dependencies as the jar that is changed is embedded. Since this is a sub minor version upgrade, no other components should have to change. Best to leave for a few days to see if this change causes any unexpected issues elsewhere which OSGi was not able to prevent. (ie leaked implementation details in Jasper). > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi >Assignee: Ian Boston > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reassigned SLING-3317: - Assignee: Ian Boston > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi >Assignee: Ian Boston > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266764#comment-16266764 ] Ian Boston commented on SLING-3317: --- [~takahito.kikuchi] has pointed out off list that bug is fixed in Tomcat 6.0.30 (verified), so the patch is not needed. > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (SLING-6609) Fix JSR305 annotations for ValueMap.get
[
https://issues.apache.org/jira/browse/SLING-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Boston reopened SLING-6609:
---
mvn clean javadoc:javadoc fails, due to this commit blocking releasing the
bundle.
Could you fix please.
The JavaDoc didn't make sense as there is no get(String) method.
{code}
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 13.165 s
[INFO] Finished at: 2017-11-02T14:32:34+00:00
[INFO] Final Memory: 27M/385M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:2.10.3:javadoc (default-cli) on
project org.apache.sling.api: An error has occurred in JavaDocs report
generation:
[ERROR] Exit code: 1 -
/Users/ieb/sling/sling-org-apache-sling-api/src/main/java/org/apache/sling/api/resource/ValueMap.java:77:
error: reference not found
[ERROR] * Therefore all implementations should internally call {@link
#get(String)} when the 2nd parameter
[ERROR] ^
[ERROR]
/Users/ieb/sling/sling-org-apache-sling-api/src/main/java/org/apache/sling/api/resource/ValueMap.java:90:
warning - Tag @link: can't find get(String) in
org.apache.sling.api.resource.ValueMap
[ERROR]
[ERROR] Command line was:
/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre/../bin/javadoc
@options @packages
[ERROR]
{code}
> Fix JSR305 annotations for ValueMap.get
> ---
>
> Key: SLING-6609
> URL: https://issues.apache.org/jira/browse/SLING-6609
> Project: Sling
> Issue Type: Bug
> Components: API
>Affects Versions: API 2.16.2
>Reporter: Konrad Windszus
>Assignee: Konrad Windszus
>Priority: Major
> Fix For: API 2.16.4
>
>
> Currently {{ T get(@Nonnull String name, T defaultValue);}} does neither
> define a JSR 305 annotation for the return value nor for the 2nd parameter.
> It makes sense to define them both as {{@Nonnull}}, because if you intend to
> get {{null}} as return value you are supposed to take the other get method
> ({{@CheckForNull T get(@Nonnull String name, @Nonnull Class type)}})
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-3317: -- Fix Version/s: Scripting JSP 2.3.4 > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218232#comment-16218232 ] Ian Boston commented on SLING-3317: --- Seeing this behaviour regularly with Oracle JDK 1.8.144 (was not previously seeing it with 1.8.131). Perhaps it should be fixed by either upgrading the Tomcat libraries or using the patch supplied ? If anyone objects please shout, otherwise I will get the patch applied. > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi > Fix For: Scripting JSP 2.3.4 > > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-7140. --- Resolution: Fixed Fix Version/s: (was: Resource Resolver 1.5.32) Sling API Change made as discussed. Change to ResourceResolver bundle reverted. Other bundles adjusted. I did not change the sling started sling.txt provisioning model since it seems that no one is adding SNAPSHOT dependencies into that bundle. When API 2.16.4, JCR Resource 3.0.6, Sevlets Get 2.1.28 is released these will need to be updated in a single operation since both Servlet Get and JCR Resource depend on API 2.16.4 (importing org.apache.sling.apa.resource.external version 1.0.0) > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214260#comment-16214260 ] Ian Boston commented on SLING-7140: --- Agreed. The version change in the ResourceResolver didn't feel right at the time. Strange that none of the IT tests I ran picked up the other issues. Probably because PaxExam doesn't test the current trunk unless to change it also. I put the interfaces in the resource api because I didnt want to create a new package, but if its Ok to create a new package, what should it be called ? A new package will be better also since I can already think of some use cases where URIProvder.Scope would need to be extended to allow CRUD. I think it should be under resource, since it's always directly related to a resource and could be used by any resource provider, not just JCR or the current Oak implementation of JCR. org.apache.sling.api.resource.convert org.apache.sling.api.resource.providers org.apache.sling.api.resource.external org.apache.sling.api.resource.ext org.apache.sling.api.resource.extension Whatever the package name is, tt would need to have the Interfaces in this commit https://github.com/apache/sling-org-apache-sling-api/commit/a228b98f3a58409507b98e0d5b2d8e52b755d94f > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4, > Resource Resolver 1.5.32 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213961#comment-16213961 ] Ian Boston commented on SLING-7140: --- FYI: here are all the commits related to this issue. https://github.com/search?q=org%3Aapache+SLING-7140=Commits > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4, > Resource Resolver 1.5.32 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213959#comment-16213959 ] Ian Boston commented on SLING-7140: --- Did you see this commit ? https://github.com/apache/sling-org-apache-sling-resourceresolver/commit/ec3844be2612f48cb3e599792bc05c36c36cc031 It updates the ResourceResolver so that it doesnt break. The patch passed a full build of Sling before it was applied, although due to the SVN-Git migration, it had to be done in multiple commits. I see the Jira only has 1 commit listed so I guess it hasnt been integrated with Git yet. The reason that it was not in the main commit was that the SVN->Git migration only moved that bundle and none of the others. I have no problem if the packages are moved to somewhere else in the Sling API bundle. I did open a RTC on sling-dev, but no one replied to it after 24h. > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4, > Resource Resolver 1.5.32 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212729#comment-16212729 ] Ian Boston commented on SLING-3317: --- Incorporating the patch into Servlets JSP 2.2.6 without and keeping the Tomcat version the same can be achieved by [1], not exactly clean as it unpacks the Jars and embeds the classes into the bundle. It also fixes a but in BeanELResolver fixed by Tomcat 7.0.42 (same issue) 1 https://github.com/ieb/sling/commit/93b3ce3092d43eee1331ffc05280dc130d6dfe0d > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-3317) Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, 100% CPU usage
[ https://issues.apache.org/jira/browse/SLING-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212461#comment-16212461 ] Ian Boston commented on SLING-3317: --- SLING-3984 is not relevant here as this stack trace occurs when evaluating EL and not when compiling a JSP. The same problem happens in Tomcat 6 and was fixed in Tomcat release of Tomcat 7.0.42 patched at [1] on . 4 Oct 2010, by putting a synchronised round the put and gets. Sling uses versions 6.0.14 1 https://github.com/apache/tomcat/commit/24494ec2d5fab4984828dc3439f32ca5515f98b8#diff-855b25d4571f8473ec18785862bcac67 > Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop, > 100% CPU usage > > > Key: SLING-3317 > URL: https://issues.apache.org/jira/browse/SLING-3317 > Project: Sling > Issue Type: Bug > Components: Scripting >Reporter: Takahito Kikuchi > > On a web application with Apache Sling Scripting JSP 2.0.24, 100% CPU usage > problem happened as following. It looks similar to the tomcat problem, > https://issues.apache.org/bugzilla/show_bug.cgi?id=50078 > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.get(WeakHashMap.java:470) > at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > >java.lang.Thread.State: RUNNABLE > at java.util.WeakHashMap.eq(WeakHashMap.java:343) > at java.util.WeakHashMap.put(WeakHashMap.java:521) > at java.util.WeakHashMap.putAll(WeakHashMap.java:640) > at org.apache.el.util.ConcurrentCache.put(ConcurrentCache.java:34) > at > org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:123) > at > org.apache.el.lang.ExpressionBuilder.build(ExpressionBuilder.java:146) > at > org.apache.el.lang.ExpressionBuilder.createValueExpression(ExpressionBuilder.java:190) > at > org.apache.el.ExpressionFactoryImpl.createValueExpression(ExpressionFactoryImpl.java:68) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.parseExpression(ExpressionEvaluatorImpl.java:45) > at > org.apache.sling.scripting.jsp.jasper.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:55) > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-7140: -- Fix Version/s: Resource Resolver 1.5.32 API 2.16.4 JCR Resource 3.0.6 Servlets Get 2.1.28 > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4, > Resource Resolver 1.5.32 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-7140. --- Resolution: Fixed > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Get 2.1.28, JCR Resource 3.0.6, API 2.16.4, > Resource Resolver 1.5.32 > > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210903#comment-16210903 ] Ian Boston commented on SLING-7140: --- Applied in 2 commits due to the SVN to Git migration. https://github.com/apache/sling-org-apache-sling-resourceresolver/commit/ec3844be2612f48cb3e599792bc05c36c36cc031 and r1812621 > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209242#comment-16209242 ] Ian Boston commented on SLING-7140: --- New Pull request created that has no dependencies on any changes in Oak. https://github.com/apache/sling/pull/262 This might need an RTC as adds to the Sling API > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185731#comment-16185731 ] Ian Boston commented on SLING-7140: --- Configuration information https://gist.github.com/ieb/f9e044e4033b8f810238fe9a27eeaa78 > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185711#comment-16185711 ] Ian Boston commented on SLING-7140: --- Pull request: https://github.com/apache/sling/pull/252 This depends on Oak 1.6 Branch https://github.com/apache/jackrabbit-oak/pull/70 there is also Oak Trunk https://github.com/apache/jackrabbit-oak/pull/69 Documentation to follow in a Gist > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
[ https://issues.apache.org/jira/browse/SLING-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185668#comment-16185668 ] Ian Boston commented on SLING-7140: --- Update from testing, also reported in Oak Tested against Sling Trunk using a patch backported to Oak 1.6. The patches are https://github.com/apache/jackrabbit-oak/compare/1.6...ieb:OAK-6575-1.6-3?expand=1 and in Sling https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2?expand=1 There were 2 fixes to the Oak patch. Its not viable to put a 4096 bit PKCS8 encoded private key into a OSGi property. The Sling provisioning model truncates it and its hard to manage, so this has been changed to a file. The file can be relative, but must be resolvable from the working directory of the JVM. In addition a configurable minimum size limit has been added so that small files, can be streamed via the JVM if desired. Also added is a log line to allow audit of all external urls. This might want to be at debug level, but for the moment it is at info level. This changes need to be forward ported to Oak Trunk, although Sling doesn't currently use Oak 1.8-SNAPSHOT. > Support redirects to URLs provided by the underlying datastore. > --- > > Key: SLING-7140 > URL: https://issues.apache.org/jira/browse/SLING-7140 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.26, JCR Resource 3.0.4 >Reporter: Ian Boston > > Incorporate changes to allow OAK-6575 to work in Sling, > Patch available at > https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 > Cant me applied until OAK-6575 is released in a version suitable for use in > Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SLING-7140) Support redirects to URLs provided by the underlying datastore.
Ian Boston created SLING-7140: - Summary: Support redirects to URLs provided by the underlying datastore. Key: SLING-7140 URL: https://issues.apache.org/jira/browse/SLING-7140 Project: Sling Issue Type: Bug Components: Servlets Affects Versions: JCR Resource 3.0.4, Servlets Get 2.1.26 Reporter: Ian Boston Incorporate changes to allow OAK-6575 to work in Sling, Patch available at https://github.com/apache/sling/compare/trunk...ieb:OAK-6575-3-2 Cant me applied until OAK-6575 is released in a version suitable for use in Sling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SLING-7047) Update Dropwizard Metrics bundles to 3.2 to eliminate sun.misc dependency.
Ian Boston created SLING-7047: - Summary: Update Dropwizard Metrics bundles to 3.2 to eliminate sun.misc dependency. Key: SLING-7047 URL: https://issues.apache.org/jira/browse/SLING-7047 Project: Sling Issue Type: Bug Reporter: Ian Boston as per https://github.com/ieb/slingmetrics/issues/1 Not certain where the Dropwizard Metrics bundle is included, but it would be good to make it work for any JDK, not just the Sun/Oracle JDK. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SLING-6979) Support authorization of access to external content
[ https://issues.apache.org/jira/browse/SLING-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6979: -- Description: This issue is a PoC. It adds a capability to Sling so that Sling can issue authorizations on request to access external data APIs. It will have a SPI allowing concrete implementations, as there are many different possible scheme. For instance, when configured with AWS S3 implementations of those SPIs, on request it will issue signed policy authorizations that allow a client to perform the authorised operation on the AWS S3 REST API, for a specific key, for a specific time period. This would support the client performing a direct upload to S3 as detailed in [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The same pattern with a different authorization mechanisms would allow Sling to emit X-Accell-Redirect headers for a Proxy LB like nginX to stream directly from the storage. This effectively removes the task of streaming bytes through Sling from Sling ensuring that all request threads in Sling are short lived, not consuming survivour heap space. Long lived threads holding onto references from stack will cause those objects to land in survivor heap costing more to GC when the operation is complete, even if the transfer is streamed via the JVM. Implementation will use a servlet attached to a resourceType. The Resource with that resource type will contain the configuration information and SPI implementation reference, so that requests to that Resource generate authorizations of the appropriate form. The configuration should be capable of mapping entire subtrees of individual resources. How the Resource path maps to a storage path is an implementation detail to follow Sling best practice in this area. (ie RESTfull) The PoC will be done in a branch, and can be deleted if a complete failure. Branch is at https://github.com/ieb/sling/tree/SLING-6979 This bundle has a servlet that responds to GET requests with either a redirect or a body. The redirect can take one of 2 forms. A standard http redirect (eg 302) or a proxy redirect like X-Accell-Redirect. The body is a Siren json body (content type application/vnd.siren+json). The bundle exposes 2 SPI interfaces. https://github.com/ieb/sling/blob/SLING-6979/contrib/commons/offload-operations/src/main/java/org/apache/sling/offload/api/OffloadAuthorization.java which is a factory service SPI for https://github.com/ieb/sling/blob/SLING-6979/contrib/commons/offload-operations/src/main/java/org/apache/sling/offload/api/OffloadAuthorizationProcessor.java. The implementation of the OffloadAuthorizationProcessor defines the contents of the response. The Servlet https://github.com/ieb/sling/blob/SLING-6979/contrib/commons/offload-operations/src/main/java/org/apache/sling/offload/impl/OfloadAuthorizationServlet.java is mapped to nodes with a resource type of sling/offloadauthdeep which has child nodes, containing configuration information. If the current user can read a child node matching the offload operation requested, the name of the OffloadAuthorization service implementation stored on that child node is used to identify that service implementation which in turn is used to obtain an instance of an OffloadAuthorizationProcessor which in turn generates the authorization. The OffloadAuthorizationProcessor may perform other checks. In the bundle there are 2 sample implementations of a S3 Signed URL offload to be used with nginX and an example of a S3 signed policy that emits the necessary authorization for operations on an S3 bucket using the singed policy pattern. See the various classes linked above was: This issue is a PoC. It adds a capability to Sling so that Sling can issue authorizations on request to access external data APIs. It will have a SPI allowing concrete implementations, as there are many different possible scheme. For instance, when configured with AWS S3 implementations of those SPIs, on request it will issue signed policy authorizations that allow a client to perform the authorised operation on the AWS S3 REST API, for a specific key, for a specific time period. This would support the client performing a direct upload to S3 as detailed in [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The same pattern with a different authorization mechanisms would allow Sling to emit X-Accell-Redirect headers for a Proxy LB like nginX to stream directly from the storage. This effectively removes the task of streaming bytes through Sling from Sling ensuring that all request threads in Sling are short lived, not consuming survivour heap space. Long lived threads holding onto references from stack will cause those objects to land in survivor heap costing more to GC when the operation is complete, even if the transfer is streamed via the JVM.
[jira] [Updated] (SLING-6979) Support authorization of access to external content
[ https://issues.apache.org/jira/browse/SLING-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6979: -- Description: This issue is a PoC. It adds a capability to Sling so that Sling can issue authorizations on request to access external data APIs. It will have a SPI allowing concrete implementations, as there are many different possible scheme. For instance, when configured with AWS S3 implementations of those SPIs, on request it will issue signed policy authorizations that allow a client to perform the authorised operation on the AWS S3 REST API, for a specific key, for a specific time period. This would support the client performing a direct upload to S3 as detailed in [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The same pattern with a different authorization mechanisms would allow Sling to emit X-Accell-Redirect headers for a Proxy LB like nginX to stream directly from the storage. This effectively removes the task of streaming bytes through Sling from Sling ensuring that all request threads in Sling are short lived, not consuming survivour heap space. Long lived threads holding onto references from stack will cause those objects to land in survivor heap costing more to GC when the operation is complete, even if the transfer is streamed via the JVM. Implementation will use a servlet attached to a resourceType. The Resource with that resource type will contain the configuration information and SPI implementation reference, so that requests to that Resource generate authorizations of the appropriate form. The configuration should be capable of mapping entire subtrees of individual resources. How the Resource path maps to a storage path is an implementation detail to follow Sling best practice in this area. (ie RESTfull) The PoC will be done in a branch, and can be deleted if a complete failure. Branch is at was: This issue is a PoC. It adds a capability to Sling so that Sling can issue authorizations on request to access external data APIs. It will have a SPI allowing concrete implementations, as there are many different possible scheme. For instance, when configured with AWS S3 implementations of those SPIs, on request it will issue signed policy authorizations that allow a client to perform the authorised operation on the AWS S3 REST API, for a specific key, for a specific time period. This would support the client performing a direct upload to S3 as detailed in [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The same pattern with a different authorization mechanisms would allow Sling to emit X-Accell-Redirect headers for a Proxy LN like nginX to stream directly from the storage. This effectively removes the task of streaming bytes through Sling from Sling ensuring that all request threads in Sling are short lived, not consuming survivour heap space. Long lived threads holding onto references from stack will cause those objects to land in survivor heap costing more to GC when the operation is complete, even if the transfer is streamed via the JVM. Implementation will use a servlet attached to a resourceType. The Resource with that resource type will contain the configuration information and SPI implementation reference, so that requests to that Resource generate authorizations of the appropriate form. The configuration should be capable of mapping entire subtrees of individual resources. How the Resource path maps to a storage path is an implementation detail to follow Sling best practice in this area. (ie RESTfull) The PoC will be done in a branch, and can be deleted if a complete failure. > Support authorization of access to external content > --- > > Key: SLING-6979 > URL: https://issues.apache.org/jira/browse/SLING-6979 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is a PoC. It adds a capability to Sling so that Sling can issue > authorizations on request to access external data APIs. It will have a SPI > allowing concrete implementations, as there are many different possible > scheme. For instance, when configured with AWS S3 implementations of those > SPIs, on request it will issue signed policy authorizations that allow a > client to perform the authorised operation on the AWS S3 REST API, for a > specific key, for a specific time period. This would support the client > performing a direct upload to S3 as detailed in > [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The > same pattern with a different authorization mechanisms would allow Sling to > emit X-Accell-Redirect headers for a Proxy LB like nginX to stream directly > from the storage. This effectively removes the task of
[jira] [Comment Edited] (SLING-6979) Support authorization of access to external content
[ https://issues.apache.org/jira/browse/SLING-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064703#comment-16064703 ] Ian Boston edited comment on SLING-6979 at 6/27/17 11:55 AM: - Branch underway at https://github.com/ieb/sling/tree/SLING-6979 https://github.com/ieb/sling/tree/SLING-6979/contrib/commons/offload-operations was (Author: ianeboston): Branch underway at https://github.com/ieb/sling/tree/SLING-6979 > Support authorization of access to external content > --- > > Key: SLING-6979 > URL: https://issues.apache.org/jira/browse/SLING-6979 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is a PoC. It adds a capability to Sling so that Sling can issue > authorizations on request to access external data APIs. It will have a SPI > allowing concrete implementations, as there are many different possible > scheme. For instance, when configured with AWS S3 implementations of those > SPIs, on request it will issue signed policy authorizations that allow a > client to perform the authorised operation on the AWS S3 REST API, for a > specific key, for a specific time period. This would support the client > performing a direct upload to S3 as detailed in > [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The > same pattern with a different authorization mechanisms would allow Sling to > emit X-Accell-Redirect headers for a Proxy LN like nginX to stream directly > from the storage. This effectively removes the task of streaming bytes > through Sling from Sling ensuring that all request threads in Sling are short > lived, not consuming survivour heap space. Long lived threads holding onto > references from stack will cause those objects to land in survivor heap > costing more to GC when the operation is complete, even if the transfer is > streamed via the JVM. > Implementation will use a servlet attached to a resourceType. The Resource > with that resource type will contain the configuration information and SPI > implementation reference, so that requests to that Resource generate > authorizations of the appropriate form. The configuration should be capable > of mapping entire subtrees of individual resources. How the Resource path > maps to a storage path is an implementation detail to follow Sling best > practice in this area. (ie RESTfull) > The PoC will be done in a branch, and can be deleted if a complete failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-6979) Support authorization of access to external content
[ https://issues.apache.org/jira/browse/SLING-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064703#comment-16064703 ] Ian Boston commented on SLING-6979: --- Branch underway at https://github.com/ieb/sling/tree/SLING-6979 > Support authorization of access to external content > --- > > Key: SLING-6979 > URL: https://issues.apache.org/jira/browse/SLING-6979 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is a PoC. It adds a capability to Sling so that Sling can issue > authorizations on request to access external data APIs. It will have a SPI > allowing concrete implementations, as there are many different possible > scheme. For instance, when configured with AWS S3 implementations of those > SPIs, on request it will issue signed policy authorizations that allow a > client to perform the authorised operation on the AWS S3 REST API, for a > specific key, for a specific time period. This would support the client > performing a direct upload to S3 as detailed in > [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The > same pattern with a different authorization mechanisms would allow Sling to > emit X-Accell-Redirect headers for a Proxy LN like nginX to stream directly > from the storage. This effectively removes the task of streaming bytes > through Sling from Sling ensuring that all request threads in Sling are short > lived, not consuming survivour heap space. Long lived threads holding onto > references from stack will cause those objects to land in survivor heap > costing more to GC when the operation is complete, even if the transfer is > streamed via the JVM. > Implementation will use a servlet attached to a resourceType. The Resource > with that resource type will contain the configuration information and SPI > implementation reference, so that requests to that Resource generate > authorizations of the appropriate form. The configuration should be capable > of mapping entire subtrees of individual resources. How the Resource path > maps to a storage path is an implementation detail to follow Sling best > practice in this area. (ie RESTfull) > The PoC will be done in a branch, and can be deleted if a complete failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SLING-6979) Support authorization of access to external content
Ian Boston created SLING-6979: - Summary: Support authorization of access to external content Key: SLING-6979 URL: https://issues.apache.org/jira/browse/SLING-6979 Project: Sling Issue Type: New Feature Reporter: Ian Boston Assignee: Ian Boston This issue is a PoC. It adds a capability to Sling so that Sling can issue authorizations on request to access external data APIs. It will have a SPI allowing concrete implementations, as there are many different possible scheme. For instance, when configured with AWS S3 implementations of those SPIs, on request it will issue signed policy authorizations that allow a client to perform the authorised operation on the AWS S3 REST API, for a specific key, for a specific time period. This would support the client performing a direct upload to S3 as detailed in [http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html]. The same pattern with a different authorization mechanisms would allow Sling to emit X-Accell-Redirect headers for a Proxy LN like nginX to stream directly from the storage. This effectively removes the task of streaming bytes through Sling from Sling ensuring that all request threads in Sling are short lived, not consuming survivour heap space. Long lived threads holding onto references from stack will cause those objects to land in survivor heap costing more to GC when the operation is complete, even if the transfer is streamed via the JVM. Implementation will use a servlet attached to a resourceType. The Resource with that resource type will contain the configuration information and SPI implementation reference, so that requests to that Resource generate authorizations of the appropriate form. The configuration should be capable of mapping entire subtrees of individual resources. How the Resource path maps to a storage path is an implementation detail to follow Sling best practice in this area. (ie RESTfull) The PoC will be done in a branch, and can be deleted if a complete failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SLING-6745) kafka-based sling.event.api implementation
[
https://issues.apache.org/jira/browse/SLING-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948945#comment-15948945
]
Ian Boston commented on SLING-6745:
---
The aim of SLING-5645 was to create an API that was compatible with
sling.event.api, however it was discovered while doing that work and testing in
a distributed environment that some features of the sling.event.api were
created with a single instance in mind and therefore did not fit a distributed
model. Once of those features was the ability to query event via the
distributed API. Looking deeper into distributed event systems like RabbitMQ,
AMQ etc the ability to randomly query the contents of an event queue is
generally not supported. For that reason SLING-5645 dropped that feature and
focused on what was supported. The API that was created is 2 layered. The outer
layer has no dependencies on any other API standard. The Inner layer requires
the JMS API and uses both Topic and Queue concepts. Most messaging providers
have Java clients that support the JMS API so this was considered to be a safe
choice. For example, AWS SMS has a Java JMS Client library.
For providers that do support JMS, all that is needed is this class
{code}
@ProviderType
public interface ConnectionFactoryService {
/**
*
* Get a connection factory.
* @return the connection factory.
*/
ConnectionFactory getConnectionFactory();
}
{code}
see
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/jms/ConnectionFactoryService.java
for example ActiveMQ
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/amq/ActiveMQConnectionFactoryService.java
If Kafka does not have a JMS Client then the following SPI needs to be
implemented.
https://github.com/apache/sling/tree/trunk/contrib/commons/mom/api/src/main/java/org/apache/sling/mom
Provided Kafka has the concept of Topics and Quues then this is relatively
trivial as in
https://github.com/apache/sling/tree/trunk/contrib/commons/mom/jms/src/main/java/org/apache/sling/jms/impl
-
Obviously there is nothing to stop anyone reimplementing an alternative MoM API
in your own way as a competing implementation to address the current Sling API.
Although...
Wouldnt it be better to improve the MoM API and fix where it doesn't work ?
There are placeholders for the query operations.
https://github.com/apache/sling/blob/trunk/contrib/commons/mom/jobs/core/src/main/java/org/apache/sling/jobs/JobManager.java#L60
I had thought if these were required an independent service would subscribe to
topics to maintain a list of active jobs. There is a topic for job status.
> kafka-based sling.event.api implementation
> --
>
> Key: SLING-6745
> URL: https://issues.apache.org/jira/browse/SLING-6745
> Project: Sling
> Issue Type: New Feature
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>
> In job handling to scale for larger deployment it is essential to be able to
> execute a job outside of the local instance. This can be in another instance
> in the same cluster (ie when ontop of documentMk) or outside of the cluster
> (in AEM eg via
> [offloading|https://docs.adobe.com/docs/en/aem/6-2/deploy/configuring/offloading.html]).
> In both cases Sling Event (Resource) stores the job in the repository
> (/var/eventing/jobs).
> It could be useful to have another variant of job execution that is managed
> entirely outside of the repository. With SLING-5645 a new API was created to
> support messaging-based implementations that would fit in this category as
> they map a job to a (one or a few) message(s).
> This new SLING-5645 Job-API is not entirely compatible with sling.event.api
> though.
> This ticket is to track an effort to provide a messaging-based implementation
> for sling.event.api - namely for Kafka. The goal is to become a drop-in
> replacement of sling event without much need for change on the user side of
> the sling.event.api.
> This might not right away become part of sling, but might start off in the
> contrib section - to underscore that it's not something supported, at least
> as of yet.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-6046. --- Resolution: Fixed Patch applied. > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar >Assignee: Ian Boston > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811728#comment-15811728 ] Ian Boston commented on SLING-6046: --- Applied patch in r1777957 > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar >Assignee: Ian Boston > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reopened SLING-6046: --- Assignee: Ian Boston > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar >Assignee: Ian Boston > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804007#comment-15804007 ] Ian Boston commented on SLING-6046: --- LGTM. +1. Lets wait 24h for any objections. > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-6046. --- Resolution: Won't Fix Consensus appears to indicate that the patch is not acceptable due to spec compliance reasons. > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-2799) Elastic Search backend for Sling: GSoC2013
[ https://issues.apache.org/jira/browse/SLING-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-2799. --- Resolution: Done GSOC2013 long ago. > Elastic Search backend for Sling: GSoC2013 > -- > > Key: SLING-2799 > URL: https://issues.apache.org/jira/browse/SLING-2799 > Project: Sling > Issue Type: New Feature > Components: Samples >Reporter: Ian Boston > Labels: elasticsearch, gsoc, gsoc2013, java, osgi, sling > > This is a proposal for GSoC2013: create an Resource Provider that allows > resources stored in Elastic Search to be exposed as Sling Resources. > Resources[1] are the basic building blocks of Sling. > ResourceProviders[2] allow data sources to be added to the core > ResourceProvider within Sling allowing those data sources to provide > Resources at pre-determined locations in the resource tree. To put it in more > familiar terms, implementing and adding a Resource provider is like mounting > or mapping a network drive. > A more recent addition to the facilities available in Sling include updatable > ResourceProviders. > Elastic Search[3] is a elastically scalable search engine based on Lucene > that has NoSQL like storage capabilities. Although written in Java, and used > by many Java applications, it is used by a multitude of scripting communities > (Python, Ruby, Php) as it exposes a RESTfull Json interface. Its NoSQL like > capabilities are supported by the ability to index in real time over multiple > shards and replicas. Notable users of Elastic Search include Wordpress, > GitHub, FourSquare, Sony and many others. > Initially this will provide read only resource access, but if there is time > in the project will allow read write access to a Elastic Search cluster. > Advanced Java skills are required, some knowledge of OSGi, Sling, Elastic > Search will be valuable as will a detailed knowledge of HTTP and RESTfull > architectures. > The following pages give more information about GSoC @apache: > * http://www.google-melange.com/gsoc/homepage/google/gsoc2013 > * http://community.apache.org/gsoc.html > * http://s.apache.org/gsoc2013ideas > 1 http://sling.apache.org/site/resources.html > 2 > http://sling.apache.org/apidocs/sling6/org/apache/sling/api/resource/ResourceProvider.html > 3 http://www.elasticsearch.org/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-5435) Decouple processes that depend on cluster leader elections from the cluster leader elections.
[ https://issues.apache.org/jira/browse/SLING-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-5435. --- Resolution: Won't Fix afaik no longer relevant. > Decouple processes that depend on cluster leader elections from the cluster > leader elections. > - > > Key: SLING-5435 > URL: https://issues.apache.org/jira/browse/SLING-5435 > Project: Sling > Issue Type: Improvement > Components: General >Reporter: Ian Boston > > Currently there are many processes in Sling that must complete before a Sling > Discovery cluster leader election is declared complete. These processes > include things like transferring all Jobs from the old leader to the new > leader and waiting for the data to appear visible on the new leader. This > introduces an additional overhead to the leader election process which > introduces a higher than desirable timeout for elections and heartbeat. This > higher than desirable timeout precludes the use of more efficient election > and distributed consensus algorithms as implemented in Etcd, Zookeeper or > implementations of RAFT. > If the election could be declared complete leaving individual components to > manage their own post election operations (ie decoupling those processes from > the election), then faster election or alternative Discovery implementations > such as the one implemented on etcd could be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-2798) Apache Cassandra backend for Sling: GSoC2013 Project
[ https://issues.apache.org/jira/browse/SLING-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-2798. --- Resolution: Done GSOC2013 long passed. > Apache Cassandra backend for Sling: GSoC2013 Project > > > Key: SLING-2798 > URL: https://issues.apache.org/jira/browse/SLING-2798 > Project: Sling > Issue Type: Bug > Components: Samples >Reporter: Ian Boston > Labels: cassandra, gsoc, gsoc2013, java, osgi, sling > Attachments: Sling Cassandra backend Architecture.jpg, > cassandra-backend-for-sling.tar.gz > > > This is a proposal for GSoC2013: create an Resource Provider that allows > resources stored in Apache Cassandra to be exposed as Sling Resources. > Resources[1] are the basic building blocks of Sling. > ResourceProviders[2] allow data sources to be added to the core > ResourceProvider within Sling allowing those data sources to provide > Resources at pre-determined locations in the resource tree. To put it in more > familiar terms, implementing and adding a Resource provider is like mounting > or mapping a network drive. > A more recent addition to the facilities available in Sling include updatable > ResourceProviders. > Apache Cassandra[3] is a column database (NoSQL) which aims to provide linear > scalability to web scale. It is used by many of the best known names on the > internet. > Initially this will provide read only resource access, but if there is time > in the project will allow read write access to a cassandra cluster. > Advanced Java skills are required, some knowledge of OSGi, Sling, Cassandra > will be valuable. > The following pages give more information about GSoC @apache: > * http://www.google-melange.com/gsoc/homepage/google/gsoc2013 > * http://community.apache.org/gsoc.html > * http://s.apache.org/gsoc2013ideas > 1 http://sling.apache.org/site/resources.html > 2 > http://sling.apache.org/apidocs/sling6/org/apache/sling/api/resource/ResourceProvider.html > 3 http://cassandra.apache.org/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-2919) Support handlebars as a scripting language
[ https://issues.apache.org/jira/browse/SLING-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-2919. --- Resolution: Won't Fix While it might be nice, there is limited need for this given the dominance of JSP and Slightly. > Support handlebars as a scripting language > -- > > Key: SLING-2919 > URL: https://issues.apache.org/jira/browse/SLING-2919 > Project: Sling > Issue Type: New Feature > Components: Scripting >Reporter: Ian Boston >Assignee: Ian Boston > > Handlebars.js is a popular scripting language used client side. The markup is > simple and straightforward, much like velocity in some ways, but without any > of the confusion ove template or MVC. There is a version of Handlebars for > java that uses an identical template language and appears to be well > supported and has some nice features such as pre-compiling scripts serverside > into Javascript functions and modules. (not all that relevant for server side > scripting, but interesting none the less). > I am going to experiment with a Sling Script engine in the whiteboard area to > see if its viable to use handlebars java as an alternative to JSP. > Handlebars Java, A2 Liencesed: https://github.com/jknack/handlebars.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (SLING-5646) Provide Message oriented Middleware API to support implementation of distributed Jobs API
[ https://issues.apache.org/jira/browse/SLING-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston closed SLING-5646. - Bundles have been released. > Provide Message oriented Middleware API to support implementation of > distributed Jobs API > - > > Key: SLING-5646 > URL: https://issues.apache.org/jira/browse/SLING-5646 > Project: Sling > Issue Type: Sub-task >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0 > > > See PoC at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs/mom-api > Needs to support Pub/Sub style messaging and Queue messaging. > Must not depend on Jobs API and must not bind to a specific technology. > Implementations are expected to be JMS or AMPQ based with an initial > implementation based on ActiveMQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (SLING-5647) Provide ActiveMQ implementation of the MoM API in SLING-5646
[ https://issues.apache.org/jira/browse/SLING-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston closed SLING-5647. - Bundles have been released. > Provide ActiveMQ implementation of the MoM API in SLING-5646 > > > Key: SLING-5647 > URL: https://issues.apache.org/jira/browse/SLING-5647 > Project: Sling > Issue Type: Sub-task >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM JMS 1.0.0 > > > Provide a default implementation of the MoM API in SLING-5646 using ActiveMQ. > The bundle must create and start a running ActiveMQ server without additional > intervention and configuration. Ideally the bundle should work in a cluster > embedding an ActiveMQ broker into each Sling instance, and ideally the bundle > should be able to integrate with an existing ActiveMQ cluster. Based on AMQ > documentation all are possible via configuration and no code changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston closed SLING-5645. - Bundles have been released. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0, MoM Jobs 1.0.0 > > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-5646) Provide Message oriented Middleware API to support implementation of distributed Jobs API
[ https://issues.apache.org/jira/browse/SLING-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5646: -- Fix Version/s: MoM JMS 1.0.0 > Provide Message oriented Middleware API to support implementation of > distributed Jobs API > - > > Key: SLING-5646 > URL: https://issues.apache.org/jira/browse/SLING-5646 > Project: Sling > Issue Type: Sub-task >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0 > > > See PoC at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs/mom-api > Needs to support Pub/Sub style messaging and Queue messaging. > Must not depend on Jobs API and must not bind to a specific technology. > Implementations are expected to be JMS or AMPQ based with an initial > implementation based on ActiveMQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-5646) Provide Message oriented Middleware API to support implementation of distributed Jobs API
[ https://issues.apache.org/jira/browse/SLING-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5646: -- Fix Version/s: MoM API 1.0.0 > Provide Message oriented Middleware API to support implementation of > distributed Jobs API > - > > Key: SLING-5646 > URL: https://issues.apache.org/jira/browse/SLING-5646 > Project: Sling > Issue Type: Sub-task >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0 > > > See PoC at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs/mom-api > Needs to support Pub/Sub style messaging and Queue messaging. > Must not depend on Jobs API and must not bind to a specific technology. > Implementations are expected to be JMS or AMPQ based with an initial > implementation based on ActiveMQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-5647) Provide ActiveMQ implementation of the MoM API in SLING-5646
[ https://issues.apache.org/jira/browse/SLING-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5647: -- Fix Version/s: MoM JMS 1.0.0 > Provide ActiveMQ implementation of the MoM API in SLING-5646 > > > Key: SLING-5647 > URL: https://issues.apache.org/jira/browse/SLING-5647 > Project: Sling > Issue Type: Sub-task >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM JMS 1.0.0 > > > Provide a default implementation of the MoM API in SLING-5646 using ActiveMQ. > The bundle must create and start a running ActiveMQ server without additional > intervention and configuration. Ideally the bundle should work in a cluster > embedding an ActiveMQ broker into each Sling instance, and ideally the bundle > should be able to integrate with an existing ActiveMQ cluster. Based on AMQ > documentation all are possible via configuration and no code changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-5645. --- Resolution: Fixed Remaining issues resolved, will do a release and start a vote. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0, MoM Jobs 1.0.0 > > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5645: -- Fix Version/s: MoM Jobs 1.0.0 MoM JMS 1.0.0 MoM API 1.0.0 > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: MoM API 1.0.0, MoM JMS 1.0.0, MoM Jobs 1.0.0 > > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545566#comment-15545566 ] Ian Boston commented on SLING-5645: --- If there are no objections, I would like to leave the code in contrib, until there is real production usage and we need to support it. Its been in contrib for 6 months already, with no indication of anyone using it. No point in declaring something is production, if its not. I'll cut a release tomorrow if I hear nothing. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542771#comment-15542771 ] Ian Boston commented on SLING-5645: --- All changes that were requested have been made except moving the subtree out of contrib. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542667#comment-15542667 ] Ian Boston commented on SLING-5645: --- Ignore that, just looked its only used inside the JMS implementation which already embeds active MQ, so no issue embedding another library. The reason AMQ was embedded, is its big with lots of complex dependencies. Embedding it reduced deployment complexity. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542656#comment-15542656 ] Ian Boston commented on SLING-5645: --- IIUC Embedding is an OSGi anti pattern. Could you explain why it would be better to embed it ? > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-5645) Provide a Jobs API and implementation suitable for widely distributed job processing.
[ https://issues.apache.org/jira/browse/SLING-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15541688#comment-15541688 ] Ian Boston commented on SLING-5645: --- GSon is being used for performance reasons. Probably not a good idea to remote it. In small file tests it is reported as being 20x faster than simple json parsing libraries like the Sling json library and 72% faster than Jackson. In large file tests is slower than Jackson. The message payloads should all be small, hence the choice. There are probably other faster encoding methods that could be used in place of Json, but the implementation is intended to be consumed by any language that can consume The AMQ protocol, not limited to java. No problem with other changes. > Provide a Jobs API and implementation suitable for widely distributed job > processing. > - > > Key: SLING-5645 > URL: https://issues.apache.org/jira/browse/SLING-5645 > Project: Sling > Issue Type: New Feature >Reporter: Ian Boston >Assignee: Ian Boston > > This issue is to track work on a proof of concept to create a Jobs API and > implementation that will work in a distributed environment where the job > submitters and job consumers are not necessarily in the same JVM or in the > same Sling cluster. > Work is being done in a branch at > https://github.com/ieb/sling/tree/jobs_28/contrib/extensions/jobs > Since the implementation needs supporting APIs/Capabilities not already > present in Sling. There are some sub-tasks associated with this issue to > address those. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6046: -- Fix Version/s: Servlets Get 2.1.20 > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Fix For: Servlets Get 2.1.20 > > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492843#comment-15492843 ] Ian Boston commented on SLING-6046: --- a) thank you for the pointer to RFC 7233, although I can see some clarifications I can't see a significant rewrite of Range handling relative to RFC 2616 which iirc was the original http 1.1 spec. b) Won't MS say its the servers problem to deal with broken streams ? imho thats true, and we need to do that as well. c) I agree, just looked at the definition of 206 which states "The request MUST have included a Range header field ..." so would probably not work. Based on that Sling, the patch is Ok and we have to deal with the resource consumption caused by the broken stream. > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492604#comment-15492604 ] Ian Boston commented on SLING-6046: --- The second request was a full GET that was aborted, probably by closing the connection, when the Accept: bytes header was sent. Although this might be a spec compliant behaviour is is a poor way of dealing with the issue. IE11 should send the Range: header in the first instance. https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html section 14.5. states clients MAY send a Range header before the Accept: bytes header is send. Only when a Accept: none header is sent must they not send a Range header. Section 14.35 of the specification allows for a Range: 0- header to be sent which allows the server to respond with the full body if it doesn't support Range headers, or the first n bytes if it does. This is the correct behaviour. The consensus is MS wont be fixing IE11 and if the same bug exists in in Edge they wont fix it there either as the spec can be interpreted either way. >From a Sling point of view the aborted request may cause problems, as, >depending on the Oak DataStore in use resources may have been allocated when >the request is aborted. If every large file request gets aborted first time >due to the Accept: bytes header being added, and those resources are allocated >through the http serving stream, then this patch as it stands may cause >problems for production deployments os Sling/AEM. Section 14.35 of the spec appears to allow a Partial content (206) response with a Content-Range header to any response the server deems fit, rather than responding with Accept: range. Proposal: Sling should look at the content length of a resource. If that content length is > a configured size it should send a 206 partial content with a Content-Range header to all responses, even those with no Range header in the request. Where there is a Range header in the request, it should respond as that header instructs according to the specification. wdyt ? Can you do a patch to cover that ? This will avoid the first request being aborted, assuming IE11 is spec compliant to 206 responses that it did not explicitly request. You might have to experiment with the size of the first response as IE11 appears to be asking for 100K at a time which seems low. > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Attachments: Accept-Range Respone Header from S3.png, > NetworkDataS3VideoFromIE11.xml, S3video.html, StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[
https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490071#comment-15490071
]
Ian Boston commented on SLING-6046:
---
Could you share a the request and response for the very first request for the
Video in IE please ?
In Chrome it loads the html file, and the very first response which doesnt know
anything about the server is:
{code}
Accept:*/*
Accept-Encoding:identity;q=1, *;q=0
Accept-Language:en-US,en;q=0.8,pt;q=0.6
Cache-Control:no-cache
Connection:keep-alive
Host:tautils.s3.amazonaws.com
Pragma:no-cache
Range:bytes=0-
User-Agent:Mozilla/5.0 ...
{code}
Then the response is
{code}
Accept-Ranges:bytes
Age:1
Connection:keep-alive
Content-Length:44811898
Content-Range:bytes 0-44811897/44811898
Content-Type:video/mp4
Date:Wed, 14 Sep 2016 10:12:10 GMT
ETag:"48217968ec2776736bc806334f3846a0"
Last-Modified:Tue, 06 Sep 2016 15:04:00 GMT
Server:AmazonS3
{code}
I am interested in what IE does as it won't have a Accept: bytes header when it
makes its first and only request to the S3 server. Using the Chrome request
saves as curl and dropping the range header indicates the S3 server will
respond with the full response leaving IE in a catch 22 situation. If it does a
GET first, with no range header it will get the whole file. It might be able to
send a HEAD first to check or it could drop the connection early and resubmit
with a range header. Which does it do ?
> While Streaming Video to IE 11, StreamRendererServlet do not use Partial
> Content Response [code 206]
>
>
> Key: SLING-6046
> URL: https://issues.apache.org/jira/browse/SLING-6046
> Project: Sling
> Issue Type: Bug
> Components: Servlets
>Affects Versions: Servlets Get 2.1.18
>Reporter: Ashok Kumar
> Attachments: Accept-Range Respone Header from S3.png, S3video.html,
> StreamRendererServlet.java.patch
>
>
> Since IE 11 expects "Accept-Ranges" [0] response header to start making
> requests with Range header, so sling lack in streaming of video content for
> IE end users. We can add Accept-Ranges = bytes header to response , either
> selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or
> always.
> Without support of partial content response (206) for IE users, all large
> video files are being downloaded in single chunk and user need to wait for
> long to see video content playing.
> [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-6046) While Streaming Video to IE 11, StreamRendererServlet do not use Partial Content Response [code 206]
[ https://issues.apache.org/jira/browse/SLING-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486994#comment-15486994 ] Ian Boston commented on SLING-6046: --- Some questions: Is it just video/mp4 that matter or all large files ? Does IE use the response from other resources to determine if it can send a range header on the first request for the large file ? Patch looks fine, I am wondering if serving these files via the JVM is the right thing to do and if it would not be better to have proper X-sendfile support protected by appropriate AuthZ. IIRC Sling uses Jetty that will transfer files using http chunked transfer encoding [1], provided no content-length header is sent in the response and the output stream is flushed at appropriate moments. If the transfer encoding is chunked, how does the video play in IE11, I realise the transfer will start from the beginning and there wont be support for fast forward, but will the video start playing immediately ? 1 https://en.wikipedia.org/wiki/Chunked_transfer_encoding > While Streaming Video to IE 11, StreamRendererServlet do not use Partial > Content Response [code 206] > > > Key: SLING-6046 > URL: https://issues.apache.org/jira/browse/SLING-6046 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Get 2.1.18 >Reporter: Ashok Kumar > Attachments: StreamRendererServlet.java.patch > > > Since IE 11 expects "Accept-Ranges" [0] response header to start making > requests with Range header, so sling lack in streaming of video content for > IE end users. We can add Accept-Ranges = bytes header to response , either > selectively only for video/mp4 mimetype ( video tag on IE looks for mp4 ) or > always. > Without support of partial content response (206) for IE users, all large > video files are being downloaded in single chunk and user need to wait for > long to see video content playing. > [0] http://stackoverflow.com/questions/25654422/http-pseudo-streaming-in-ie11 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-6027. --- Resolution: Fixed Fixed in r1759789 Test scripts can be found in bundles/servlets/post/developer-tests/testFileUploads.sh and uptodate documentation of the protocol with differences from the published protocol discovered in the code at bundles/servlets/post/Protocols.md > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14 > > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470863#comment-15470863 ] Ian Boston commented on SLING-6027: --- Nothing can be done to eliminate the IO from copying chunks into the final result other than using streams directly wrapped into a containing stream. Discussion on potential improvements to Oak are being discussed on oak-dev at http://markmail.org/thread/lnzkwhcsqhvxzj2p. Implementing both approaches in https://github.com/ieb/sling/compare/trunk...ieb:SLING-6027 > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14 > > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6027: -- Fix Version/s: Servlets Post 2.3.14 > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14 > > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6017) Streaming Uploads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470506#comment-15470506 ] Ian Boston commented on SLING-6017: --- Thanks for testing. I am still working on SLING-6027 (support for the Sling Chunked Upload protocol) and think we should wait till that is fixed before doing any releases. Can you work with the Snapshots for the time being ? > Streaming Uploads detection of request parameters is wrong. > --- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14, Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reassigned SLING-6027: - Assignee: Ian Boston > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston >Assignee: Ian Boston > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-6017) Streaming Uploads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-6017. --- Resolution: Fixed Fixed in https://svn.apache.org/repos/asf/sling/trunk@1758986 Same curl tests mentioned earlier pass. > Streaming Uploads detection of request parameters is wrong. > --- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14, Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6017) Streaming Uploads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6017: -- Fix Version/s: Servlets Post 2.3.14 > Streaming Uploads detection of request parameters is wrong. > --- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14, Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (SLING-6017) Streaming Uploads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston reopened SLING-6017: --- Reopening. Will make the Engine impl of Part match the Servlet API and will adjust the Sling POST Servlet bundle to match the documentation. > Streaming Uploads detection of request parameters is wrong. > --- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6017) Streaming Uploads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458884#comment-15458884 ] Ian Boston commented on SLING-6017: --- The behaviour documented for Sling is in conflict with the behaviour in the Servlet API. Changing Part to match the Servlet API and keeping the Sling behaviour requires a patch to the POST Servlet implementation, which I will do. If you are implementing a replacement for the the Sling POST Servlet be certain to follow the documented behaviour for Sling. > Streaming Uploads detection of request parameters is wrong. > --- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455539#comment-15455539 ] Ian Boston edited comment on SLING-6027 at 9/1/16 2:31 PM: --- The current Chunked upload mechanism does not have a way of isolating different sessions performing different uploads. It assumes that only one client will be performing an upload to an asset at the same time, and returns a 500 error if any other client tries to start an upload. For any other client to recover from this it would have to abandon the current upload and start again which would fail the first client. Since this is the only recovery route, it will be used. Other systems including the AWS S3 Multipart upload specification referenced from the Sling specification use a session ID. Note also, that the S3 Multipart upload specification aggregates the parts on completion under S3, presumably at a low level without consuming Disk IO. was (Author: ianeboston): The current Chunked upload mechanism does not have a way of isolating different sessions performing different uploads. It assumes that only one client will be performing an upload to an asset at the same time. If 2 are performing uploads there is a significant risk that the result will be corrupted. Take 2 clients each performing a chunked upload to a folder /content/reports both with the file name 2014_results.pdf, each with a different version. Both will create chunks inder 2014_results.pdf and when one completes the chunks will be merged resulting in a corrupted file. Other systems establish an upload session id before starting the upload, or on the first body part so that body parts from different sessions are not mixed up. Subsequent upload operations from the same client use the same session id. > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455539#comment-15455539 ] Ian Boston commented on SLING-6027: --- The current Chunked upload mechanism does not have a way of isolating different sessions performing different uploads. It assumes that only one client will be performing an upload to an asset at the same time. If 2 are performing uploads there is a significant risk that the result will be corrupted. Take 2 clients each performing a chunked upload to a folder /content/reports both with the file name 2014_results.pdf, each with a different version. Both will create chunks inder 2014_results.pdf and when one completes the chunks will be merged resulting in a corrupted file. Other systems establish an upload session id before starting the upload, or on the first body part so that body parts from different sessions are not mixed up. Subsequent upload operations from the same client use the same session id. > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[
https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455510#comment-15455510
]
Ian Boston commented on SLING-6027:
---
Looking at how other applications address this problem, most do not rely on
parameters but use headers instead. This avoids a reliance on parameter order.
Most use the standard HTTP Header Content-Range which is defined in RFC 2616
section 14.16 as:
"The Content-Range entity-header is sent with a partial entity-body to specify
where in the full entity-body the partial body should be applied."
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
The Google Drive API supports partial and resumable uploads using the
Content-Range header as documented at
https://developers.google.com/drive/v3/web/manage-uploads
The advantage this would have over the current implementation is each part of a
multipart upload would be a complete definition of the Part of the Entity body.
The current Sling Specific protocol is:
{code}
POST /content/dam/folder HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Length"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
1000
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf@Offset"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
400
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf"; filename="catalog.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
$binary-data
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe--
{code}
Following RFC 2616 14.16 results in 1 Part defining the upload with no reliance
on order.
{code}
POST /content/dam/folder HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: localhost:4502
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe
Content-Disposition: form-data; name="catalog.pdf"; filename="catalog.pdf"
Content-Type: application/pdf
Content-Range: 400-799/1000
Content-Transfer-Encoding: binary
$binary-data
--CbZDcL_DxJIVQqSG1WkYaIoLWqT3FGYCVe--
{code}
A Content-Length header could also be added to the Part indicating the length
of the Part although this is not absolutely necessary. Content-Range can follow
any valid specification of range and length supported by the RFC.
This appears to be the approach adopted by the Google Drive API (and other
APIs), although it uses http status codes and the Range header to indicate what
was received rather than an html body.
> Support existing Chunked upload functionality in streaming mode.
>
>
> Key: SLING-6027
> URL: https://issues.apache.org/jira/browse/SLING-6027
> Project: Sling
> Issue Type: Bug
> Components: Servlets
>Affects Versions: Servlets Post 2.3.12
>Reporter: Ian Boston
>
> The non streaming uploads support a partial upload protocol implemented in
> request parameters that is known in Sling terms as "Chunked" upload and
> documented at
> https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support
> (not to be confused with Chunked Transfer encoding or the use of Http Range
> headers).
> Sling Chunked uploading sends a sequence of POSTs containing multiple parts
> of a file upload. When all the parts are uploaded a final request is sent
> that causes all the parts to be merged into a single file in the JCR. From a
> streaming point of view, each part can be streamed with the streaming
> implementation supported by SLING-5948. Some additional code will be required
> to set the file name appropriately and the struture.
> However, when the upload is completed, Sling must merge all the parts. To
> maintain the streaming nature of the upload, this must be achieved without
> incurring any local IO, otherwise the benefits of a streamed upload are lost.
> I am not certain how to achieve the merge given the limitations of the JCR
> API other than by transferring all the body parts via the local JVM. That
> won't incur local Disk IO but will multiply the overall IO requirement by 3x.
> If JCR/Oak had the functionality to concatenate Binaries it could do this
> more efficiently depending on the DS implementation. If JCR/Oak exposed an
> Seekable OutputStream the Application could avoid needing to save uploads to
> the JCR as individual files. If JCR/Oak allowed an update to a binary to
> start at a known location, again
[jira] [Commented] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
[ https://issues.apache.org/jira/browse/SLING-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455187#comment-15455187 ] Ian Boston commented on SLING-6027: --- To aggregate a stream create a wrapped InputStream that reads from each chunk in turn as the commit is processed. If the DS is an S3 DS (for example) this wont incur any local Disk IO, assuming the S3 DS doesn't incur local Disk IO. If the DS is a remove FS DS all the IO may be served via the OS Disk cache. Over a direct streamed upload there is 1 read per body part + 1 write for the final file in addition to the streaming for all the body parts. > Support existing Chunked upload functionality in streaming mode. > > > Key: SLING-6027 > URL: https://issues.apache.org/jira/browse/SLING-6027 > Project: Sling > Issue Type: Bug > Components: Servlets >Affects Versions: Servlets Post 2.3.12 >Reporter: Ian Boston > > The non streaming uploads support a partial upload protocol implemented in > request parameters that is known in Sling terms as "Chunked" upload and > documented at > https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support > (not to be confused with Chunked Transfer encoding or the use of Http Range > headers). > Sling Chunked uploading sends a sequence of POSTs containing multiple parts > of a file upload. When all the parts are uploaded a final request is sent > that causes all the parts to be merged into a single file in the JCR. From a > streaming point of view, each part can be streamed with the streaming > implementation supported by SLING-5948. Some additional code will be required > to set the file name appropriately and the struture. > However, when the upload is completed, Sling must merge all the parts. To > maintain the streaming nature of the upload, this must be achieved without > incurring any local IO, otherwise the benefits of a streamed upload are lost. > I am not certain how to achieve the merge given the limitations of the JCR > API other than by transferring all the body parts via the local JVM. That > won't incur local Disk IO but will multiply the overall IO requirement by 3x. > If JCR/Oak had the functionality to concatenate Binaries it could do this > more efficiently depending on the DS implementation. If JCR/Oak exposed an > Seekable OutputStream the Application could avoid needing to save uploads to > the JCR as individual files. If JCR/Oak allowed an update to a binary to > start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLING-6027) Support existing Chunked upload functionality in streaming mode.
Ian Boston created SLING-6027: - Summary: Support existing Chunked upload functionality in streaming mode. Key: SLING-6027 URL: https://issues.apache.org/jira/browse/SLING-6027 Project: Sling Issue Type: Bug Components: Servlets Affects Versions: Servlets Post 2.3.12 Reporter: Ian Boston The non streaming uploads support a partial upload protocol implemented in request parameters that is known in Sling terms as "Chunked" upload and documented at https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support (not to be confused with Chunked Transfer encoding or the use of Http Range headers). Sling Chunked uploading sends a sequence of POSTs containing multiple parts of a file upload. When all the parts are uploaded a final request is sent that causes all the parts to be merged into a single file in the JCR. From a streaming point of view, each part can be streamed with the streaming implementation supported by SLING-5948. Some additional code will be required to set the file name appropriately and the struture. However, when the upload is completed, Sling must merge all the parts. To maintain the streaming nature of the upload, this must be achieved without incurring any local IO, otherwise the benefits of a streamed upload are lost. I am not certain how to achieve the merge given the limitations of the JCR API other than by transferring all the body parts via the local JVM. That won't incur local Disk IO but will multiply the overall IO requirement by 3x. If JCR/Oak had the functionality to concatenate Binaries it could do this more efficiently depending on the DS implementation. If JCR/Oak exposed an Seekable OutputStream the Application could avoid needing to save uploads to the JCR as individual files. If JCR/Oak allowed an update to a binary to start at a known location, again this could be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6017: -- Component/s: (was: Servlets) > Streaming Uplads detection of request parameters is wrong. > -- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6017: -- Affects Version/s: (was: Servlets Post 2.3.12) > Streaming Uplads detection of request parameters is wrong. > -- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-6017: -- Fix Version/s: (was: Servlets Post 2.3.14) Engine 2.6.4 > Streaming Uplads detection of request parameters is wrong. > -- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Engine 2.6.4 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-6017. --- Resolution: Fixed Fix Version/s: Servlets Post 2.3.14 Fixed in https://svn.apache.org/repos/asf/sling/trunk@1758401 > Streaming Uplads detection of request parameters is wrong. > -- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14 > > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[
https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449194#comment-15449194
]
Ian Boston commented on SLING-6017:
---
Checked and verified with the following curl commands
{code}
# Standard upload to create /content/test correctly.
curl -v -H "X-uploadmode: stream" -F
P1060839.jpg=@/Users/ieb/Desktop/P1060839.jpg
http://admin:admin@localhost:8080/content/test
# Streamed upload with a key value. The key is ignored.
curl -v -H "Sling-uploadmode: stream" -F key1=value1 -F
P1060839.jpg=@/Users/ieb/Desktop/P1060839.jpg -F
PPP1060839.jpg=@/Users/ieb/Desktop/P1060839.jpg -F key2=admin2
http://admin:admin@localhost:8080/content/test
# Streamed upload with a 2 key values. A warning is logged as key2 is not
available when the uploads are streamed.
# 2 files are created * uses P1060839.jpg as per the Sling API and the second
is PPP1060839.jpg
curl -v -H "Sling-uploadmode: stream" -F key1=value1 -F
*=@/Users/ieb/Desktop/P1060839.jpg -F
PPP1060839.jpg=@/Users/ieb/Desktop/P1060839.jpg -F key2=admin2
http://admin:admin@localhost:8080/content/test
{code}
> Streaming Uplads detection of request parameters is wrong.
> --
>
> Key: SLING-6017
> URL: https://issues.apache.org/jira/browse/SLING-6017
> Project: Sling
> Issue Type: Bug
> Components: Engine, Servlets
>Affects Versions: Servlets Post 2.3.12, Engine 2.6.2
>Reporter: Ian Boston
>Assignee: Ian Boston
>
> The way in which a request field that is not a file upload is detected is
> wrong. Reported in SLING-5948.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[ https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449164#comment-15449164 ] Ian Boston commented on SLING-6017: --- Also the behaviour documented here https://sling.apache.org/documentation/bundles/manipulating-content-the-slingpostservlet-servlets-post.html#file-uploads is not followed. > Streaming Uplads detection of request parameters is wrong. > -- > > Key: SLING-6017 > URL: https://issues.apache.org/jira/browse/SLING-6017 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.6.2 >Reporter: Ian Boston >Assignee: Ian Boston > > The way in which a request field that is not a file upload is detected is > wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
[
https://issues.apache.org/jira/browse/SLING-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449035#comment-15449035
]
Ian Boston commented on SLING-6017:
---
Reported in SLING-5948
{code}
curl -o /dev/null -v -F key1=value1 -F file=@file.txt -w
%{time_connect}:%{time_starttransfer}:%{time_total}
http://admin:admin@localhost:8080/temp/file4?uploadmode=stream
{code}
Causes key1=value1 to be processed as a file upload since isFormField(part) [1]
returns false due to part.getSubmittedFile() being non null [2].
The JavaDoc [3] indicates that getName() should be used to get the file name of
submitted by the client, and not getFieldName(), although in some examples
getFieldName appears to have been used incorrectly. This may be because
Part.getName() is the Servlet API 3.1 equivalent of FileItem.getFiledName() and
Part.getSubmittedFileName() is the Servlet API 3.1 equivalent of
FileItem.getName(). see [4]
1
https://github.com/apache/sling/blob/trunk/bundles/servlets/post/src/main/java/org/apache/sling/servlets/post/impl/operations/StreamedUploadOperation.java#L95
2
https://github.com/apache/sling/blob/trunk/bundles/engine/src/main/java/org/apache/sling/engine/impl/parameters/RequestPartsIterator.java#L148
3
https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileItemStream.html
4 https://docs.oracle.com/javaee/7/api/javax/servlet/http/Part.html
> Streaming Uplads detection of request parameters is wrong.
> --
>
> Key: SLING-6017
> URL: https://issues.apache.org/jira/browse/SLING-6017
> Project: Sling
> Issue Type: Bug
> Components: Engine, Servlets
>Affects Versions: Servlets Post 2.3.12, Engine 2.6.2
>Reporter: Ian Boston
>Assignee: Ian Boston
>
> The way in which a request field that is not a file upload is detected is
> wrong. Reported in SLING-5948.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449000#comment-15449000 ] Ian Boston edited comment on SLING-5948 at 8/30/16 1:24 PM: The way in which the file name is detected my be wrong. It delegates to the Commons File Upload FileItemStream.getFieldName() as was indicated in various examples, however, the JavaDoc at https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileItemStream.html#getName-- indicates that the getName method should be used. I have opened an new issue SLING-6017 to fix this. We should not implement header parsing in the Sling code unless the code in FileUpload contains a bug. was (Author: ianeboston): The way in which the file name is detected my be wrong. It delegates to the Commons File Upload FileItemStream.getFieldName() as was indicated in various examples, however, the JavaDoc at https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileItemStream.html#getName-- indicates that the getName method should be used. I have opened an new issue SLING-6019 to fix this. We should not implement header parsing in the Sling code unless the code in FileUpload contains a bug. > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Labels: Sling-9-ReleaseNotes > Fix For: Servlets Post 2.3.14, Engine 2.6.0 > > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part >
[jira] [Commented] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449000#comment-15449000 ] Ian Boston commented on SLING-5948: --- The way in which the file name is detected my be wrong. It delegates to the Commons File Upload FileItemStream.getFieldName() as was indicated in various examples, however, the JavaDoc at https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileItemStream.html#getName-- indicates that the getName method should be used. I have opened an new issue SLING-6019 to fix this. We should not implement header parsing in the Sling code unless the code in FileUpload contains a bug. > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Labels: Sling-9-ReleaseNotes > Fix For: Servlets Post 2.3.14, Engine 2.6.0 > > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need
[jira] [Created] (SLING-6017) Streaming Uplads detection of request parameters is wrong.
Ian Boston created SLING-6017: - Summary: Streaming Uplads detection of request parameters is wrong. Key: SLING-6017 URL: https://issues.apache.org/jira/browse/SLING-6017 Project: Sling Issue Type: Bug Components: Engine, Servlets Affects Versions: Engine 2.6.2, Servlets Post 2.3.12 Reporter: Ian Boston Assignee: Ian Boston The way in which a request field that is not a file upload is detected is wrong. Reported in SLING-5948. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston resolved SLING-5948. --- Resolution: Fixed Fix Version/s: Engine 2.5.2 Servlets Post 2.3.14 Fixed in sequence ending https://svn.apache.org/repos/asf/sling/trunk@1756172 > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Fix For: Servlets Post 2.3.14, Engine 2.5.2 > > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need a new API to make this work. request.adaptTo(PartsIterator.class), which > PartsIterator extends Iterator. > * Supporting the full breadth of the Sling Operation protocol in the Sling > Upload Operation will require wide scale duplication of code from the > ModifyOperation implementation as the ModifyOperation expects RequestProperty > maps and wont work with a streamed part. > * Forces the Sling Post bundle to depend
[jira] [Commented] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418729#comment-15418729 ] Ian Boston commented on SLING-5948: --- JDK8 builds passed. JDK7 builds failed at launchpad, as it requires JDK8. Will commit. > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need a new API to make this work. request.adaptTo(PartsIterator.class), which > PartsIterator extends Iterator. > * Supporting the full breadth of the Sling Operation protocol in the Sling > Upload Operation will require wide scale duplication of code from the > ModifyOperation implementation as the ModifyOperation expects RequestProperty > maps and wont work with a streamed part. > * Forces the Sling Post bundle to depend on Servlet 3 to get the Part API, > requiring some patches to the existing test classes. > To support both
[jira] [Commented] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418641#comment-15418641 ] Ian Boston commented on SLING-5948: --- Patch in Git has been updated with comments so far and some cleanup. https://github.com/apache/sling/compare/trunk...ieb:SLING-5948-P2?expand=1 Doing a full build prior and final testing to committing. > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need a new API to make this work. request.adaptTo(PartsIterator.class), which > PartsIterator extends Iterator. > * Supporting the full breadth of the Sling Operation protocol in the Sling > Upload Operation will require wide scale duplication of code from the > ModifyOperation implementation as the ModifyOperation expects RequestProperty > maps and wont work with a streamed part. > * Forces the Sling Post bundle to
[jira] [Commented] (SLING-5948) Support Streaming uploads.
[
https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418449#comment-15418449
]
Ian Boston commented on SLING-5948:
---
Agreed, thank you for the rfc reference. Header is now {{Sling-uploadmode}},
patch update and git commit pending for this change.
> Support Streaming uploads.
> --
>
> Key: SLING-5948
> URL: https://issues.apache.org/jira/browse/SLING-5948
> Project: Sling
> Issue Type: Bug
> Components: Engine, Servlets
>Affects Versions: Servlets Post 2.3.12, Engine 2.5.0
>Reporter: Ian Boston
>Assignee: Ian Boston
> Attachments: SLING-5948-Proposal1-illustration.patch,
> SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch,
> TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png
>
>
> Currently multipart POST request made to sling use the commons file upload
> component that parses the request fully before processing. If uploads are
> small they are stored in byte[], over a configurable limit they are sent to
> disk. This creates additional IO overhead, increases heap usage and increases
> upload time.
> Having searched the SLing jira, and sling-dev I have failed to find an issue
> relating to this area, although it has been discussed in the past.
> I have 2 proposals.
> The SlingMain Servlet processes all requests, identifying the request type
> and parsing the request body. If the body is multipart the Commons File
> Upload library is used to process the request body in full when the
> SlingServletRequest is created or the first parameter is requested. To enable
> streaming of a request this behaviour needs to be modified. Unfortunately,
> processing a streamed request requires that the ultimate processor requests
> multipar parts in a the correct order to avoid non streaming, so a streaming
> behaviour will not be suitable for most POST requests and can only be used if
> the ultimate Servlet has been written to process a stream rather than a map
> of parameters.
> Both proposals need to identify requests that should be processed as a
> stream. This identification must happen in the headers or URI as any
> identification later than the headers may be too late. Something like a
> custom header (x-uploadmode: stream) or a query string (?uploadmode=stream)
> or possibly a selector (/path/to/target.stream) would work and each have
> advantages and disadvantages.
> h1. Proposal 1
> When a POST request is identified as multipart and streaming, create a
> LazyParameterMap that uses the Commons File Upload Streaming API
> (https://commons.apache.org/proper/commons-fileupload/streaming.html) to
> process the request on demand as parameters are requested. If parameters are
> requested out of sequence, do something sensible attempting to maintain
> streaming behaviour, but if the code really breaks streaming, throw an
> exception to alert servlet developer early.
> h2. Pros
> * Follows a similar pattern to currently using the Servlet API.
> h2. Cons
> * [] params will be hard to support when the [] is out of order, and almost
> impossible if the [] is an upload body.
> * May not work when a request is routed incorrectly as getParameter requests
> will be out of streaming sequence.
> h2. Proposal 2
> When a POST request is identified as multipart and streaming, create a
> NullParameterMap that returns null for all parameter get operations. In
> addition set a request Attribute containing a Iterator that allows
> access to the request stream in a similar way to the Commons File Upload
> Streaming API. Servlets that process uploads streams will use the
> Iterator object retrieved from the request. Part is the Servlet 3 Part
> https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html.
> IIUC This API is already used in the Sling Engine and exported by a bundle.
> h2. Pros
> * Won't get broken by existing getParameter calls, which all return null and
> do no harm to the stream.
> * Far simpler implementation as the Servlet implementation has to get the
> request data in streaming order.
> h2. Cons
> * Needs a custom Sling Upload Operation that understand how to process the
> Iterator
> * Can't use the adaptTo mechanism on the request, as
> request.adaptTo(Iterator.class) doesn't make sense being too generic. Would
> need a new API to make this work. request.adaptTo(PartsIterator.class), which
> PartsIterator extends Iterator.
> * Supporting the full breadth of the Sling Operation protocol in the Sling
> Upload Operation will require wide scale duplication of code from the
> ModifyOperation implementation as the ModifyOperation expects RequestProperty
> maps and wont work with a streamed part.
> * Forces the Sling Post bundle to depend on Servlet 3 to get the Part API,
> requiring some patches to the
[jira] [Commented] (SLING-5948) Support Streaming uploads.
[
https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417521#comment-15417521
]
Ian Boston commented on SLING-5948:
---
{{X-sling-uploadmode}} yes, no problem.
> Support Streaming uploads.
> --
>
> Key: SLING-5948
> URL: https://issues.apache.org/jira/browse/SLING-5948
> Project: Sling
> Issue Type: Bug
> Components: Engine, Servlets
>Affects Versions: Servlets Post 2.3.12, Engine 2.5.0
>Reporter: Ian Boston
>Assignee: Ian Boston
> Attachments: SLING-5948-Proposal1-illustration.patch,
> SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch,
> TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png
>
>
> Currently multipart POST request made to sling use the commons file upload
> component that parses the request fully before processing. If uploads are
> small they are stored in byte[], over a configurable limit they are sent to
> disk. This creates additional IO overhead, increases heap usage and increases
> upload time.
> Having searched the SLing jira, and sling-dev I have failed to find an issue
> relating to this area, although it has been discussed in the past.
> I have 2 proposals.
> The SlingMain Servlet processes all requests, identifying the request type
> and parsing the request body. If the body is multipart the Commons File
> Upload library is used to process the request body in full when the
> SlingServletRequest is created or the first parameter is requested. To enable
> streaming of a request this behaviour needs to be modified. Unfortunately,
> processing a streamed request requires that the ultimate processor requests
> multipar parts in a the correct order to avoid non streaming, so a streaming
> behaviour will not be suitable for most POST requests and can only be used if
> the ultimate Servlet has been written to process a stream rather than a map
> of parameters.
> Both proposals need to identify requests that should be processed as a
> stream. This identification must happen in the headers or URI as any
> identification later than the headers may be too late. Something like a
> custom header (x-uploadmode: stream) or a query string (?uploadmode=stream)
> or possibly a selector (/path/to/target.stream) would work and each have
> advantages and disadvantages.
> h1. Proposal 1
> When a POST request is identified as multipart and streaming, create a
> LazyParameterMap that uses the Commons File Upload Streaming API
> (https://commons.apache.org/proper/commons-fileupload/streaming.html) to
> process the request on demand as parameters are requested. If parameters are
> requested out of sequence, do something sensible attempting to maintain
> streaming behaviour, but if the code really breaks streaming, throw an
> exception to alert servlet developer early.
> h2. Pros
> * Follows a similar pattern to currently using the Servlet API.
> h2. Cons
> * [] params will be hard to support when the [] is out of order, and almost
> impossible if the [] is an upload body.
> * May not work when a request is routed incorrectly as getParameter requests
> will be out of streaming sequence.
> h2. Proposal 2
> When a POST request is identified as multipart and streaming, create a
> NullParameterMap that returns null for all parameter get operations. In
> addition set a request Attribute containing a Iterator that allows
> access to the request stream in a similar way to the Commons File Upload
> Streaming API. Servlets that process uploads streams will use the
> Iterator object retrieved from the request. Part is the Servlet 3 Part
> https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html.
> IIUC This API is already used in the Sling Engine and exported by a bundle.
> h2. Pros
> * Won't get broken by existing getParameter calls, which all return null and
> do no harm to the stream.
> * Far simpler implementation as the Servlet implementation has to get the
> request data in streaming order.
> h2. Cons
> * Needs a custom Sling Upload Operation that understand how to process the
> Iterator
> * Can't use the adaptTo mechanism on the request, as
> request.adaptTo(Iterator.class) doesn't make sense being too generic. Would
> need a new API to make this work. request.adaptTo(PartsIterator.class), which
> PartsIterator extends Iterator.
> * Supporting the full breadth of the Sling Operation protocol in the Sling
> Upload Operation will require wide scale duplication of code from the
> ModifyOperation implementation as the ModifyOperation expects RequestProperty
> maps and wont work with a streamed part.
> * Forces the Sling Post bundle to depend on Servlet 3 to get the Part API,
> requiring some patches to the existing test classes.
> To support both methods a standard Servlet to handle streamed
[jira] [Updated] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5948: -- Attachment: TarMKDSNotStreamed.png TarMKStreamed.png TarMKDSStreamed.png Tested with a 6GB file for both TarMK and TarMK+FDS with a 5.6GB zip file. AEM running on a SSD. Source file on a External HDD over USB3. No network involved. Streamed upload into TarMK 4m1.138s elapsed. (23MB/s) Streamed upload into TarMK with FDS 2m44.614s (34MB/s) Non Streamed upload into TarMK with FDS 14m33.138s (6.56MB/s) Direct OS level copy (cp) between disks 2m15.823s (42MB/s) The streaming upload shows a 5x improvement in upload performance in this test. TarMK with FDS streams directly to a tmp file in the FS DS location which is moved into the DS when complete with an OS level move. CPU usage is greater perhaps due to the way Oak performs the stream transfer (ie not using native channels) TarMK without FDS must put blobs into the segment store, hence the greater cost in terms of time and GC activity. CPU is lower, perhaps because it takes longer. TarMK with DS non streaming shows 2x the IO overhead and higher CPU usage. Not certain why the high CPU usage. > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch, > TarMKDSNotStreamed.png, TarMKDSStreamed.png, TarMKStreamed.png > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken
[jira] [Updated] (SLING-5948) Support Streaming uploads.
[
https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Boston updated SLING-5948:
--
Attachment: SLING-5948-Proposal2v3.patch
Added unit test coverage and tested successfully in in Sling launchpad.
Apply the patch,
{code}
cd bundles/engine
mvn clean install
cd ../servlets/post
mvn clean install
cd ../../launchpad/builder
mvn clean install
java -jar target/org.apache.sling.launchpad-9-SNAPSHOT.jar &
tail -f sling/logs/error.log
{code}
Then Test normal upload
{code}
curl -v -F file=@P1060839.jpg http://admin:admin@localhost:8080/content/test
{code}
Then test streamed upload
{code}
curl -v -F P1060839.jpg=@P1060839.jpg
http://admin:admin@localhost:8080/content/test?uploadmode=stream
{code}
or
{code}
curl -v -H "X-uploadmode: stream" -F P1060839.jpg=@P1060839.jpg
http://admin:admin@localhost:8080/content/test
{code}
> Support Streaming uploads.
> --
>
> Key: SLING-5948
> URL: https://issues.apache.org/jira/browse/SLING-5948
> Project: Sling
> Issue Type: Bug
> Components: Engine, Servlets
>Affects Versions: Servlets Post 2.3.12, Engine 2.5.0
>Reporter: Ian Boston
>Assignee: Ian Boston
> Attachments: SLING-5948-Proposal1-illustration.patch,
> SLING-5948-Proposal2v2.patch, SLING-5948-Proposal2v3.patch
>
>
> Currently multipart POST request made to sling use the commons file upload
> component that parses the request fully before processing. If uploads are
> small they are stored in byte[], over a configurable limit they are sent to
> disk. This creates additional IO overhead, increases heap usage and increases
> upload time.
> Having searched the SLing jira, and sling-dev I have failed to find an issue
> relating to this area, although it has been discussed in the past.
> I have 2 proposals.
> The SlingMain Servlet processes all requests, identifying the request type
> and parsing the request body. If the body is multipart the Commons File
> Upload library is used to process the request body in full when the
> SlingServletRequest is created or the first parameter is requested. To enable
> streaming of a request this behaviour needs to be modified. Unfortunately,
> processing a streamed request requires that the ultimate processor requests
> multipar parts in a the correct order to avoid non streaming, so a streaming
> behaviour will not be suitable for most POST requests and can only be used if
> the ultimate Servlet has been written to process a stream rather than a map
> of parameters.
> Both proposals need to identify requests that should be processed as a
> stream. This identification must happen in the headers or URI as any
> identification later than the headers may be too late. Something like a
> custom header (x-uploadmode: stream) or a query string (?uploadmode=stream)
> or possibly a selector (/path/to/target.stream) would work and each have
> advantages and disadvantages.
> h1. Proposal 1
> When a POST request is identified as multipart and streaming, create a
> LazyParameterMap that uses the Commons File Upload Streaming API
> (https://commons.apache.org/proper/commons-fileupload/streaming.html) to
> process the request on demand as parameters are requested. If parameters are
> requested out of sequence, do something sensible attempting to maintain
> streaming behaviour, but if the code really breaks streaming, throw an
> exception to alert servlet developer early.
> h2. Pros
> * Follows a similar pattern to currently using the Servlet API.
> h2. Cons
> * [] params will be hard to support when the [] is out of order, and almost
> impossible if the [] is an upload body.
> * May not work when a request is routed incorrectly as getParameter requests
> will be out of streaming sequence.
> h2. Proposal 2
> When a POST request is identified as multipart and streaming, create a
> NullParameterMap that returns null for all parameter get operations. In
> addition set a request Attribute containing a Iterator that allows
> access to the request stream in a similar way to the Commons File Upload
> Streaming API. Servlets that process uploads streams will use the
> Iterator object retrieved from the request. Part is the Servlet 3 Part
> https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html.
> IIUC This API is already used in the Sling Engine and exported by a bundle.
> h2. Pros
> * Won't get broken by existing getParameter calls, which all return null and
> do no harm to the stream.
> * Far simpler implementation as the Servlet implementation has to get the
> request data in streaming order.
> h2. Cons
> * Needs a custom Sling Upload Operation that understand how to process the
> Iterator
> * Can't use the adaptTo mechanism on the request, as
> request.adaptTo(Iterator.class) doesn't make sense being too
[jira] [Updated] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5948: -- Affects Version/s: Servlets Post 2.3.12 > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need a new API to make this work. request.adaptTo(PartsIterator.class), which > PartsIterator extends Iterator. > * Supporting the full breadth of the Sling Operation protocol in the Sling > Upload Operation will require wide scale duplication of code from the > ModifyOperation implementation as the ModifyOperation expects RequestProperty > maps and wont work with a streamed part. > * Forces the Sling Post bundle to depend on Servlet 3 to get the Part API, > requiring some patches to the existing test classes. > To support both methods a standard Servlet to handle streamed uploads would > be needed, connecting the file request stream to the Resource output stream. > In some cases (Oak S3 DS Async Uploads,
[jira] [Updated] (SLING-5948) Support Streaming uploads.
[ https://issues.apache.org/jira/browse/SLING-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Boston updated SLING-5948: -- Component/s: Servlets > Support Streaming uploads. > -- > > Key: SLING-5948 > URL: https://issues.apache.org/jira/browse/SLING-5948 > Project: Sling > Issue Type: Bug > Components: Engine, Servlets >Affects Versions: Servlets Post 2.3.12, Engine 2.5.0 >Reporter: Ian Boston >Assignee: Ian Boston > Attachments: SLING-5948-Proposal1-illustration.patch, > SLING-5948-Proposal2v2.patch > > > Currently multipart POST request made to sling use the commons file upload > component that parses the request fully before processing. If uploads are > small they are stored in byte[], over a configurable limit they are sent to > disk. This creates additional IO overhead, increases heap usage and increases > upload time. > Having searched the SLing jira, and sling-dev I have failed to find an issue > relating to this area, although it has been discussed in the past. > I have 2 proposals. > The SlingMain Servlet processes all requests, identifying the request type > and parsing the request body. If the body is multipart the Commons File > Upload library is used to process the request body in full when the > SlingServletRequest is created or the first parameter is requested. To enable > streaming of a request this behaviour needs to be modified. Unfortunately, > processing a streamed request requires that the ultimate processor requests > multipar parts in a the correct order to avoid non streaming, so a streaming > behaviour will not be suitable for most POST requests and can only be used if > the ultimate Servlet has been written to process a stream rather than a map > of parameters. > Both proposals need to identify requests that should be processed as a > stream. This identification must happen in the headers or URI as any > identification later than the headers may be too late. Something like a > custom header (x-uploadmode: stream) or a query string (?uploadmode=stream) > or possibly a selector (/path/to/target.stream) would work and each have > advantages and disadvantages. > h1. Proposal 1 > When a POST request is identified as multipart and streaming, create a > LazyParameterMap that uses the Commons File Upload Streaming API > (https://commons.apache.org/proper/commons-fileupload/streaming.html) to > process the request on demand as parameters are requested. If parameters are > requested out of sequence, do something sensible attempting to maintain > streaming behaviour, but if the code really breaks streaming, throw an > exception to alert servlet developer early. > h2. Pros > * Follows a similar pattern to currently using the Servlet API. > h2. Cons > * [] params will be hard to support when the [] is out of order, and almost > impossible if the [] is an upload body. > * May not work when a request is routed incorrectly as getParameter requests > will be out of streaming sequence. > h2. Proposal 2 > When a POST request is identified as multipart and streaming, create a > NullParameterMap that returns null for all parameter get operations. In > addition set a request Attribute containing a Iterator that allows > access to the request stream in a similar way to the Commons File Upload > Streaming API. Servlets that process uploads streams will use the > Iterator object retrieved from the request. Part is the Servlet 3 Part > https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/Part.html. > IIUC This API is already used in the Sling Engine and exported by a bundle. > h2. Pros > * Won't get broken by existing getParameter calls, which all return null and > do no harm to the stream. > * Far simpler implementation as the Servlet implementation has to get the > request data in streaming order. > h2. Cons > * Needs a custom Sling Upload Operation that understand how to process the > Iterator > * Can't use the adaptTo mechanism on the request, as > request.adaptTo(Iterator.class) doesn't make sense being too generic. Would > need a new API to make this work. request.adaptTo(PartsIterator.class), which > PartsIterator extends Iterator. > * Supporting the full breadth of the Sling Operation protocol in the Sling > Upload Operation will require wide scale duplication of code from the > ModifyOperation implementation as the ModifyOperation expects RequestProperty > maps and wont work with a streamed part. > * Forces the Sling Post bundle to depend on Servlet 3 to get the Part API, > requiring some patches to the existing test classes. > To support both methods a standard Servlet to handle streamed uploads would > be needed, connecting the file request stream to the Resource output stream. > In some cases (Oak S3 DS Async Uploads, Mongo DS) this wont
