Re: [openstack-dev] [Storlets] Swift copy middlleware
<kajina...@nttdata.co.jp> wrote on 16/05/2016 04:32:58 AM: > From: <kajina...@nttdata.co.jp> > To: <openstack-dev@lists.openstack.org> > Date: 16/05/2016 04:36 AM > Subject: Re: [openstack-dev] [Storlets] Swift copy middlleware > > > I agree with Kota, and I think this is the easiest way to fix the problem. > I noticed that placing storlets_middleware outside copy may require > some changes about slo handling, > because now it is assumed that storlets_middleware is placed > "inside" slo, right? > (I noticed this just after sending my previous e-mail :-( ) > > I think we had better make sure about what should be fixed in each > solutions, at this time. > Agree. Here is what I see at a high level: Option 1: Putting the storlet middleware at the left of the copy middleware: - Minimal code changes. We do need to take care of the utility Functions check_copy_from_header and check_destination_header I think are now private to the copy middleware. - Have some duplicate with the copy middleware, unless we can use it as "an API" Option 2: Leave the middleware where it is, and make sure we do not execute the storlet in both the get and the put. - Relies on internal perhaps not documented and not backwards Compatible way to determine that the GET/PUT are copy middleware generated. Personally, I prefer option 2, but need to better understand what We are relying on for knowing the GET/PUT are generated by the copy Middleware. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Storlets] Swift copy middlleware
> I agree with Kota, and I think this is the easiest way to fix the problem. I noticed that placing storlets_middleware outside copy may require some changes about slo handling, because now it is assumed that storlets_middleware is placed "inside" slo, right? (I noticed this just after sending my previous e-mail :-( ) I think we had better make sure about what should be fixed in each solutions, at this time. Thanks, Takashi -Original Message- From: kajina...@nttdata.co.jp [mailto:kajina...@nttdata.co.jp] Sent: Monday, May 16, 2016 10:20 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Storlets] Swift copy middlleware Hi Eran and Kota, > For temprary, that would work but I thought we could (*not sure*) fix the > issue just replace the order of pipeline, right? (i.e. storlets handler > should be the left of copy middleware) That is because the storlets middlware > have the logic to translate COPY/PUT(X-COPY-FROM) into GET(no storelets > running)/PUT(execute at Proxy). If it happens before the request reaching to > copy middleware, it looks like just PUT or GET at copy middleware itself (so > nothing to oparate there). I agree with Kota, and I think this is the easiest way to fix the problem. On the other hand, it's not efficient that Storlets has its original implementation about COPY inside it, which is very similar to copy middleware. As we discussed at Bristol, we had better make Storlets get rid of its original implementation and reuse ServerSideComyMiddleware or ServerSideCopyWebContext in copy middleware, to reduce that duplicated work, as the next step. # I need a deep dive about copy middleware patch, now. > I believe that for Storlets what would happen is that both PUT and GET > cause a storlet invocation, where in fact we want that invocation to > happen Eithrer in the GET or in the PUT (but not both) I believe that > if we are OK with running the storlet on the put, we can use The > swift_source SSC as an indicator that the get is generated from the > Copy middleware and disregard the X-Run-Storlet header. I also like this idea, if possible. Dealing with COPY only in the one point (in copy middleware) looks better, because it enables us to maintain the functionality more easily. Thanks, Takashi -Original Message- From: Kota TSUYUZAKI [mailto:tsuyuzaki.k...@lab.ntt.co.jp] Sent: Monday, May 16, 2016 9:25 AM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [Storlets] Swift copy middlleware Hey Eran, This is what I was concerning in Bristol Hackathon :/ > As a quick and temporary resolution I have changes the tox.ini > dependency to be 2.7.0 Instead of master. We still need, however, to > port the code accordingly, For temprary, that would work but I thought we could (*not sure*) fix the issue just replace the order of pipeline, right? (i.e. storlets handler should be the left of copy middleware) That is because the storlets middlware have the logic to translate COPY/PUT(X-COPY-FROM) into GET(no storelets running)/PUT(execute at Proxy). If it happens before the request reaching to copy middleware, it looks like just PUT or GET at copy middleware itself (so nothing to oparate there). I'll start to make sure my thought in this week but thanks to raise a flag to the community :) Thanks, Kota (2016/05/16 3:42), Eran Rom wrote: > Today the Swift team has merged copy middleware - congrats! > For us, however, it breaks the copy code path, which in fact can get > much simpler now. > > As a quick and temporary resolution I have changes the tox.ini > dependency to be 2.7.0 Instead of master. We still need, however, to > port the code accordingly, > > Here is a suggestion: > The copy middleware will process the COPY / PUT & X-Copy-From and will: > 1. Do a GET of the source object > 2. Do a PUT to the target object > > I believe that for Storlets what would happen is that both PUT and GET > cause a storlet invocation, where in fact we want that invocation to > happen Eithrer in the GET or in the PUT (but not both) I believe that > if we are OK with running the storlet on the put, we can use The > swift_source SSC as an indicator that the get is generated from the > Copy middleware and disregard the X-Run-Storlet header. > > Thoughts? > > Thanks, > Eran > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mai
Re: [openstack-dev] [Storlets] Swift copy middlleware
Hi Eran and Kota, > For temprary, that would work but I thought we could (*not sure*) fix the > issue just replace the order of pipeline, right? (i.e. storlets handler > should be the left of copy middleware) That is because the storlets middlware > have the logic to translate COPY/PUT(X-COPY-FROM) into GET(no storelets > running)/PUT(execute at Proxy). If it happens before the request reaching to > copy middleware, it looks like just PUT or GET at copy middleware itself (so > nothing to oparate there). I agree with Kota, and I think this is the easiest way to fix the problem. On the other hand, it's not efficient that Storlets has its original implementation about COPY inside it, which is very similar to copy middleware. As we discussed at Bristol, we had better make Storlets get rid of its original implementation and reuse ServerSideComyMiddleware or ServerSideCopyWebContext in copy middleware, to reduce that duplicated work, as the next step. # I need a deep dive about copy middleware patch, now. > I believe that for Storlets what would happen is that both PUT and GET > cause a storlet invocation, where in fact we want that invocation to > happen Eithrer in the GET or in the PUT (but not both) I believe that > if we are OK with running the storlet on the put, we can use The > swift_source SSC as an indicator that the get is generated from the > Copy middleware and disregard the X-Run-Storlet header. I also like this idea, if possible. Dealing with COPY only in the one point (in copy middleware) looks better, because it enables us to maintain the functionality more easily. Thanks, Takashi -Original Message- From: Kota TSUYUZAKI [mailto:tsuyuzaki.k...@lab.ntt.co.jp] Sent: Monday, May 16, 2016 9:25 AM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [Storlets] Swift copy middlleware Hey Eran, This is what I was concerning in Bristol Hackathon :/ > As a quick and temporary resolution I have changes the tox.ini > dependency to be 2.7.0 Instead of master. We still need, however, to > port the code accordingly, For temprary, that would work but I thought we could (*not sure*) fix the issue just replace the order of pipeline, right? (i.e. storlets handler should be the left of copy middleware) That is because the storlets middlware have the logic to translate COPY/PUT(X-COPY-FROM) into GET(no storelets running)/PUT(execute at Proxy). If it happens before the request reaching to copy middleware, it looks like just PUT or GET at copy middleware itself (so nothing to oparate there). I'll start to make sure my thought in this week but thanks to raise a flag to the community :) Thanks, Kota (2016/05/16 3:42), Eran Rom wrote: > Today the Swift team has merged copy middleware - congrats! > For us, however, it breaks the copy code path, which in fact can get > much simpler now. > > As a quick and temporary resolution I have changes the tox.ini > dependency to be 2.7.0 Instead of master. We still need, however, to > port the code accordingly, > > Here is a suggestion: > The copy middleware will process the COPY / PUT & X-Copy-From and will: > 1. Do a GET of the source object > 2. Do a PUT to the target object > > I believe that for Storlets what would happen is that both PUT and GET > cause a storlet invocation, where in fact we want that invocation to > happen Eithrer in the GET or in the PUT (but not both) I believe that > if we are OK with running the storlet on the put, we can use The > swift_source SSC as an indicator that the get is generated from the > Copy middleware and disregard the X-Run-Storlet header. > > Thoughts? > > Thanks, > Eran > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Storlets] Swift copy middlleware
Hey Eran, This is what I was concerning in Bristol Hackathon :/ > As a quick and temporary resolution I have changes the tox.ini dependency > to be 2.7.0 > Instead of master. We still need, however, to port the code accordingly, For temprary, that would work but I thought we could (*not sure*) fix the issue just replace the order of pipeline, right? (i.e. storlets handler should be the left of copy middleware) That is because the storlets middlware have the logic to translate COPY/PUT(X-COPY-FROM) into GET(no storelets running)/PUT(execute at Proxy). If it happens before the request reaching to copy middleware, it looks like just PUT or GET at copy middleware itself (so nothing to oparate there). I'll start to make sure my thought in this week but thanks to raise a flag to the community :) Thanks, Kota (2016/05/16 3:42), Eran Rom wrote: > Today the Swift team has merged copy middleware - congrats! > For us, however, it breaks the copy code path, which in fact can get much > simpler now. > > As a quick and temporary resolution I have changes the tox.ini dependency > to be 2.7.0 > Instead of master. We still need, however, to port the code accordingly, > > Here is a suggestion: > The copy middleware will process the COPY / PUT & X-Copy-From and will: > 1. Do a GET of the source object > 2. Do a PUT to the target object > > I believe that for Storlets what would happen is that both PUT and GET > cause a storlet invocation, where in fact we want that invocation to > happen > Eithrer in the GET or in the PUT (but not both) > I believe that if we are OK with running the storlet on the put, we can > use > The swift_source SSC as an indicator that the get is generated from the > Copy middleware and disregard the X-Run-Storlet header. > > Thoughts? > > Thanks, > Eran > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Storlets] Swift copy middlleware
Today the Swift team has merged copy middleware - congrats! For us, however, it breaks the copy code path, which in fact can get much simpler now. As a quick and temporary resolution I have changes the tox.ini dependency to be 2.7.0 Instead of master. We still need, however, to port the code accordingly, Here is a suggestion: The copy middleware will process the COPY / PUT & X-Copy-From and will: 1. Do a GET of the source object 2. Do a PUT to the target object I believe that for Storlets what would happen is that both PUT and GET cause a storlet invocation, where in fact we want that invocation to happen Eithrer in the GET or in the PUT (but not both) I believe that if we are OK with running the storlet on the put, we can use The swift_source SSC as an indicator that the get is generated from the Copy middleware and disregard the X-Run-Storlet header. Thoughts? Thanks, Eran __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
