This is a great addition, and it ensures that developers can control the caching behavior at all levels of the stack (makeRequest, getProxyUrl, and now rewritten content).
On Wed, Jul 16, 2008 at 1:43 PM, Louis Ryan <[EMAIL PROTECTED]> wrote: > Hi, > > > Now that we've had some experience with the rewriter I'd like to propose an > additional feature to allow developers to control the caching behavior of > the open-proxy through which rewritten content is directed. The new param > 'expires' can have a value of either HTTP or a positive integer that > represents the TTL of the content in the browser cache and in any cache the > open-proxy may use internally. If the value is HTTP then the cache-headers > on the original content are respected by the cache used by the proxy and > carried through to the browser. > > This gives gadget developers finer-grained control over their contents TTL > while still getting the latency benefit of the cache. It is also useful for > developers that host content at sites where they dont have explicit control > of the cache-headers. Containers are still free to enforce a minimum TTL if > they so chose and should document it for developers. > > <Optional feature="content-rewrite"> > <Param name="expires">86400</Param> > <Param name="include-tags">script,link,img,embed</Param> > <Param name="include-pattern">.*</Param> > <Param name="exclude-pattern"></Param> > </Optional> > > Thoughts? > > -Louis > > > On Wed, Jun 4, 2008 at 4:18 AM, Louis Ryan <[EMAIL PROTECTED]> wrote: > >> Yes, for this kind of security you should just use a regex to exclude the >> swf from the proxy. Many apps use swf's for simple animations and other non >> security sensitive displays for which proxying would be perfectly fine. If >> you want to use a secure & trusted communication channel you may want to use >> the flash to js bridge to call signed makeRequest or do the equivalent >> yourself in flash. >> >> >> On Tue, Jun 3, 2008 at 4:08 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: >> >>> I think the best option is probably to just opt out for the rewriting of >>> the swf files. Automatic rewriting flash isn't very practical anyway due to >>> IE's limitations, so use of gadgets.flash.embedFlash would be the >>> recommended way to handle this. >>> >>> >>> On Mon, Jun 2, 2008 at 5:57 PM, Luke <[EMAIL PROTECTED]> wrote: >>> >>>> >>>> This relates more to implementation of the proxying server itself >>>> rather than the gadget spec... I would like to see that the proxying >>>> be useful for people who do flash development. The proxying, if not >>>> implemented with flash in mind, will be a security hole for flash >>>> developers. We use crossdomain.xml as a security mechanism to prevent >>>> unwanted access and including *.gmodules.com, *.msappspace.com, or >>>> *.hi5modules.com as an allowed domain in our crossdomain isn't secure >>>> if those domains proxy arbitrary urls. We may as well allow access >>>> from all domains. >>>> >>>> I see 2 ways we can get around the problem: >>>> >>>> 1) Create urls for proxied content using the original domain as part >>>> of the subdomain. Thus, something a url like >>>> http://static.myminilife.com/someswf.swf >>>> would be cached at static.myminilife.gmodules.com/... . Thus, we can >>>> allow the static.myminilife.gmodules.com specifically in the >>>> crossdomain file. The proxying server would have to ensure that the >>>> subdomain is the same as the url being proxied. >>>> >>>> 2) Do filtering of the request's http host against the proxied url >>>> being asked for. With that in place, then I can create a CNAME >>>> pointing one of our subdomains to the server doing the proxying. Ie, >>>> gmodules.myminilife.com points to gmodules.com. When I have a swf at >>>> static.myminilife.com/someswf.swf, I can replace that url with >>>> gmodules.myminilife.com, which will retrieve the content from the >>>> proxying server. Like above, The proxying server has to ensure that >>>> the request's http host is the same as the content being proxied. >>>> >>>> On May 28, 10:21 am, "Louis Ryan" <[EMAIL PROTECTED]> wrote: >>>> > Glad to hear it. Currently the implementation in Shindig is a work in >>>> > progress though it already provides useful functionality. In >>>> particular >>>> > because the configurable rewriting rules need to carry across a chain >>>> of >>>> > requests emanating from a gadget we need to carry the ability to >>>> lookup the >>>> > rule through that chain, this is something that is not yet >>>> implemented. Nor >>>> > for that matter is the ability to include or exclude URLs be regex >>>> though >>>> > that will be added soon. >>>> > >>>> > On Wed, May 28, 2008 at 1:12 AM, Kevin Brown <[EMAIL PROTECTED]> wrote: >>>> > > On Wed, May 28, 2008 at 12:39 AM, Paul Walker <[EMAIL PROTECTED]> >>>> wrote: >>>> > >>>> > >> Great news. Without caching rules, we are definitely prepared to >>>> > >> support this. >>>> > >>>> > >> However, speaking of Shindig and the process of proposals…since you >>>> have >>>> > >> contributed the code to Shindig, is it just expected to be approved >>>> as a >>>> > >> proposal without voting? >>>> > >>>> > > No -- Shindig will always implement whatever the standard specifies, >>>> so if >>>> > > the standard deviates from Louis' current proposal Shindig will be >>>> modified >>>> > > to support it. Shindig is a useful test bed for ironing out issues >>>> in a >>>> > > proposal before they are finalized, with enough users to easily test >>>> a wide >>>> > > range of container deployment needs. >>>> > >>>> > >> And on the specifics of what has been contributed to Shindig, >>>> what does >>>> > >> it actually do? >>>> > >>>> > > What Louis has contributed so far rewrites links as proposed on this >>>> > > thread. The content is examined for any links and those links are >>>> rewritten >>>> > > to go through a caching proxy. Several optimizations are done to try >>>> to >>>> > > reduce http overhead, such as merging contiguous http requests into >>>> one. >>>> > > We've deployed this to the Orkut sandbox as well to get some >>>> real-world >>>> > > feedback from developers. You can take a look at the code here: >>>> > > >>>> http://svn.apache.org/viewvc/incubator/shindig/trunk/java/gadgets/src. >>>> .. >>>> > >>>> > >> Is this a specific to your provider? Where does the code you >>>> contributed >>>> > >> actually persist the content and rewrite the URLs to? >>>> > >>>> > > Shindig's proxy provides a cache interface that users can implement >>>> to >>>> > > store the data however they like. A default in memory LRU cache is >>>> provided. >>>> > >>>> > >> Thanks, >>>> > >>>> > >> Paul >>>> > >>>> > >> *From:* [EMAIL PROTECTED] [mailto: >>>> > >> [EMAIL PROTECTED] *On Behalf Of *Louis >>>> Ryan >>>> > >> *Sent:* Thursday, May 22, 2008 2:11 PM >>>> > >> *To:* [EMAIL PROTECTED] >>>> > >> *Cc:* [EMAIL PROTECTED]; Bryan Green >>>> > >>>> > >> *Subject:* Re: PROPOSAL : Rewriting links in generated and proxied >>>> > >> content >>>> > >>>> > >> Paul, >>>> > >>>> > >> You certainly can use the include rule to specify the parameter >>>> matching >>>> > >> implementation and just document that for your container. The >>>> implementation >>>> > >> in shindig no longer cares about recognized file extensions but >>>> rather uses >>>> > >> the referencing HTML tag (script, link, img & embed currently) to >>>> determine >>>> > >> whether rewriting should occur so the revised proposal looks >>>> like.... >>>> > >>>> > >> <Optional feature="content-rewrite"> >>>> > >> <Param name="include-tags">script,link,img,embed</Param> >>>> > >> <Param name="include-pattern">.*</Param> >>>> > >>>> > >> <Param name="exclude-pattern"></Param> >>>> > >> </Optional> >>>> > >>>> > >> Params are shown with their default values, defaults are used if >>>> param is >>>> > >> omitted. >>>> > >>>> > >> Exclude pattern overrides include pattern if exclude pattern is >>>> defined. >>>> > >> The feature can be turned off by explicitly setting include-tags to >>>> an empty >>>> > >> list of exclude-pattern to '.*' >>>> > >>>> > >> I didnt mean to imply that the cache-control rules are defined by >>>> this >>>> > >> feature, thats an implementation detail that containers are free to >>>> change >>>> > >> at their discretion. I provided the discussion to give folks some >>>> idea what >>>> > >> a typical container might do. Completely agree that good developers >>>> will >>>> > >> version content and that containers are free to enforce this policy >>>> by >>>> > >> setting cache-control to eons on rewritten content. >>>> > >>>> > >> -Louis >>>> > >>>> > >> On Fri, May 16, 2008 at 7:10 PM, Paul Walker <[EMAIL PROTECTED]> >>>> wrote: >>>> > >>>> > >> Our thoughts at MySpace were to parse and persist external >>>> resources on >>>> > >> our end and the developer would simply opt in by supplying a >>>> standardized >>>> > >> query parameter on the URI of the resource. I think we can >>>> understand why >>>> > >> developers would want us to persist/cache some images/swfs for them >>>> and not >>>> > >> others. We need a way for users to opt in individual files and >>>> while the >>>> > >> pattern matching would work, the parameter just seems easier. I >>>> suppose for >>>> > >> users that create apps on our developer site, we could publish the >>>> use of >>>> > >> this parameter as a pattern match and still be compliant with the >>>> pattern >>>> > >> matching detail of this proposal. >>>> > >>>> > >> We'd like to obey caching headers, but would need to have a >>>> significant >>>> > >> floor. We will be storing the parsed resources on Akamai and would >>>> have to >>>> > >> run a job that updated these files and I don't like the messiness >>>> of that. >>>> > >> I don't see developers having a huge issue versioning URIs as is >>>> typically >>>> > >> done w/ JS libraries and update their gadget xml, so I'm not so >>>> concerned >>>> > >> about this. I'm more concerned about the 90%+ developers that >>>> don't >>>> > >> properly use cache control headers. Developers that are >>>> sophisticated >>>> > >> enough to and have the need to invalidate these resources >>>> periodically are >>>> > >> sophisticated enough to use URI versioning and update their gadget >>>> xml. >>>> > >> Again, they also have the option of hosting the resource themselves >>>> and >>>> > >> controlling any complicated/varying user-agent caching they have on >>>> the >>>> > >> resource. >>>> > >>>> > >> We pre-process all markup when it is saved/published and the live >>>> version >>>> > >> for installed users is retrieved already compressed to gzip (for >>>> users that >>>> > >> don't already have it cached on their user-agent), so we don't have >>>> the same >>>> > >> performance implications as shindig it sounds like. I assume that >>>> Shindig >>>> > >> could store a lookup of cached urls and check/replace where it >>>> finds matches >>>> > >> and where it does not, throw the work of fetching/caching to a >>>> background >>>> > >> task and not rewrite those until it has, but I'll leave that up to >>>> the >>>> > >> Shindig experts. >>>> > >>>> > >> Cc'ed Bryan Green, the main developer who has already finished most >>>> of our >>>> > >> implementation, if someone would like to discuss. We're keen on >>>> this >>>> > >> support so a general BIG +1 here. Could we separate the cache >>>> control >>>> > >> portion of this proposal and move forward with resource >>>> parsing/persisting >>>> > >> w/out it for now? >>>> > >>>> > >> Thx… >>>> > >>>> > >> ~Paul >>>> > >>>> > >> *From:* [EMAIL PROTECTED] [mailto: >>>> > >> [EMAIL PROTECTED] *On Behalf Of *Louis >>>> Ryan >>>> > >> *Sent:* Friday, May 16, 2008 4:47 PM >>>> > >> *To:* [EMAIL PROTECTED] >>>> > >> *Cc:* [EMAIL PROTECTED] >>>> > >> *Subject:* Re: PROPOSAL : Rewriting links in generated and proxied >>>> > >> content >>>> > >>>> > >> An initial implementation of this is now available as a patch for >>>> Shindig >>>> > >>>> > >>https://issues.apache.org/jira/browse/SHINDIG-276 >>>> > >>>> > >> On Wed, May 7, 2008 at 5:22 PM, Louis Ryan <[EMAIL PROTECTED]> >>>> wrote: >>>> > >>>> > >> 1. Yes >>>> > >> 2. Yes, My sample implementation is now doing this >>>> > >> 3. To some extent. Developers have a lot of control using the >>>> > >> cache-control headers. Containers are likely to require a minimum >>>> value >>>> > >>>> > >> On Wed, May 7, 2008 at 4:40 PM, Graham Spencer <[EMAIL PROTECTED]> >>>> wrote: >>>> > >>>> > >> Thanks Louis. A few minor questions: >>>> > >>>> > >> [1] I assume DEFAULT should also include CSS files. It might make >>>> sense to >>>> > >> base DEFAULT not on pattern matching targets but rather on the >>>> context (e.g. >>>> > >> <IMG>, CSS, etc.). >>>> > >>>> > >> [2] Will CSS files be parsed to extract background images and such? >>>> > >>>> > >> [3] Should we give developers control over timeouts? >>>> > >>>> > >> --g >>>> > >>>> > >> On Tue, May 6, 2008 at 4:48 PM, Louis Ryan <[EMAIL PROTECTED]> >>>> wrote: >>>> > >>>> > >> Hi, >>>> > >>>> > >> Many containers offer the ability to rewrite links to use their >>>> proxy >>>> > >> loading mechanism using *gadgets.io.getProxyUrl() *Typically the >>>> > >> generated proxy URL can be expected to give better download >>>> performance for >>>> > >> users of that particular container and also potentially reducing >>>> the load on >>>> > >> gadget developer backends. Currently however there is no standard >>>> way for a >>>> > >> gadget to have links in its generated markup be rewritten to use >>>> the proxy >>>> > >> URL format. I would like to propose creating a new feature to allow >>>> for >>>> > >> rewriting of links in generated gadget content >>>> > >>>> > >> Create a new standard gadget feature called *proxy-rewriter *which >>>> allows >>>> > >> gadgets to control whether they want content re-writing enabled. >>>> Containers >>>> > >> can choose to turn the feature on by default for all gadgets and >>>> gadgets can >>>> > >> use this mechanism to opt-out. >>>> > >>>> > >> An 'include' param is used to control which URLs to rewrite: >>>> > >>>> > >> - ALL - All URLs in the content >>>> > >> - DEFAULT - is recognized static file extension types such as >>>> .js, >>>> > >> .png, .gif ... >>>> > >> - NONE - disables the feature even if it is enabled by default >>>> by the >>>> > >> container >>>> > >>>> > >> An 'include-pattern' and 'exclude-pattern' can be specified to >>>> implement >>>> > >> more exact filtering rules. Patterns are applied to the URL to >>>> rewrite, >>>> > >> excludes are processed after includes >>>> > >>>> > >> An 'apply-to' is used to specify the comma separated list of >>>> mime-types >>>> > >> which the rewriter should recognize and rewrite. By default the >>>> list is >>>> > >> text/html,text/xml,application/xml (suggestions welcome here) >>>> > >>>> > >> <Optional feature="proxy-rewriter"> >>>> > >> <Param name="include">DEFAULT</Param> >>>> > >> <Param >>>> name="include-pattern">.*\/mystaticcontent\/.*</Param> >>>> > >> <Param >>>> name="exclude-pattern">.*\/mynonstaticcontent\/.*</Param> >>>> > >> <Param name="apply-to">text/html</Param> >>>> > >> </Optional> >>>> > >>>> > >> This feature will not only impact the content generated when the >>>> gadget is >>>> > >> rendered but is also used to control whether any content fetched >>>> through >>>> > >> makeRequest, Preload and proxied URLs is also rewritten. >>>> > >>>> > >> It is probably also >>>> > >>>> > ... >>>> > >>>> > read more » >>>> >>>> >>> >>> >>> >> > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google Groups > "OpenSocial and Gadgets Specification Discussion" group. > To post to this group, send email to > [EMAIL PROTECTED] > To unsubscribe from this group, send email to > [EMAIL PROTECTED] > For more options, visit this group at > http://groups.google.com/group/opensocial-and-gadgets-spec?hl=en > -~----------~----~----~----~------~----~------~--~--- > >
