Ethan, we should probably update the issue to properly reflect what you're saying- that the metadata shouldn't be encoded. Note that URL-encoding is not an issue, cannot be avoided and is required for the Twitter API (as in the API of twitter.com) as well.
I think this could be implemented reasonably easily, but handling would probably be different in XML and JSON representations. Let me look and I could recommend some way of doing it if someone wants to implement it. On Sat, Jul 17, 2010 at 9:08 PM, Ethan Jewett <[email protected]> wrote: > Oh .... I see how it works. Thanks for the example. I was never able > to understand this before. This functionality is not what we want, but > just for completeness, here is a summary of how passing in metadata > through the API currently works (basic XML example): > > Send in: > <outer><meta><metameta>Hello</metameta></meta><onlymeta>Meta</onlymeta></outer> > > Get out: HelloMeta > > Note: this is the text contents of the XML tags concatenated together > in "order", whatever "order" means for XML content. > > Here's a JSON example: > > Send in: > <anytag>"meta":[{"place":{"place_type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Down"}}]</anytag> > > Get out: > "meta":[{"place":{"place_type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Down"}}] > > Note: This is, again, the text contents of the XML tags. The output is > HTML entity encoded, as it was before, but there weren't any > characters in the previous example that would have been encoded. > > Second note: The input *must* be wrapped in valid XML tags or it is ignored. > > I believe that this is completely wrong. The API should simply take > the text assigned to the metadata parameter, store it as part of the > message, and return exactly the same string in the message response. > It should not care what is in the text and it should not modify it in > any way. > > I think that making this work is going to take some revamping of the > internal handling of metadata, moving any kind of parsing and/or HTML > entity encoding to the UI code and have the internal Message model > simply store the metadata without thinking about it so hard. This > could clearly result in breakage, so I'm not sure how best to proceed. > > We could just put code into the API to make this behave how we want by > URLencoding it wrapping it in a special XML tag before storing it as > part of the message, then un-entity encode it and then un-URLencode it > before sending it back to the client, but that is going to result in a > horrible mess stored in the actual message. I think it will be better > to change the way that the actual Message class handles metadata. > > Does anyone know what functionality of the web UI currently uses > metadata, so that we can test for breakage if we change how this > works? > > Imtiaz, if you want to work on making this behave in a sane way, that > would be fantastic. It's a big job, but we've got to start somewhere. > I think ESME-242 is a good umbrella issue for this and we can modify > the issue to better reflect the problem. > > Ethan > > On Sat, Jul 17, 2010 at 6:57 PM, Imtiaz Ahmed H E <[email protected]> wrote: >> For example, for the response of the following (GET) maybe we need to url >> decode the response before we (the api) respond with it ? (Reference: see >> the reponse to the POST of a message in my previous mail, forwarded below). >> >> curl -b headers >> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/user/messages >> >> Of course this may be a typical naive one from me...! But let me know. Dick >> had said maybe I look at 242 before Vassil made his point & this discussion >> took place... >> >> Imtiaz >> >> ----- Original Message ----- From: "Imtiaz Ahmed H E" <[email protected]> >> To: <[email protected]>; "Ethan Jewett" <[email protected]> >> Sent: Saturday, July 17, 2010 9:35 PM >> Subject: Re: Release planning >> >> >>> Well...here you are... >>> >>> imt...@imtiaz-20100131 /cygdrive/d/temp >>> $ curl --dump-header headers -d "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL" >>> http:/ >>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session >>> <?xml version="1.0" encoding="UTF-8"?> >>> >>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh >>> ole_name>I A 2 H E</whole_name></user></session></api> >>> >>> imt...@imtiaz-20100131 /cygdrive/d/temp >>> $ cat headers >>> HTTP/1.1 200 OK >>> Server: Apache-Coyote/1.1 >>> Set-Cookie: JSESSIONID=28DBA5607A4D94C9E6E2BD1E61492EAD; >>> Path=/esme-server-apach >>> e-esme-1.0-RC1-incubating >>> Expires: Sat, 17 Jul 2010 15:58:22 UTC >>> Date: Sat, 17 Jul 2010 15:58:22 GMT >>> Pragma: no-cache >>> Cache-Control: no-cache; private; no-store >>> X-Lift-Version: 2.0-SNAPSHOT >>> Content-Type: text/xml;charset=utf-8 >>> Content-Length: 178 >>> >>> >>> imt...@imtiaz-20100131 /cygdrive/d/temp >>> $ curl -b headers -d >>> 'message=test72&metadata=<metadata>"meta":[{"place":{"plac >>> >>> e_type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+L >>> et+Me+Down"}}]</metadata>' >>> http://localhost:8080/esme-server-apache-esme-1.0-RC >>> 1-incubating/api2/user/messages >>> <?xml version="1.0" encoding="UTF-8"?> >>> <api><message> >>> <id>10</id> >>> <date>Sat, 17 Jul 2010 15:58:44 UTC</date> >>> <source>api2</source> >>> <body>test72</body> >>> >>> >>> <metadata>"meta":[{"place":{"place_type":"c >>> ity","region":"CA >>> "}},{"song":{"artist&q >>> uot;:"Prince","songtitle":"Never Let Me >>> Down"}}]</ >>> metadata> >>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>> <tags></tags><replyto></replyto><conversation></conversation> >>> </message></api> >>> >>> imt...@imtiaz-20100131 /cygdrive/d/temp >>> $ >>> >>> Imtiaz >>> >>> ----- Original Message ----- From: "Ethan Jewett" <[email protected]> >>> To: <[email protected]>; <[email protected]> >>> Sent: Saturday, July 17, 2010 8:30 PM >>> Subject: Re: Release planning >>> >>> >>> Re ESME-242 - I think you (and Vassil) are right, but I haven't been >>> able to independently confirm and I haven't seen a working API call >>> attached to the Jira issue, so I'm not willing to close it yet. If we >>> had working metadata examples using properly escaped XML and JSON, I >>> would be very happy :-) >>> >>> Ethan >>> >>> On Sat, Jul 17, 2010 at 4:32 PM, <[email protected]> wrote: >>>> >>>> About ESME-242, >>>> Per Vassil Dichev's mail to this group that http POST text used in the >>>> shell curl command must not be quoted and must be url encoded or must be >>>> quoted using single quotes, this Jira issue should be resolved to 'not a >>>> bug', if I recall correctly, that is! >>>> Imtiaz >>>> >>>> Sent from BlackBerryŽ on Airtel >>>> >>>> -----Original Message----- >>>> From: Ethan Jewett <[email protected]> >>>> Date: Sat, 17 Jul 2010 16:11:44 >>>> To: <[email protected]> >>>> Reply-To: [email protected] >>>> Subject: Re: Release planning >>>> >>>> Thoughts on items remaining for this release: >>>> >>>> ESME-225 - "need unit test for GET api2/users/actions don't retrieve >>>> disabled actions" - I don't think this is required for release and >>>> should be moved to the backlog. >>>> >>>> ESME-242 - "The use of metadata in the api2 currently doesn't work" - >>>> I think this is important but also not required for release and should >>>> be moved to the backlog. >>>> >>>> I agree that ESME-205 and ESME-212 are major. ESME-212 at least is a >>>> blocker. >>>> >>>> >>>> >>>> Regarding changes to be made to API2: I've created JIRA issues for all >>>> the todos at the bottom of the file (except a couple that were already >>>> done). It was lazy of me to put a todo-list in the file - I should >>>> really avoid that and force myself to have everything in Jira. I will >>>> be checking in a version of the file without the todo-list shortly. >>>> >>>> >>>> Cheers, >>>> Ethan >>>> >>>> On Fri, Jul 16, 2010 at 12:40 PM, Richard Hirsch <[email protected]> >>>> wrote: >>>>> >>>>> I've been working on the JIRA items and we have 10 items left for this >>>>> release. >>>>> >>>>> The following items are linked to IE 7: >>>>> * ESME-207 >>>>> * ESME-239 >>>>> * ESME-208 >>>>> >>>>> Some of these items are CSS-related (for example, 207) but the other >>>>> ones are javascript / jquery related. >>>>> >>>>> I think the two main issues are: >>>>> >>>>> * ESME-205 Search is broken (This only happens when you use a JDBC >>>>> based compass indexes - first noticed on Stax but it also is present >>>>> locally) >>>>> * ESME-212 Messages from pools aren't hidden >>>>> >>>>> I'm going to be gone the next four weeks on vacation, so either >>>>> someone else can coordinate the release or we wait until I return to >>>>> continue with our release preparations. >>>>> >>>>> If anyone wants to work on other items until the release, here are >>>>> other items that I consider important: >>>>> >>>>> * ESME-108 - View my pools" functionality >>>>> * ESME-170 Pubsubhubbub support for Atom & RSS actions# >>>>> * ESME-214 Add container-based authentication >>>>> * ESME-213 Twitter API is out-of-date and API calls don't always >>>>> return the correct info >>>>> >>>>> -- other api2 related changes - see the ToDos at the bottom of the >>>>> API2.scala page . >>>>> >>>>> D. >>>>> >>>> >>> >> >> >
