Done. The commit email should show up in a few minutes. @Imtiaz - I also committed (separately) changes to the two failing unit tests for the metadata issue on the metadata branch. You can run the unit tests by doing "mvn test" or "mvn clean test", the same way you would do "mvn jetty:run".
The two remaining problems as far as I can tell: 1. Response is HTML entity-encoded. We need to decode it before we send it back to the client, or make sure it is never encoded in the first place. 2. There is an extra <metadata></metadata> tag around the metadata (which is then entity-encoded). You can see this in your test results. I'm not sure if these problems are internal to the Message class or in the API code, but if you want to work on fixing them, please feel free. Ethan On Fri, Jul 23, 2010 at 11:20 AM, Anne Kathrine Petterøe <[email protected]> wrote: > Thank you for you work on ESME-242 Imtiaz! > Ethan, will you commit the changes to metadata branc so that we can test it? > > /Anne > > > On 23. juli 2010, at 11.03, Imtiaz Ahmed H E wrote: > >> Maybe I'm just cc'ing myself too !!! >> Sorry! >> >> ----- Original Message ----- From: "Imtiaz Ahmed H E" <[email protected]> >> To: <[email protected]> >> Sent: Friday, July 23, 2010 2:30 PM >> Subject: Re: Metadata handling (was "Release planning") >> >> >>> By the way, someone please tell me why I have started receiving my own >>> posts to the lists also since the last couple or so of posts. It's pretty >>> irritating :) >>> >>> Imtiaz >>> >>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <[email protected]> >>> To: <[email protected]> >>> Sent: Friday, July 23, 2010 2:24 PM >>> Subject: Re: Metadata handling (was "Release planning") >>> >>> >>>> I have attached a fix (Esme_branches_metadata_Jira242Fix.diff) to Jira >>>> ticket 242, with the following change >>>> >>>> The line, >>>> >>>> lazy val metadata: String = originalXml \ "metadata" text >>>> >>>> commented and replaced by the line >>>> >>>> lazy val metadata: String = (originalXml \ "metadata").toString >>>> >>>> See my earlier mail to esme-dev for results from the fix. >>>> >>>> The fix is only for the metadata branch in the SVN repo directory >>>> 'branches'. >>>> >>>> This is pending further evaluation by esme-dev and improvements etc... >>>> >>>> Imtiaz >>>> >>>> ----- Original Message ----- From: "Ethan Jewett" <[email protected]> >>>> To: <[email protected]> >>>> Sent: Wednesday, July 21, 2010 7:51 PM >>>> Subject: Re: Metadata handling (was "Release planning") >>>> >>>> >>>>> Sounds good to me, at least until we can figure out what the next >>>>> step/problem is on the metadata front. Just ping the list when you've >>>>> uploaded the patch. >>>>> >>>>> You don't happen to have any JDBC/JNDI and/or Lucene experience do >>>>> you? I'm totally stumped on the issue with search when we have it run >>>>> using the database instead of the filesystem (ESME-205). >>>>> >>>>> Ethan >>>>> >>>>> On Wed, Jul 21, 2010 at 4:06 PM, Imtiaz Ahmed H E <[email protected]> >>>>> wrote: >>>>>> Oops! I think I wrote that in a hurry, not even thinking!! >>>>>> >>>>>> Actually, I think I should just submit a patch to the metadata branch >>>>>> with >>>>>> my current change and move on to other Jira tickets while we wait for >>>>>> consensus/finalization. >>>>>> >>>>>> Imtiaz >>>>>> >>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" >>>>>> <[email protected]> >>>>>> To: <[email protected]> >>>>>> Sent: Wednesday, July 21, 2010 7:30 PM >>>>>> Subject: Re: Metadata handling (was "Release planning") >>>>>> >>>>>> >>>>>>> How about it if I just store the metadata as a literal string and return >>>>>>> it as such for now and submit a patch to the metadata branch with that >>>>>>> and >>>>>>> move on to another Jira ticket ? >>>>>>> >>>>>>> Then, when we all agree on something final for the output format and >>>>>>> representation of message metadata we can look at further work on >>>>>>> ESME-242. >>>>>>> >>>>>>> Imtiaz >>>>>>> >>>>>>> ----- Original Message ----- From: "Ethan Jewett" <[email protected]> >>>>>>> To: <[email protected]> >>>>>>> Sent: Wednesday, July 21, 2010 7:01 PM >>>>>>> Subject: Re: Metadata handling (was "Release planning") >>>>>>> >>>>>>> >>>>>>> Imtiaz, >>>>>>> >>>>>>> Excellent! That's good progress. >>>>>>> >>>>>>> Any idea what the API response is still getting HTML entity-encoded? >>>>>>> That's what all of those < and > things are. Does the API do >>>>>>> this for all of its responses or is it just the metadata ones? We >>>>>>> probably don't need to be entity-encoding since we're returning XML, >>>>>>> not HTML. >>>>>>> >>>>>>> Also, I just created a branch called "metadata". Can you work against >>>>>>> this branch and feel free to submit patches against that branch to the >>>>>>> Jira ticket. Vassil, Anne, or I can then apply the patches directly to >>>>>>> the branch without the level of review that would be needed for a >>>>>>> patch against trunk. This should help allow us to try out your changes >>>>>>> and come to consensus and understanding more quickly. >>>>>>> >>>>>>> Thanks! >>>>>>> Ethan >>>>>>> >>>>>>> On Wed, Jul 21, 2010 at 7:20 AM, Imtiaz Ahmed H E <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> By the way, all I did was, >>>>>>>> >>>>>>>> in Message.scala, >>>>>>>> replace, >>>>>>>> >>>>>>>> lazy val metadata: String = originalXml \ "metadata" text >>>>>>>> >>>>>>>> by... >>>>>>>> >>>>>>>> lazy val metadata: String = (originalXml \ "metadata").toString >>>>>>>> >>>>>>>> to get the results you see in my previous mail... >>>>>>>> >>>>>>>> Imtiaz >>>>>>>> >>>>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" >>>>>>>> <[email protected]> >>>>>>>> To: <[email protected]> >>>>>>>> Sent: Wednesday, July 21, 2010 10:34 AM >>>>>>>> Subject: Re: Metadata handling (was "Release planning") >>>>>>>> >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> In the absence of decisive consensus yet, I made a small change in >>>>>>>>> Message.scala as a first step in fixing ESME-242. >>>>>>>>> Here's curl output for the Jira ticket examples given there by >>>>>>>>> Ethan...There is maybe obvious unacceptable stuff here, but let me >>>>>>>>> know... >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ curl --dump-header headers -d >>>>>>>>> "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL" >>>>>>>>> http:/ >>>>>>>>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> >>>>>>>>> >>>>>>>>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh >>>>>>>>> ole_name>I A 2 H E</whole_name></user></session></api> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ cat headers >>>>>>>>> HTTP/1.1 200 OK >>>>>>>>> Server: Apache-Coyote/1.1 >>>>>>>>> Set-Cookie: JSESSIONID=C7B688BAF8E99B2A638EF432885E310E; >>>>>>>>> Path=/esme-server-apach >>>>>>>>> e-esme-1.0-RC1-incubating >>>>>>>>> Expires: Wed, 21 Jul 2010 04:56:51 UTC >>>>>>>>> Date: Wed, 21 Jul 2010 04:56:51 GMT >>>>>>>>> Pragma: no-cache >>>>>>>>> Cache-Control: no-cache; private; no-store >>>>>>>>> X-Lift-Version: 2.0-SNAPSHOT >>>>>>>>> Content-Type: text/xml;charset=utf-8 >>>>>>>>> Content-Length: 178 >>>>>>>>> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ curl -b headers -d >>>>>>>>> 'message=test200&metadata=<outer><meta><metameta>Hello</me >>>>>>>>> tameta></meta><onlymeta>Meta</onlymeta></outer>' >>>>>>>>> http://localhost:8080/esme-ser >>>>>>>>> ver-apache-esme-1.0-RC1-incubating/api2/user/messages >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> <api><message> >>>>>>>>> <id>28</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test200</body> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <metadata><metadata><outer><meta><metameta>Hello</m >>>>>>>>> >>>>>>>>> >>>>>>>>> etameta></meta><onlymeta>Meta</onlymeta></outer></ >>>>>>>>> metadata></metadata> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message></api> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ curl -b headers -d >>>>>>>>> 'message=test201&metadata=<anytag>"meta":[{"place":{"place >>>>>>>>> >>>>>>>>> >>>>>>>>> _type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Le >>>>>>>>> t+Me+Down"}}]</anytag>' >>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-i >>>>>>>>> ncubating/api2/user/messages >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> <api><message> >>>>>>>>> <id>29</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test201</body> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <metadata><metadata><anytag>&quot;meta&quot;:[{&quot;p >>>>>>>>> >>>>>>>>> >>>>>>>>> lace&quot;:{&quot;place_type&quot;:&quot;city&quot;,&quo >>>>>>>>> t;region&quot;:&quot;CA >>>>>>>>> &quot;}},{&quot;song&quot;:{&quo >>>>>>>>> >>>>>>>>> >>>>>>>>> t;artist&quot;:&quot;Prince&quot;,&quot;songtitle&quot;:& >>>>>>>>> ;quot;Never Let Me >>>>>>>>> Down&quot;}}]</anytag></metadata></metadata> >>>>>>>>> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message></api> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ curl -b headers -d >>>>>>>>> 'message=test202&metadata="meta":[{"place":{"place_type":" >>>>>>>>> >>>>>>>>> >>>>>>>>> city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Dow >>>>>>>>> n"}}]' >>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/us >>>>>>>>> er/messages >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> <api><message> >>>>>>>>> <id>30</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test202</body> >>>>>>>>> <metadata></metadata> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message></api> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ curl -b headers >>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubat >>>>>>>>> ing/api2/user/messages >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> <api><messages><message> >>>>>>>>> <id>30</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test202</body> >>>>>>>>> <metadata></metadata> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message><message> >>>>>>>>> <id>29</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test201</body> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <metadata><metadata><anytag>&quot;meta&quot;:[{&quot;p >>>>>>>>> >>>>>>>>> >>>>>>>>> lace&quot;:{&quot;place_type&quot;:&quot;city&quot;,&quo >>>>>>>>> t;region&quot;:&quot;CA >>>>>>>>> &quot;}},{&quot;song&quot;:{&quo >>>>>>>>> >>>>>>>>> >>>>>>>>> t;artist&quot;:&quot;Prince&quot;,&quot;songtitle&quot;:& >>>>>>>>> ;quot;Never Let Me >>>>>>>>> Down&quot;}}]</anytag></metadata></metadata> >>>>>>>>> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message><message> >>>>>>>>> <id>28</id> >>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date> >>>>>>>>> <source>api2</source> >>>>>>>>> <body>test200</body> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <metadata><metadata><outer><meta><metameta>Hello</m >>>>>>>>> >>>>>>>>> >>>>>>>>> etameta></meta><onlymeta>Meta</onlymeta></outer></ >>>>>>>>> metadata></metadata> >>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author> >>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation> >>>>>>>>> </message></messages></api> >>>>>>>>> >>>>>>>>> imt...@imtiaz-20100131 /cygdrive/d/temp >>>>>>>>> $ >>>>>>>>> >>>>>>>>> ----- Original Message ----- From: "Vassil Dichev" >>>>>>>>> <[email protected]> >>>>>>>>> To: <[email protected]> >>>>>>>>> Sent: Tuesday, July 20, 2010 11:27 AM >>>>>>>>> Subject: Re: Metadata handling (was "Release planning") >>>>>>>>> >>>>>>>>> >>>>>>>>>>> I'd prefer option 1 (separate attribute from text). Within this >>>>>>>>>>> separate attribute there is the question of how data is >>>>>>>>>>> stored/represented. I'm ok with either raw string or a tuple-based >>>>>>>>>>> structure like Twitter's. I kind of like the tuple (key-value) >>>>>>>>>>> approach. >>>>>>>>>> >>>>>>>>>> I'm not too interested in how the data is stored, because it's fairly >>>>>>>>>> trivial to implement either way. It's currently not yet clear to me >>>>>>>>>> what the requirements for the output format are. >>>>>>>>>> >>>>>>>>>>> What I'm insisting on and what I was saying we got wrong is that >>>>>>>>>>> what >>>>>>>>>>> goes in needs to be the same as what comes out. If it's tuple-based >>>>>>>>>>> and I send in a tuple, then I should get that tuple (key and value) >>>>>>>>>>> back out when I request the metadata for a message. Right now I >>>>>>>>>>> think >>>>>>>>>>> we only get a concatenated list of values from the metadata and >>>>>>>>>>> metaData methods and we're bound to an XML format. >>>>>>>>>> >>>>>>>>>> I don't get it. A tuple is an abstraction which might be expressed in >>>>>>>>>> a specific format. So what goes in is not what comes out depending on >>>>>>>>>> the format. Let me quote the specific example Twitter provides. >>>>>>>>>> >>>>>>>>>> This comes in: >>>>>>>>>> >>>>>>>>>> "annotations": >>>>>>>>>> [{"type":{"another_attribute":"value", "attribute":"value"}}] >>>>>>>>>> >>>>>>>>>> This comes out: >>>>>>>>>> >>>>>>>>>> <annotations type="array"> >>>>>>>>>> <annotation> >>>>>>>>>> <type>foo</type> >>>>>>>>>> <attributes> >>>>>>>>>> <attribute> >>>>>>>>>> <name>bar</name> >>>>>>>>>> <value>baz</value> >>>>>>>>>> </attribute> >>>>>>>>>> </attributes> >>>>>>>>>> </annotation> >>>>>>>>>> </annotations> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> As far as requiring a particular format, I think the internal format >>>>>>>>>>> should be either a raw string or a immutable hashmap with raw >>>>>>>>>>> strings >>>>>>>>>>> as keys and values. We can handle converting this to XML or JSON in >>>>>>>>>>> the API or view code. >>>>>>>>>> >>>>>>>>>> Again, it's not so interesting what's internally there, let's just >>>>>>>>>> treat it as a black box. What I want to know is, do we want to have >>>>>>>>>> for instance XML in a JSON reply returned: >>>>>>>>>> >>>>>>>>>> "annotations": >>>>>>>>>> [{"type":{"<attributes> <attribute> <name>bar</name> >>>>>>>>>> <value>baz</value> </attribute> </attributes>"}}] >>>>>>>>>> >>>>>>>>>> or, inversely, do we want JSON inside an XML reply? Something like: >>>>>>>>>> >>>>>>>>>> <annotations type="array"> >>>>>>>>>> <annotation> >>>>>>>>>> <type>foo</type> >>>>>>>>>> <attributes> >>>>>>>>>> {"another_attribute":"value", "attribute":"value"} >>>>>>>>>> </attributes> >>>>>>>>>> </annotation> >>>>>>>>>> </annotations> >>>>>>>>>> >>>>>>>>>> because the latter will need to be escaped. >>>>>>>>>> >>>>>>>>>> Sorry for being too dense, but in any case, we either have to escape >>>>>>>>>> the metadata or we have to transform the structure to XML/JSON when >>>>>>>>>> we >>>>>>>>>> return it back to the user. None of these is "what goes in needs to >>>>>>>>>> be >>>>>>>>>> the same as what comes out" >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>> >> > >
