Agreed. Created DAFFODIL-2077 to track that issue. - Steve
On 2/25/19 2:48 PM, Beckerle, Mike wrote: > This root attribute issue sounds like a bug also to me. > > > The root for a parser/unparserTestCase should be a QName. > > > For compatibility it can be assumed to be in example.com if unqualified, > though ideally that decision (what the default namespace is) ought to be left > to the TDML file author. > > ________________________________ > From: Steve Lawrence <[email protected]> > Sent: Monday, February 25, 2019 2:16:59 PM > To: [email protected]; Christofer Dutz > Subject: Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1 > > Ok, I've managed to track down the issue. The TDML runner was changed to > be a bit more strict about namespaces, and now wants to know the > namespace of the root element used for a parse. > > But the "root" attribute in a parserTestCase does not contain namespace > information (it probably should, but we used to be very lax about > namespaces, so many of our tests do not provide one). So in order to get > the namespace information, the TDML runner queries the namespace of the > root of the expected infoset. For positive test cases, it can do this > and determine the namespace should be "s7". But this information does > not exist for negative test cases. > > For negative tests that use an embedded schema (like you have via the > defineSchema), daffodil autogenerates a schema with root elements in the > "example.com" namespace. And so we hardcode that the namespace for the > root attribute should be "example.com" for negative tests. But your root > element is actually imported with a different namespace, which doesn't > match what we expect. This is why you see the error about not finding > the element "{http://example.com}TpktMessage";. > > Fortunately, it only tries example.com for embedded schemas. For > non-embedded schemas, it's a bit more lax about the namesapce and will > try its best to find a reasonable root element. Your TDML doesn't really > use an embedded schema (it just imports the actual schema), so you could > delete your defineSchema and use the following for parserTestCases instead: > > model="org/apache/plc4x/protocols/s7/protocol.dfdl.xsdd" > > It's a bit longer, but is actually a better way to test your schema. > With an embedded schema Daffodil creates a temporary schema with various > things assumed (like namespaces). By directly referencing the main > schema, you bypass all that. > > - Steve > > > On 2/25/19 12:25 PM, Christofer Dutz wrote: >> Hi Steve, >> >> I created a feature branch with all of my changes: >> https://github.com/apache/incubator-plc4x/tree/feature/daffodil-2.3.0/protocols/s7 >> >> Chris >> >> >> >> Am 25.02.19, 17:15 schrieb "Steve Lawrence" <[email protected]>: >> >> Negative tests should still work. The PCAP schema [1] has been updated >> to use 2.3.0 and it has negative tests that pass and didn't require any >> changes. Can you push your changes somewhere we can take a look? >> >> We did make some fairly significant changes to how the TDML runner >> works, but that should be mostly behind the scenes. >> >> Thanks, >> - Steve >> >> On 2/25/19 10:55 AM, Christofer Dutz wrote: >> > Yeah, it was previously working like that with 2.2.0, when updating I >> had to >> > change quite a bit. Now the types and names are aligned with the >> schema. >> > >> > However I can't seem to be able to define rules that expect parse >> errors (and >> > therefore don't return any XML) any longer. >> > >> > Chris >> > >> > Outlook für Android <https://aka.ms/ghei36> herunterladen >> > >> > >> -------------------------------------------------------------------------------- >> > *From:* Steve Lawrence <[email protected]> >> > *Sent:* Monday, February 25, 2019 4:50:53 PM >> > *To:* [email protected]; Christofer Dutz >> > *Subject:* Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1 >> > Looking at your tdml file in github [1], it looks like your root >> element >> > in defineSchema is "tpktMessage" with a lower-case "t", but your root >> > element in the below parserTestCase has a capital "T". Maybe it's just >> a >> > typo? >> > >> > - Steve >> > >> > [1] >> > >> https://github.com/apache/incubator-plc4x/blob/develop/protocols/s7/src/test/resources/org/apache/plc4x/protocols/s7/protocol.tdml >> > >> > >> > On 2/25/19 10:40 AM, Christofer Dutz wrote: >> >> Hi Steve, >> >> >> >> so that worked >> >> >> >> fn:ceiling(xs:double(../rawLength div 8)) >> >> >> >> Now the last thing I'm having a problem with in TDML is that all >> test-cases where I expect a parsing error - such as: >> >> >> >> <!-- >> >> In this test-case the payload of the packet doesn't match the >> >> length provided in the packet header (Too short) >> >> --> >> >> <tdml:parserTestCase name="TPKT: tooShortPayload" >> >> root="TpktMessage" >> >> model="s7Schema"> >> >> <tdml:document> >> >> <tdml:documentPart >> type="byte">0300001611D00001000200C00109C102010</tdml:documentPart> >> >> </tdml:document> >> >> >> >> <tdml:errors> >> >> <tdml:error>Parse Error</tdml:error> >> >> </tdml:errors> >> >> </tdml:parserTestCase> >> >> >> >> I am getting error messages like this: >> >> >> >> org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: >> daffodil) Did not find diagnostic message "Parse Error" in any of the actual >> diagnostic messages: >> >> Schema Definition Error: No global element found for >> {http://example.com}TpktMessage >> >> Schema context: SchemaSet Location in >> file:/var/folders/4m/d8xs244900sd8ypzndkf4_3w0000gn/T/s7Schema1558250055908446006.dfdl.xsd >> >> Schema Definition Warning: attributeFormDefault is not part of DFDL >> and will be ignored >> >> Schema context: XMLSchemaDocument Location line 1 column 2 in >> file:/var/folders/4m/d8xs244900sd8ypzndkf4_3w0000gn/T/s7Schema1558250055908446006.dfdl.xsd >> >> >> >> at >> org.apache.daffodil.tdml.TDMLException$.apply(TDMLException.scala:31) >> >> at >> org.apache.daffodil.tdml.VerifyTestCase$.$anonfun$verifyAllDiagnosticsFound$4(TDMLRunner.scala:1404) >> >> at >> org.apache.daffodil.tdml.VerifyTestCase$.$anonfun$verifyAllDiagnosticsFound$4$adapted(TDMLRunner.scala:1393) >> >> at scala.collection.immutable.List.foreach(List.scala:389) >> >> >> >> What do I have to change here? >> >> >> >> Chris >> >> >> >> >> >> >> >> Am 25.02.19, 16:22 schrieb "Steve Lawrence" <[email protected]>: >> >> >> >> I'm guessing that this was broken by commit a20ff41728, which >> changed >> >> the result of division to always be an xs:decimal instead of an >> xs:float >> >> or xs:double. Which is the correct behavior and did fix some >> potential bugs. >> >> >> >> What isn't the correct behavior, which appears to have been >> broken for a >> >> while, is rounding of xs:decimals. For example: >> >> >> >> fn:ceiling(xs:decimal(0.125)) >> >> >> >> which results in 0.125 and is clearly not correct. When you cast >> that to >> >> a short, it becomes 0. >> >> >> >> Fortunately, it looks like rounding of a float/double works as >> expected: >> >> >> >> fn:ceiling(xs:double(0.125)) >> >> >> >> So a temporary workaround is to cast the result of your division >> to a >> >> double, e.g: >> >> >> >> fn:ceiling(xs:double(../rawLength div 8)) >> >> >> >> I'll open a bug for this. >> >> >> >> Thanks, >> >> - Steve >> >> >> >> On 2/25/19 9:53 AM, Christofer Dutz wrote: >> >> > Hi Steve, >> >> > >> >> > I don't think it's a blocker either ... >> >> > >> >> > However as part of my tests, I updated the dependency in PLC4X >> to the new Daffodil version. >> >> > I had to fix quite a bit of things afterwards, however I got >> most to work. Mostly Daffodil is now a little pickier than previously, >> however one thing is currently blocking me: >> >> > >> >> > The following expression previously evaluated to >> lengthInBytes=1, now it evaluates to 0 (the transport size is 3 and the >> rawLength is 1) >> >> > >> >> > <xs:element name="lengthInBytes" >> type="s7:short" dfdl:inputValueCalc="{ >> >> > >> xs:unsignedShort(if((../transportSize eq 3) or (../transportSize eq 4) or >> (../transportSize eq 5)) >> >> > then >> xs:short(fn:ceiling(../rawLength div 8)) >> >> > else >> ../rawLength)}"/> >> >> > >> >> > So in general if should be: >> >> > >> >> > lengthInBytes = fn:ceiling(1 div 8) >> >> > >> >> > which should be 1 but is evaluated to 0 now ... even if I >> hard-code it like "fn:ceiling(1 div 8)" the result is 0 >> >> > >> >> > Not quite sure why this is happening ... but it's currently >> breaking my build. >> >> > >> >> > Chris >> >> > >> >> > >> >> > Am 25.02.19, 14:35 schrieb "Steve Lawrence" >> <[email protected]>: >> >> > >> >> > Thanks for tracking this issue down. I can reproduce the >> issue by >> >> > setting -Duser.country and user.language. This is >> definitely a bug, >> >> > anything that affects parsing should come from a DFDL >> property rather >> >> > than the environment. I've created DAFFODIL-2074 to track >> this issue. >> >> > >> >> > That said, I don't think this should be considered a >> blocker for 2.3.0. >> >> > It should definitely be fixed for next release, but setting >> the locale >> >> > is a reasonable workaround for now. >> >> > >> >> > - Steve >> >> > >> >> > On 2/25/19 5:12 AM, Christofer Dutz wrote: >> >> > > Hi, >> >> > > >> >> > > guess I found the main problem. >> >> > > It seems to have been related to me not being a US guy >> ... but wasn't related to the time-zone, but my computer's locale settings. >> >> > > >> >> > > Mine default to "de_DE" where the number formats are >> different. In Germany (and probably some other areas too) we use "," for the >> fractional separator and "." for thousands separator (which is the opposite >> to the US convention). >> >> > > If I add the following settings to my SBT_OPTS: >> -Duser.country=EN -Duser.language=us >> >> > > Almost all previously failing tests succeed, however >> still one still fails: >> >> > > org.apache.daffodil.section05.simple_types.TestSimpleTypes >> >> > > >> >> > > Here looking at the failing tests, it's too probably >> related. For me it looks as if the default locale settings are not correctly >> picked up ... >> >> > > >> >> > > In the end I'm building with these SBT_OPTS ... probably >> it's worth adding a suggested SBT_OPTS in the readme as without it, it is >> impossible to build. >> >> > > >> >> > > export SBT_OPTS="-Xss2M -Xmx5G -Xms5G -Duser.country=EN >> -Duser.language=us" >> >> > > >> >> > > >> >> > > Chris >> >> > > >> >> > > >> >> > > Am 25.02.19, 09:31 schrieb "Christofer Dutz" >> <[email protected]>: >> >> > > >> >> > > Hi guys, >> >> > > >> >> > > being back from my trip I was planning on continuing >> my testing ... especially finding out why some tests are failing on my side. >> >> > > I'll open Issues for them, if I find something. >> >> > > >> >> > > Also did I port some scripts we use in PLC4X and >> Edgent to do the downloading of release candidates which also do the hash >> checking and signature checking ... is there any interest in that? >> >> > > I would be happy to provide them. >> >> > > >> >> > > Chris >> >> > > >> >> > > >> >> > > >> >> > > Am 23.02.19, 10:39 schrieb "Christofer Dutz" >> <[email protected]>: >> >> > > >> >> > > HI and sorry for posting that back to the >> incubator ... I simply hit reply on my phone and for some reason it added >> all other emails. >> >> > > >> >> > > Do you want me to create issues for anything I >> found? It's not stuff that is required to be addressed before the release, I >> think ... but could be helpful for the next. >> >> > > >> >> > > Chris >> >> > > >> >> > > Am 23.02.19, 01:13 schrieb "Christofer Dutz" >> <[email protected]>: >> >> > > >> >> > > I'll check that out after coming home from my >> party trip to Utrecht ;-) >> >> > > >> >> > > If this is required to build, I think it >> should be mentioned in the readme. >> >> > > >> >> > > Chris >> >> > > >> >> > > Outlook für Android<https://aka.ms/ghei36> >> herunterladen >> >> > > >> >> > > ________________________________ >> >> > > From: Craig Russell <[email protected]> >> >> > > Sent: Saturday, February 23, 2019 1:12:06 AM >> >> > > To: Incubator >> >> > > Cc: [email protected] >> >> > > Subject: Re: [VOTE] Release Apache Daffodil >> (incubating) 2.3.0-rc1 >> >> > > >> >> > > > On Feb 19, 2019, at 1:18 PM, Dave Fisher >> <[email protected]> wrote: >> >> > > > >> >> > > > Hi - >> >> > > > >> >> > > > (1) To be fixed on the next release: >> >> > > > (a) Please do not include the KEYS >> file in the source release zip. Refer the users back to the location: >> https://dist.apache.org/repos/dist/release/incubator/daffodil/KEYS >> > <https://dist.apache.org/repos/dist/release/incubator/daffodil/KEYS> >> >> > > >> >> > > Yes the KEYS must not be distributed in the >> source release. But the canonical location for KEYS is >> https://www.apache.org/dist/incubator/daffodil/KEYS >> > <https://www.apache.org/dist/incubator/daffodil/KEYS> >> >> > > >> >> > > The dist.apache.org url is not for public use. >> >> > > >> >> > > Craig >> >> > > >> >> > > > (b) Discuss vendor specific tests >> that are name *ibm* and *tresys* on dev@daffodil. I think it would help >> community building to have something more feature specific, but I could be >> wrong. >> >> > > > (2) Rat Check passes. >> >> > > > (3) All Signatures pass. >> >> > > > (4) All Checksums pass. >> >> > > > (5) LICENSE, NOTICE and DISCLAIMER look >> good. (MSI was not checked. I do not have a windows machine available.) >> >> > > > (6) Compiles >> >> > > > (7) sbt test “runs”. I did have a large >> number of OOM and other errors and they took over 3 hours to run on a >> MacBook Pro with the latest macOS. >> >> > > > (8) sbt it:test worked well. >> >> > > > >> >> > > > +1 to all packages except for MSI. >> >> > > > +0 on MSI binary convenience package - I >> cannot check it. >> >> > > > >> >> > > > Regards, >> >> > > > Dave >> >> > > > >> >> > > >> On Feb 19, 2019, at 5:21 AM, Steve >> Lawrence <[email protected]> wrote: >> >> > > >> >> >> > > >> Hi all, >> >> > > >> >> >> > > >> The Apache Daffodil community has voted >> and approved the proposed >> >> > > >> release of Apache Daffodil (incubating) >> 2.3.0-rc1. >> >> > > >> >> >> > > >> We now kindly request the Incubator PMC >> members review and vote on this >> >> > > >> incubator release. >> >> > > >> >> >> > > >> Daffodil is an open source implementation >> of the DFDL specification that >> >> > > >> uses DFDL schemas to parse fixed format >> data into an infoset, which is >> >> > > >> most commonly represented as either XML or >> JSON. This allows the use of >> >> > > >> well-established XML or JSON technologies >> and libraries to consume, >> >> > > >> inspect, and manipulate fixed format data >> in existing solutions. >> >> > > >> Daffodil is also capable of the reverse by >> serializing or "unparsing" an >> >> > > >> XML or JSON infoset back to the original >> data format. >> >> > > >> >> >> > > >> Vote thread: >> >> > > >> >> https://lists.apache.org/thread.html/0ec1161ad296aebda54e86c219e3f1c4498a7de640c6342e6888aea1@%3Cdev.daffodil.apache.org%3E >> >> > > >> >> >> > > >> Result thread: >> >> > > >> >> >> > > >> >> https://lists.apache.org/thread.html/a195a77640235501109bdea5ec7fb9a34c6f50644c7173fb8a919989@%3Cdev.daffodil.apache.org%3E >> >> > > >> >> >> > > >> All distribution packages, including >> signatures, digests, etc. can be >> >> > > >> found at: >> >> > > >> >> >> > > >> >> https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.3.0-rc1/ >> >> > > >> >> >> > > >> Staging artifacts can be found at: >> >> > > >> >> >> > > >> >> https://repository.apache.org/content/repositories/orgapachedaffodil-1008/ >> >> > > >> >> >> > > >> This release has been signed with PGP key >> 36F3494B033AE661, >> >> > > >> corresponding to [email protected], >> which is included in the >> >> > > >> repository's KEYS file. This key can be >> found on keyservers, such as: >> >> > > >> >> >> > > >> >> http://pgp.mit.edu/pks/lookup?op=get&search=0x36F3494B033AE661 >> >> > > >> >> >> > > >> It is also listed here: >> >> > > >> >> >> > > >> >> https://people.apache.org/keys/committer/slawrence.asc >> >> > > >> >> >> > > >> The release candidate has been tagged in >> git with v2.3.0-rc1 on commit >> >> > > >> hash >> 193c84690401debd93c1f48e39e564fec09e9337. >> >> > > >> >> >> > > >> For reference, here is a list of all >> closed JIRAs tagged with 2.3.0: >> >> > > >> >> >> > > >> >> https://issues.apache.org/jira/browse/DAFFODIL-2067?jql=project%20%3D%20DAFFODIL%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.3.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC >> >> > > >> >> >> > > >> For a summary of the changes in this >> release, see: >> >> > > >> >> >> > > >> https://daffodil.apache.org/releases/2.3.0/ >> >> > > >> >> >> > > >> Please review and vote. The vote will be >> open for at least 72 hours. >> >> > > >> >> >> > > >> [ ] +1 approve >> >> > > >> [ ] +0 no opinion >> >> > > >> [ ] -1 disapprove (and reason why) >> >> > > >> >> >> > > >> Thanks, >> >> > > >> - Steve >> >> > > >> >> >> > > >> >> --------------------------------------------------------------------- >> >> > > >> To unsubscribe, e-mail: >> [email protected] >> >> > > >> For additional commands, e-mail: >> [email protected] >> >> > > >> >> >> > > > >> >> > > >> >> > > Craig L Russell >> >> > > Secretary, Apache Software Foundation >> >> > > [email protected] <mailto:[email protected]> >> http://db.apache.org/jdo <http://db.apache.org/jdo> >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > >> >> > >> >> > >> >> >> >> >> >> >> > >> >> >> >
