Ok, I've managed to track down the issue. The TDML runner was changed to
be a bit more strict about namespaces, and now wants to know the
namespace of the root element used for a parse.
But the "root" attribute in a parserTestCase does not contain namespace
information (it probably should, but we used to be very lax about
namespaces, so many of our tests do not provide one). So in order to get
the namespace information, the TDML runner queries the namespace of the
root of the expected infoset. For positive test cases, it can do this
and determine the namespace should be "s7". But this information does
not exist for negative test cases.
For negative tests that use an embedded schema (like you have via the
defineSchema), daffodil autogenerates a schema with root elements in the
"example.com" namespace. And so we hardcode that the namespace for the
root attribute should be "example.com" for negative tests. But your root
element is actually imported with a different namespace, which doesn't
match what we expect. This is why you see the error about not finding
the element "{http://example.com}TpktMessage";.
Fortunately, it only tries example.com for embedded schemas. For
non-embedded schemas, it's a bit more lax about the namesapce and will
try its best to find a reasonable root element. Your TDML doesn't really
use an embedded schema (it just imports the actual schema), so you could
delete your defineSchema and use the following for parserTestCases instead:
model="org/apache/plc4x/protocols/s7/protocol.dfdl.xsdd"
It's a bit longer, but is actually a better way to test your schema.
With an embedded schema Daffodil creates a temporary schema with various
things assumed (like namespaces). By directly referencing the main
schema, you bypass all that.
- Steve
On 2/25/19 12:25 PM, Christofer Dutz wrote:
> Hi Steve,
>
> I created a feature branch with all of my changes:
> https://github.com/apache/incubator-plc4x/tree/feature/daffodil-2.3.0/protocols/s7
>
> Chris
>
>
>
> Am 25.02.19, 17:15 schrieb "Steve Lawrence" <[email protected]>:
>
> Negative tests should still work. The PCAP schema [1] has been updated
> to use 2.3.0 and it has negative tests that pass and didn't require any
> changes. Can you push your changes somewhere we can take a look?
>
> We did make some fairly significant changes to how the TDML runner
> works, but that should be mostly behind the scenes.
>
> Thanks,
> - Steve
>
> On 2/25/19 10:55 AM, Christofer Dutz wrote:
> > Yeah, it was previously working like that with 2.2.0, when updating I
> had to
> > change quite a bit. Now the types and names are aligned with the schema.
> >
> > However I can't seem to be able to define rules that expect parse
> errors (and
> > therefore don't return any XML) any longer.
> >
> > Chris
> >
> > Outlook für Android <https://aka.ms/ghei36> herunterladen
> >
> >
> --------------------------------------------------------------------------------
> > *From:* Steve Lawrence <[email protected]>
> > *Sent:* Monday, February 25, 2019 4:50:53 PM
> > *To:* [email protected]; Christofer Dutz
> > *Subject:* Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1
> > Looking at your tdml file in github [1], it looks like your root element
> > in defineSchema is "tpktMessage" with a lower-case "t", but your root
> > element in the below parserTestCase has a capital "T". Maybe it's just a
> > typo?
> >
> > - Steve
> >
> > [1]
> >
> https://github.com/apache/incubator-plc4x/blob/develop/protocols/s7/src/test/resources/org/apache/plc4x/protocols/s7/protocol.tdml
> >
> >
> > On 2/25/19 10:40 AM, Christofer Dutz wrote:
> >> Hi Steve,
> >>
> >> so that worked
> >>
> >> fn:ceiling(xs:double(../rawLength div 8))
> >>
> >> Now the last thing I'm having a problem with in TDML is that all
> test-cases where I expect a parsing error - such as:
> >>
> >> <!--
> >> In this test-case the payload of the packet doesn't match the
> >> length provided in the packet header (Too short)
> >> -->
> >> <tdml:parserTestCase name="TPKT: tooShortPayload"
> >> root="TpktMessage"
> >> model="s7Schema">
> >> <tdml:document>
> >> <tdml:documentPart
> type="byte">0300001611D00001000200C00109C102010</tdml:documentPart>
> >> </tdml:document>
> >>
> >> <tdml:errors>
> >> <tdml:error>Parse Error</tdml:error>
> >> </tdml:errors>
> >> </tdml:parserTestCase>
> >>
> >> I am getting error messages like this:
> >>
> >> org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: daffodil)
> Did not find diagnostic message "Parse Error" in any of the actual diagnostic
> messages:
> >> Schema Definition Error: No global element found for
> {http://example.com}TpktMessage
> >> Schema context: SchemaSet Location in
> file:/var/folders/4m/d8xs244900sd8ypzndkf4_3w0000gn/T/s7Schema1558250055908446006.dfdl.xsd
> >> Schema Definition Warning: attributeFormDefault is not part of DFDL
> and will be ignored
> >> Schema context: XMLSchemaDocument Location line 1 column 2 in
> file:/var/folders/4m/d8xs244900sd8ypzndkf4_3w0000gn/T/s7Schema1558250055908446006.dfdl.xsd
> >>
> >> at
> org.apache.daffodil.tdml.TDMLException$.apply(TDMLException.scala:31)
> >> at
> org.apache.daffodil.tdml.VerifyTestCase$.$anonfun$verifyAllDiagnosticsFound$4(TDMLRunner.scala:1404)
> >> at
> org.apache.daffodil.tdml.VerifyTestCase$.$anonfun$verifyAllDiagnosticsFound$4$adapted(TDMLRunner.scala:1393)
> >> at scala.collection.immutable.List.foreach(List.scala:389)
> >>
> >> What do I have to change here?
> >>
> >> Chris
> >>
> >>
> >>
> >> Am 25.02.19, 16:22 schrieb "Steve Lawrence" <[email protected]>:
> >>
> >> I'm guessing that this was broken by commit a20ff41728, which
> changed
> >> the result of division to always be an xs:decimal instead of an
> xs:float
> >> or xs:double. Which is the correct behavior and did fix some
> potential bugs.
> >>
> >> What isn't the correct behavior, which appears to have been broken
> for a
> >> while, is rounding of xs:decimals. For example:
> >>
> >> fn:ceiling(xs:decimal(0.125))
> >>
> >> which results in 0.125 and is clearly not correct. When you cast
> that to
> >> a short, it becomes 0.
> >>
> >> Fortunately, it looks like rounding of a float/double works as
> expected:
> >>
> >> fn:ceiling(xs:double(0.125))
> >>
> >> So a temporary workaround is to cast the result of your division
> to a
> >> double, e.g:
> >>
> >> fn:ceiling(xs:double(../rawLength div 8))
> >>
> >> I'll open a bug for this.
> >>
> >> Thanks,
> >> - Steve
> >>
> >> On 2/25/19 9:53 AM, Christofer Dutz wrote:
> >> > Hi Steve,
> >> >
> >> > I don't think it's a blocker either ...
> >> >
> >> > However as part of my tests, I updated the dependency in PLC4X
> to the new Daffodil version.
> >> > I had to fix quite a bit of things afterwards, however I got
> most to work. Mostly Daffodil is now a little pickier than previously,
> however one thing is currently blocking me:
> >> >
> >> > The following expression previously evaluated to
> lengthInBytes=1, now it evaluates to 0 (the transport size is 3 and the
> rawLength is 1)
> >> >
> >> > <xs:element name="lengthInBytes"
> type="s7:short" dfdl:inputValueCalc="{
> >> >
> xs:unsignedShort(if((../transportSize eq 3) or (../transportSize eq 4) or
> (../transportSize eq 5))
> >> > then
> xs:short(fn:ceiling(../rawLength div 8))
> >> > else
> ../rawLength)}"/>
> >> >
> >> > So in general if should be:
> >> >
> >> > lengthInBytes = fn:ceiling(1 div 8)
> >> >
> >> > which should be 1 but is evaluated to 0 now ... even if I
> hard-code it like "fn:ceiling(1 div 8)" the result is 0
> >> >
> >> > Not quite sure why this is happening ... but it's currently
> breaking my build.
> >> >
> >> > Chris
> >> >
> >> >
> >> > Am 25.02.19, 14:35 schrieb "Steve Lawrence"
> <[email protected]>:
> >> >
> >> > Thanks for tracking this issue down. I can reproduce the
> issue by
> >> > setting -Duser.country and user.language. This is definitely
> a bug,
> >> > anything that affects parsing should come from a DFDL
> property rather
> >> > than the environment. I've created DAFFODIL-2074 to track
> this issue.
> >> >
> >> > That said, I don't think this should be considered a blocker
> for 2.3.0.
> >> > It should definitely be fixed for next release, but setting
> the locale
> >> > is a reasonable workaround for now.
> >> >
> >> > - Steve
> >> >
> >> > On 2/25/19 5:12 AM, Christofer Dutz wrote:
> >> > > Hi,
> >> > >
> >> > > guess I found the main problem.
> >> > > It seems to have been related to me not being a US guy ...
> but wasn't related to the time-zone, but my computer's locale settings.
> >> > >
> >> > > Mine default to "de_DE" where the number formats are
> different. In Germany (and probably some other areas too) we use "," for the
> fractional separator and "." for thousands separator (which is the opposite
> to the US convention).
> >> > > If I add the following settings to my SBT_OPTS:
> -Duser.country=EN -Duser.language=us
> >> > > Almost all previously failing tests succeed, however still
> one still fails:
> >> > > org.apache.daffodil.section05.simple_types.TestSimpleTypes
> >> > >
> >> > > Here looking at the failing tests, it's too probably
> related. For me it looks as if the default locale settings are not correctly
> picked up ...
> >> > >
> >> > > In the end I'm building with these SBT_OPTS ... probably
> it's worth adding a suggested SBT_OPTS in the readme as without it, it is
> impossible to build.
> >> > >
> >> > > export SBT_OPTS="-Xss2M -Xmx5G -Xms5G -Duser.country=EN
> -Duser.language=us"
> >> > >
> >> > >
> >> > > Chris
> >> > >
> >> > >
> >> > > Am 25.02.19, 09:31 schrieb "Christofer Dutz"
> <[email protected]>:
> >> > >
> >> > > Hi guys,
> >> > >
> >> > > being back from my trip I was planning on continuing
> my testing ... especially finding out why some tests are failing on my side.
> >> > > I'll open Issues for them, if I find something.
> >> > >
> >> > > Also did I port some scripts we use in PLC4X and
> Edgent to do the downloading of release candidates which also do the hash
> checking and signature checking ... is there any interest in that?
> >> > > I would be happy to provide them.
> >> > >
> >> > > Chris
> >> > >
> >> > >
> >> > >
> >> > > Am 23.02.19, 10:39 schrieb "Christofer Dutz"
> <[email protected]>:
> >> > >
> >> > > HI and sorry for posting that back to the
> incubator ... I simply hit reply on my phone and for some reason it added all
> other emails.
> >> > >
> >> > > Do you want me to create issues for anything I
> found? It's not stuff that is required to be addressed before the release, I
> think ... but could be helpful for the next.
> >> > >
> >> > > Chris
> >> > >
> >> > > Am 23.02.19, 01:13 schrieb "Christofer Dutz"
> <[email protected]>:
> >> > >
> >> > > I'll check that out after coming home from my
> party trip to Utrecht ;-)
> >> > >
> >> > > If this is required to build, I think it
> should be mentioned in the readme.
> >> > >
> >> > > Chris
> >> > >
> >> > > Outlook für Android<https://aka.ms/ghei36>
> herunterladen
> >> > >
> >> > > ________________________________
> >> > > From: Craig Russell <[email protected]>
> >> > > Sent: Saturday, February 23, 2019 1:12:06 AM
> >> > > To: Incubator
> >> > > Cc: [email protected]
> >> > > Subject: Re: [VOTE] Release Apache Daffodil
> (incubating) 2.3.0-rc1
> >> > >
> >> > > > On Feb 19, 2019, at 1:18 PM, Dave Fisher
> <[email protected]> wrote:
> >> > > >
> >> > > > Hi -
> >> > > >
> >> > > > (1) To be fixed on the next release:
> >> > > > (a) Please do not include the KEYS
> file in the source release zip. Refer the users back to the location:
> https://dist.apache.org/repos/dist/release/incubator/daffodil/KEYS
> > <https://dist.apache.org/repos/dist/release/incubator/daffodil/KEYS>
> >> > >
> >> > > Yes the KEYS must not be distributed in the
> source release. But the canonical location for KEYS is
> https://www.apache.org/dist/incubator/daffodil/KEYS
> > <https://www.apache.org/dist/incubator/daffodil/KEYS>
> >> > >
> >> > > The dist.apache.org url is not for public use.
> >> > >
> >> > > Craig
> >> > >
> >> > > > (b) Discuss vendor specific tests
> that are name *ibm* and *tresys* on dev@daffodil. I think it would help
> community building to have something more feature specific, but I could be
> wrong.
> >> > > > (2) Rat Check passes.
> >> > > > (3) All Signatures pass.
> >> > > > (4) All Checksums pass.
> >> > > > (5) LICENSE, NOTICE and DISCLAIMER look
> good. (MSI was not checked. I do not have a windows machine available.)
> >> > > > (6) Compiles
> >> > > > (7) sbt test “runs”. I did have a large
> number of OOM and other errors and they took over 3 hours to run on a MacBook
> Pro with the latest macOS.
> >> > > > (8) sbt it:test worked well.
> >> > > >
> >> > > > +1 to all packages except for MSI.
> >> > > > +0 on MSI binary convenience package - I
> cannot check it.
> >> > > >
> >> > > > Regards,
> >> > > > Dave
> >> > > >
> >> > > >> On Feb 19, 2019, at 5:21 AM, Steve Lawrence
> <[email protected]> wrote:
> >> > > >>
> >> > > >> Hi all,
> >> > > >>
> >> > > >> The Apache Daffodil community has voted and
> approved the proposed
> >> > > >> release of Apache Daffodil (incubating)
> 2.3.0-rc1.
> >> > > >>
> >> > > >> We now kindly request the Incubator PMC
> members review and vote on this
> >> > > >> incubator release.
> >> > > >>
> >> > > >> Daffodil is an open source implementation
> of the DFDL specification that
> >> > > >> uses DFDL schemas to parse fixed format
> data into an infoset, which is
> >> > > >> most commonly represented as either XML or
> JSON. This allows the use of
> >> > > >> well-established XML or JSON technologies
> and libraries to consume,
> >> > > >> inspect, and manipulate fixed format data
> in existing solutions.
> >> > > >> Daffodil is also capable of the reverse by
> serializing or "unparsing" an
> >> > > >> XML or JSON infoset back to the original
> data format.
> >> > > >>
> >> > > >> Vote thread:
> >> > > >>
> https://lists.apache.org/thread.html/0ec1161ad296aebda54e86c219e3f1c4498a7de640c6342e6888aea1@%3Cdev.daffodil.apache.org%3E
> >> > > >>
> >> > > >> Result thread:
> >> > > >>
> >> > > >>
> https://lists.apache.org/thread.html/a195a77640235501109bdea5ec7fb9a34c6f50644c7173fb8a919989@%3Cdev.daffodil.apache.org%3E
> >> > > >>
> >> > > >> All distribution packages, including
> signatures, digests, etc. can be
> >> > > >> found at:
> >> > > >>
> >> > > >>
> https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.3.0-rc1/
> >> > > >>
> >> > > >> Staging artifacts can be found at:
> >> > > >>
> >> > > >>
> https://repository.apache.org/content/repositories/orgapachedaffodil-1008/
> >> > > >>
> >> > > >> This release has been signed with PGP key
> 36F3494B033AE661,
> >> > > >> corresponding to [email protected],
> which is included in the
> >> > > >> repository's KEYS file. This key can be
> found on keyservers, such as:
> >> > > >>
> >> > > >>
> http://pgp.mit.edu/pks/lookup?op=get&search=0x36F3494B033AE661
> >> > > >>
> >> > > >> It is also listed here:
> >> > > >>
> >> > > >>
> https://people.apache.org/keys/committer/slawrence.asc
> >> > > >>
> >> > > >> The release candidate has been tagged in
> git with v2.3.0-rc1 on commit
> >> > > >> hash
> 193c84690401debd93c1f48e39e564fec09e9337.
> >> > > >>
> >> > > >> For reference, here is a list of all closed
> JIRAs tagged with 2.3.0:
> >> > > >>
> >> > > >>
> https://issues.apache.org/jira/browse/DAFFODIL-2067?jql=project%20%3D%20DAFFODIL%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.3.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> >> > > >>
> >> > > >> For a summary of the changes in this
> release, see:
> >> > > >>
> >> > > >> https://daffodil.apache.org/releases/2.3.0/
> >> > > >>
> >> > > >> Please review and vote. The vote will be
> open for at least 72 hours.
> >> > > >>
> >> > > >> [ ] +1 approve
> >> > > >> [ ] +0 no opinion
> >> > > >> [ ] -1 disapprove (and reason why)
> >> > > >>
> >> > > >> Thanks,
> >> > > >> - Steve
> >> > > >>
> >> > > >>
> ---------------------------------------------------------------------
> >> > > >> To unsubscribe, e-mail:
> [email protected]
> >> > > >> For additional commands, e-mail:
> [email protected]
> >> > > >>
> >> > > >
> >> > >
> >> > > Craig L Russell
> >> > > Secretary, Apache Software Foundation
> >> > > [email protected] <mailto:[email protected]>
> http://db.apache.org/jdo <http://db.apache.org/jdo>
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
