Re: Daffodil ignoring dfdl:length=0 when dfdl:lengthKind=explicit

2019-01-15 Thread Beckerle, Mike
Christofer, So looking at your schemas I think the concept you are missing is that just because something is zero length doesn't mean the parser won't try to parse against it. A DFDL schema could parse, with zero data, consume zero bits of course, yet still populate an infoset with default

Re: Daffodil ignoring dfdl:length=0 when dfdl:lengthKind=explicit

2019-01-15 Thread Beckerle, Mike
Christofer, So in DFDL, unlike regular XSD, only elements can be repeating/optional. You can't have max/minOccurs on a choice or sequence, only on an element. DFDL isn't only for XML. Many data models that the DFDL infoset could be projected into, those data models don't allow repeating or

Re: First success with modeling the message formats for the Siemens S7 Protocol with Daffodil

2019-01-17 Thread Beckerle, Mike
Very nice schema Christofer. I recommend adding tests in TDML so as to exercise these in your ordinary unit test/regression using Daffodil. This article: https://daffodil.apache.org/dfdl-layout/ explains the way we do this for sbt. Minor adaptations needed for maven I'm sure. ...mike

Re: First success with modeling the message formats for the Siemens S7 Protocol with Daffodil

2019-01-17 Thread Beckerle, Mike
tion about the TDML format is at: https://daffodil.apache.org/tdml - Steve On 1/17/19 9:44 AM, Beckerle, Mike wrote: > Very nice schema Christofer. > > > I recommend adding tests in TDML so as to exercise these in your ordinary unit test/reg

Re: Using DFDL to generate model, parser and generator?

2019-01-10 Thread Beckerle, Mike
d probably build the maven plugin in a way that it makes things easier to run it on the Command line or build plugins for SBT, Gradle, Ant whatsoever ... What do you think? Chris Am 09.01.19, 20:10 schrieb "Beckerle, Mike" : Christofer, Yes what you suggest is possible,

Re: Using DFDL to generate model, parser and generator?

2019-01-10 Thread Beckerle, Mike
es or exactly this number of parameters which summed up size matches a total parameter size ... Hope it is possible to model stuff like this with DFDL. Chris [1] https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java Am 10.01.19, 14:47 schrieb "Beckerl

Re: Using DFDL to generate model, parser and generator?

2019-01-09 Thread Beckerle, Mike
Christofer, Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly this right now. Effectively what you are describing is a code-generator backend for Daffodil. I think this is a great idea. I

Re: [DISCUSS] Release Apache Daffodil 2.3.0

2019-01-28 Thread Beckerle, Mike
I know of nothing that should be blocking the release. I closed PR 158. I will update our roadmap on the project wiki. From: Steve Lawrence Sent: Monday, January 28, 2019 7:55:55 AM To: dev@daffodil.apache.org Subject: [DISCUSS] Release Apache Daffodil 2.3.0

Re: Defining Protocol State-Machines

2019-01-28 Thread Beckerle, Mike
Christofer, Your idea here sounds to me like a toolkit using on an XML-based language, for describing communicating state machines, or protocol handshakes. This is a good idea. From a systems-security perspective, creating these by starting with a Turing-complete programming language is

Re: Further design difficulties with TypeValueCalculators

2019-04-04 Thread Beckerle, Mike
The type stMinlength1 is clearly not suitable for many kinds of reuse. Not all top level type defs will be suitable for use as a rep type because they are just incompletely specified. As you point out here, if used without supplying a different lengthKind it will SDE. This is ok. It is just a

Re: Further design difficulties with TypeValueCalculators

2019-04-04 Thread Beckerle, Mike
ill probably never be "correct", ________ From: Beckerle, Mike Sent: Thursday, April 4, 2019 6:44:18 PM To: dev@daffodil.apache.org Subject: Re: Further design difficulties with TypeValueCalculators The type stMinlength1 is clearly not suitable for many kinds of reuse.

Re: Errors in existing test schemas

2019-04-04 Thread Beckerle, Mike
Correct with one caveat. If the types are unused then sure, delete. But could they be illegal like this on purpose for negative tests checking for those diagnostics? If those diagnostics aren't being captured by oolag and saved then that could cause the tests to be failing now. Get Outlook

Re: Exposing latent SDEs

2019-04-05 Thread Beckerle, Mike
Do we have to compile simple types even if unused? Cant we compile them lazily if used. I am very happy to restrict expressions that use simple type qnames for them to have to be literal constants. Then compiling the expressions would provide the qnames of the types actually being used. Get

Re: Accessing schema information from runtime system

2019-04-01 Thread Beckerle, Mike
The lookup table should become part of the SchemaSetRuntimeData. That way it is global to all. Get Outlook for Android From: Sloane, Brandon Sent: Monday, April 1, 2019 4:43:18 PM To: dev@daffodil.apache.org Subject: Accessing schema

Re: Accessing schema information from runtime system

2019-04-01 Thread Beckerle, Mike
complete reference to the ParseOrUnparseState is related to suspending unparse; and that (since the mapping is static) there shouldn't be any issue in my adding a pointer to it in DState? ________ From: Beckerle, Mike Sent: Monday, April 1, 2019 5:03:43 PM To: dev@daffodil.ap

Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1

2019-02-25 Thread Beckerle, Mike
This root attribute issue sounds like a bug also to me. The root for a parser/unparserTestCase should be a QName. For compatibility it can be assumed to be in example.com if unqualified, though ideally that decision (what the default namespace is) ought to be left to the TDML file author.

Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1

2019-02-25 Thread Beckerle, Mike
You will need to fix all Schema Definition errors before any parse error would ever be detected. Schema Definition errors stop compilation of the schema, no parse ever is run if there is any SDE. From: Christofer Dutz Sent: Monday, February 25, 2019

Re: Improving the compatibility with IntellJ

2019-02-25 Thread Beckerle, Mike
Chris, So I have IntelliJ IDEA (Ultimate 2018.2). I don't use it often, I've been able to manage with Eclipse via ScalaIDE, and I know eclipse better. IDEA does look quite good though, and I hope to learn it better. I was able to checkout master of Daffodil, then just start IDEA, and create

Re: [DRAFT] Daffodil Podling Report - March 2019

2019-03-04 Thread Beckerle, Mike
Looks good to me. From: Steve Lawrence Sent: Monday, March 4, 2019 10:03:53 AM To: dev@daffodil.apache.org Subject: [DRAFT] Daffodil Podling Report - March 2019 Daffodil Apache Daffodil is an implementation of the Data Format Description Language (DFDL) used to

Re: Travis auto build/test failed - but I can't figure out why. Was: Fw: Failed: mbeckerle/incubator-daffodil#101 (daffodil-2092 - 0972ac9)

2019-03-08 Thread Beckerle, Mike
Apologies. Nevermind. As usual, pilot error. From: Beckerle, Mike Sent: Friday, March 8, 2019 5:42:43 PM To: dev@daffodil.apache.org Subject: Travis auto build/test failed - but I can't figure out why. Was: Fw: Failed: mbeckerle/incubator-daffodil#101 (daffodil

Travis auto build/test failed - but I can't figure out why. Was: Fw: Failed: mbeckerle/incubator-daffodil#101 (daffodil-2092 - 0972ac9)

2019-03-08 Thread Beckerle, Mike
nce impact. Any insights appreciated. ...mike beckerle Tresys From: Travis CI Sent: Friday, March 8, 2019 5:08 PM To: Beckerle, Mike Subject: Failed: mbeckerle/incubator-daffodil#101 (daffodil-2092 - 0972ac9) <https://travis-ci.com/mbeckerle/incub

Re: Multiple children of DIDocument

2019-03-19 Thread Beckerle, Mike
I have no issues. The DIDocument is effectively a degenerate DIComplex node. Your suggested change makes it ever so slightly less degenerate, and only temporarily during the lifetime of the repType quasi-element. Get Outlook for Android

Re: [General] Any plans for submitting anything to ApacheConNA?

2019-03-21 Thread Beckerle, Mike
I am definitely planning to submit a topic. We're doing some interesting things enhancing DFDL schemas to do unit-normalization so that the output from parsing is transformed from exactly what is in the data, to more useful units of measure that are directly application-usable.

Analysis of copying/backpointers - schema compiler space/speed - Daffodil-1444

2019-02-13 Thread Beckerle, Mike
Re: https://issues.apache.org/jira/browse/DAFFODIL-1444 I pulled the thread on the backpointers/copying stuff until I ran out of things for it to effect. This page is a list of the results of the experiment. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090465

Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1

2019-02-14 Thread Beckerle, Mike
+1 From: Steve Lawrence Sent: Thursday, February 14, 2019 1:11:13 PM To: dev@daffodil.apache.org Subject: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1 Hi all, I'd like to call a vote to release Apache Daffodil (incubating) 2.3.0-rc1. All distribution

Re: Exposing latent SDEs

2019-04-08 Thread Beckerle, Mike
this gets us is the ability to compile broken schema so long as the broken part is not being used. Apart from backwards compatibility concerns, I am not sure we are doing anyone any favors by allowing this. ____ From: Beckerle, Mike Sent: Friday, April 5, 2019 5:59:

Re: Exposing latent SDEs

2019-04-10 Thread Beckerle, Mike
art is not being used. Apart from backwards compatibility concerns, I am not sure we are doing anyone any favors by allowing this. ________ From: Beckerle, Mike Sent: Friday, April 5, 2019 5:59:12 PM To: dev@daffodil.apache.org Subject: Re: Exposing latent SDEs Do we have

Re: NodeInfo confusion

2019-04-15 Thread Beckerle, Mike
that SignedInteger is never actually used (beyond as a glorified alias to Integer in the scala type system). From: Beckerle, Mike Sent: Friday, April 12, 2019 7:21:52 PM To: dev@daffodil.apache.org Subject: Re: NodeInfo confusion Type system matches the dfdl/XML schema type

Re: NodeInfo confusion

2019-04-12 Thread Beckerle, Mike
Type system matches the dfdl/XML schema type system, not the scala type system, but has some extra nodes that are there for convenience also. In dfdl/XML decimal is super type of integer. Get Outlook for Android From: Sloane, Brandon Sent: Friday, April 12, 6:17 PM

Re: [DRAFT] Daffodil Podling Report - June 2019

2019-05-31 Thread Beckerle, Mike
+1. From: Steve Lawrence Sent: Friday, May 31, 2019 8:12:58 AM To: dev@daffodil.apache.org Subject: [DRAFT] Daffodil Podling Report - June 2019 Daffodil Apache Daffodil is an implementation of the Data Format Description Language (DFDL) used to convert between

Re: Proposal: Add support for lookahead in DFDL expressions

2019-05-31 Thread Beckerle, Mike
Even if we already had it implemented, I think it would still be worthwhile to have a lookAhead function, just because its usage is simpler. I'll go ahead and create the wiki page, then add the feature to Daffodil. ________ From: Beckerle, Mike Sent: Thursday, May 30, 201

Re: dfdl:nil* properties on xs:simpleType allowed?

2019-06-07 Thread Beckerle, Mike
You are correct. The property combining logic isn't (or wasn't) enforcing that the nil-related properties are only allowed on elements. Personally I think this is a mistake. But it is what the spec says, so we should change our tests to comply. From: Steve

Re: Scala is broken?

2019-06-06 Thread Beckerle, Mike
Yup. Autoboxing mixed with type erasure means the Numbers.asAnyRef conversion can't tell the difference between a JLong and a Maybe[JLong]. Both are, or are silently converted to AnyRef. We should be able to live without Numbers.asAnyRef. It's supposed to be for numbers also, so it ought to

Re: Proposal: Add support for lookahead in DFDL expressions

2019-05-30 Thread Beckerle, Mike
ounter other formats that require it. If this were a more complicated/invasive feature, I would share your concern about adding it to DFDL, but it does not strike me as something that would impose significant maintenance or future development constraints. From: Be

status w.r.t. interoperability goals

2019-05-29 Thread Beckerle, Mike
We are down to exactly 2 schemas that can and should interoperate, but don't yet: * VCard - needs the unordered sequences feature to be added to daffodil * EDIFACT - not sure why not working yet. All the other schemas that can interoperate and have tests, do so including HL7,

Re: Proposal: Add support for lookahead in DFDL expressions

2019-05-29 Thread Beckerle, Mike
Yup. This is a real situation. I think this happens in fixed length data for very mundane reasons explained in the attached slides. Practically speaking, in Link16 this comes up like twice or 3 times, so this isn't worth enhancing DFDL/Daffodil for unless this phenomenon is observed in more

Re: Proposal: Add choiceBranchKeyRanges annotation

2019-06-14 Thread Beckerle, Mike
Brandon, can you add to your proposal discussion of how many practical use-cases there are for this? We can add features to Daffodil of course if we want. I want to understand vis a vis influencing the DFDL standard in the future, how many cases there are of this situation.

Re: Character Encodings - No Statement

2019-06-17 Thread Beckerle, Mike
This sounds like fixed length data fields, or min-length data fields. So the character to use wants to be similar in concept to the pad character - i.e., it is used to add length to a fixed length field, but has no significance. I suggest using U+A0 which is "Non Break Space". This is a space

Re: [DISCUSS] Release Daffodil 2.4.0

2019-06-25 Thread Beckerle, Mike
I concur we need to do a release. I prioritized two bugs as blocking/critical a while back since I think they're quite important. https://issues.apache.org/jira/browse/DAFFODIL-1931 https://issues.apache.org/jira/browse/DAFFODIL-1477

Re: Missing typecalc functions

2019-06-25 Thread Beckerle, Mike
namic type calculators, but maybe that flexibility isn't necessary. On 6/25/19 12:02 PM, Beckerle, Mike wrote: > Where Double is needed, one often finds that Decimal is needed. > > > Anywhere that base-10 rounding semantics is required one tends to need > Decimal. > > > For

Re: Missing typecalc functions

2019-06-25 Thread Beckerle, Mike
Where Double is needed, one often finds that Decimal is needed. Anywhere that base-10 rounding semantics is required one tends to need Decimal. For date/time I know of one exception example that came up in the DFDL workgroup. They had time expressed in units of measure not supported by

Re: Missing typecalc functions

2019-06-25 Thread Beckerle, Mike
full runtime state > along, that way we will not need to re-do the boilerplate if we find another > piece of state we need to include. > > > From: Beckerle, Mike > Sent: Tuesday, June 25, 2019 1:28:19 PM > To: dev@daffodil.apache.org > Subject:

Re: Proposal: Add choiceBranchKeyRanges annotation

2019-06-19 Thread Beckerle, Mike
eptually, in is just providing a shorthand specifying a large number of branch keys. ________ From: Beckerle, Mike Sent: Friday, June 14, 2019 11:53:25 AM To: dev@daffodil.apache.org Subject: Re: Proposal: Add choiceBranchKeyRanges annotation Brandon, can you add to your

Re: Proposal: Add choiceBranchKeyRanges annotation

2019-06-19 Thread Beckerle, Mike
support for multiple choiceBranchKeys, but I cannot find that one. From: Beckerle, Mike Sent: Wednesday, June 19, 2019 11:11:38 AM To: dev@daffodil.apache.org Subject: Re: Proposal: Add choiceBranchKeyRanges annotation Seems sensible to add

Re: Using tns: prefix in tdml files

2019-06-19 Thread Beckerle, Mike
Daffodil's TDML runner takes the schemas found inside the ".tdml" and creates a schema and defines some namespace definitions around them for xmlns:ex="http://example.com;, and for tns (same URL). This is done by SchemaUtils.dfdlTestSchema(...) in daffodil-lib. It also provides xs and xsd,

JIRA tickets in "Cleanup" category - easy pickin'

2019-06-20 Thread Beckerle, Mike
The Cleanups category seems particularly easy. Many of these are easy to fix things that were just parked for expediency to avoid forgetting them, but then were forgotten in the backlog of 400+ tickets. I marked a bunch of these beginner label, because they involve no functional change they

Re: Character Encodings - No Statement

2019-06-24 Thread Beckerle, Mike
not notice the difference, but is this desireable? Additionally, it appears to not be simply a padding character. In my test data, I observed the string: "ADPADP". ________ From: Beckerle, Mike Sent: Tuesday, June 18, 2019 9:28:07 AM To: dev@daffodil.apache.org Su

Re: Character Encodings - No Statement

2019-06-18 Thread Beckerle, Mike
using the schema. Assuming we are correct that this character in intended as padding, we can probably avoid this issue by advising schema writers to specify U+A0 as a padding character, so it doesn't actually make it into the infoset. ____ From: Beckerle, Mike Sent: Mo

Re: Using enumerations for DFDL choices?

2019-05-09 Thread Beckerle, Mike
Keep 'em coming Chris. So yes, we're working hard on a enumerations feature for Daffodil that we need in products that have a similar need as yours. They are processing messages, which have lots of numeric enums defined, and we want to translate the enums to symbolic values that are then

Re: Differing behaviour between CLI and SAPI

2019-05-22 Thread Beckerle, Mike
Wild guess: you are not linking against the same daffodil jars that the CLI is using? From: Sloane, Brandon Sent: Wednesday, May 22, 2019 11:08:41 AM To: dev@daffodil.apache.org Subject: Differing behaviour between CLI and SAPI I am having an issue where

Re: Differing behaviour between CLI and SAPI

2019-05-22 Thread Beckerle, Mike
_____ From: Beckerle, Mike Sent: Wednesday, May 22, 2019 11:20:18 AM To: dev@daffodil.apache.org Subject: Re: Differing behaviour between CLI and SAPI Wild guess: you are not linking against the same daffodil jars that the CLI is using? From: Sloane, Brandon Sent:

Re: Schema Definition Error vs Processing Error for empty choiceDispatchKey

2019-05-20 Thread Beckerle, Mike
Nevermind that response. I thought we were still on the changes for a default branch. As for clarification that choiceDispatchKey="" is an SDE, I think that we should just assume this is SDE. That is the conservative design choice. ____ From: Beckerle,

Re: Adding character encoding?

2019-05-20 Thread Beckerle, Mike
The encodings are in daffodil-io, and for these small charsets it's very very easy to support them. There is already a 6 bit charset for DFI 264 DUI 001, and also some 5-bit encodings. All this was motivated by Link16. Take a look at file X_DFDL_6_BIT_DFI_264_DUI_001.scala

Re: Schema Definition Error vs Processing Error for empty choiceDispatchKey

2019-05-20 Thread Beckerle, Mike
at choiceDispatchKey="" is an SDE, I think that we should just assume this is SDE. That is the conservative design choice. ____ From: Beckerle, Mike Sent: Monday, May 20, 2019 11:04:39 AM To: dev@daffodil.apache.org Subject: Re: Schema Definition Error

Re: New commit to enumeration PR

2019-05-20 Thread Beckerle, Mike
No objections from me. From: Sloane, Brandon Sent: Monday, May 20, 2019 11:23:37 AM To: dev@daffodil.apache.org Subject: New commit to enumeration PR I know that this PR has already been reviewed, but when I rebased it back onto the latest version of master, I

Scala Code Formatting and Eclipse

2019-05-21 Thread Beckerle, Mike
Recent code reviews have turned up lots of spurious whitespace changes in my code. I have finally deciphered the issue so am sharing here to save others the trouble. So I use the scalaIDE for most development. This is just a bundle of stuff in Eclipse. I recently discovered that the scala

Re: Scala Code Formatting and Eclipse

2019-05-21 Thread Beckerle, Mike
guidelines. I've also created DAFFODIL-2133 to enable the sbt-sclariform plugin so that we can have automated checks for style during builds, and could also give guidance to people on how to configure their IDE on what options should be set. - Steve On 5/21/19 6:57 PM, Beckerle, Mike wrote

What is status on PR for typeCalc and enums?

2019-05-16 Thread Beckerle, Mike
This PR has been sitting for a while with comments saying "not done reviewing". I approved it, though one such comment is mine. It has one approval, one not done by SL. Brandon, can you comment on whether you are ok with this being merged or have pending changes? This is an important

Re: What is status on PR for typeCalc and enums?

2019-05-16 Thread Beckerle, Mike
are good there. I believe I addressed all of Josh's comments as well, but his review seems to have disappeared (likely due to a force-push on my end) From: Beckerle, Mike Sent: Thursday, May 16, 2019 2:01:29 PM To: dev@daffodil.apache.org Subject: What is status

Review attention needed

2019-05-16 Thread Beckerle, Mike
In case people didn't see it, the PR for the big (freakin' huge) change set for the sequences-and-separators change is now current and pending reviews. It's 100+ files touched, though I think well motivated. I had previously put out this PR, but was "not for merging". This one is now "the

Re: Adding character encoding?

2019-05-20 Thread Beckerle, Mike
with this usecase. From: Beckerle, Mike Sent: Monday, May 20, 2019 12:57:41 PM To: dev@daffodil.apache.org Subject: Re: Adding character encoding? The encodings are in daffodil-io, and for these small charsets it's very very easy to support them. There is already

Re: choiceLengthKind='explicit'

2019-05-06 Thread Beckerle, Mike
I think your interpretation is correct. It may be possible to detect an error if the branch is all fixed length stuff and is longer or shorter than the choiceLength. In fact, I think there is a DFDL language issue here. It's not really a bug as much as a missing feature. I think there should

Re: choiceLengthKind='explicit'

2019-05-07 Thread Beckerle, Mike
kipped and silently ignored. Maybe something like dfdl:unusedRegionPolicy with possible values if "allow", "error", "report", etc.. - Steve On 5/6/19 2:21 PM, Beckerle, Mike wrote: > I think your interpretation is correct. > > > It may be possible to detect

FYI: progress report on separated sequences and separator suppression

2019-05-03 Thread Beckerle, Mike
This email is FYI only. You can skip it unless you care about this specific development topic. I have made quite a lot of progress this past week so I thought it worth reporting about given that fixing these issues so that EDIFACT and TLOG can work has been delayed for so long. There are

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

2019-07-03 Thread Beckerle, Mike
The link for the list of tickets isn't correct. It opens up one specific ticket for me. The correct URL is https://issues.apache.org/jira/issues/?filter=-1=project%20%3D%20DAFFODIL%20AND%20fixVersion%20%3D%202.4.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

2019-07-03 Thread Beckerle, Mike
your link looks like it requires a log-in. I've created an apache short url, does this look correct: https://s.apache.org/daffodil-issues-2.4.0 On 7/3/19 10:15 AM, Beckerle, Mike wrote: > The link for the list of tickets isn't correct. It opens up one specific > ticket for me. The co

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

2019-07-03 Thread Beckerle, Mike
I have tested existing schemas to be sure we haven't broken anything. Alas, I have an iCalendar schema that works in 2.2.0, and 2.3.0, but fails to compile on 2.4.0 with this diagnostic: [error] Test com.mitre.iCalendar.Test_iCalendar.test_ex02 failed:

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

2019-07-03 Thread Beckerle, Mike
in the schema, assuming the DFDL spec is right. - Steve On 7/3/19 12:00 PM, Beckerle, Mike wrote: > > I have tested existing schemas to be sure we haven't broken anything. > > > Alas, I have an iCalendar schema that works in 2.2.0, and 2.3.0, but fails to > compile on 2.4.0 wi

Re: Justification for seperate DPathCompileInfo object

2019-06-27 Thread Beckerle, Mike
Some things are passed by name because they are, by definition circular dependencies - a DPathCompileInfo points to its parent, which points to its child. Those linkages are created by way of the common lazy and call-by-name idiom that lets objects be allocated before all the args to the

Re: Justification for seperate DPathCompileInfo object

2019-06-27 Thread Beckerle, Mike
hose will still cause coupling/circular bugs. Those could be decoupled by following the pattern used by the RuntimeData objects and passing all their args by name. I don't know of a better way to do this in Scala. -mike From: Beckerle, Mike Sent: Thursday, June 27, 201

Re: SDE during pre-serialization

2019-06-26 Thread Beckerle, Mike
So you are trying to not compute the things that go into the slots of the ExpressionTypeCalculator at the time the object is constructed, but later when they are demanded. Since this is part of the schema compiler, one way to snag the exceptions is to make sure that an OOLAG LV is used to do

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

2019-07-03 Thread Beckerle, Mike
From: Beckerle, Mike Sent: Wednesday, July 3, 2019 12:11:06 PM To: dev@daffodil.apache.org Subject: Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1 Ah! yes I will correct the schema. That property isn't allowed there, and prior releases were just ignoring that. I also

Re: Understanding bitOrder=leastSignificantFirst

2019-04-22 Thread Beckerle, Mike
Your test cant work. Bit order doesnt work that way. Unless a partial byte is involved bit order does nothing. The values of whole bytes don't change. There is a bit order tutorial in the daffodil tutorials module. Suggest you start there. If you view the bitOrder tdml.xml file with firefox it

Re: [DRAFT] Daffodil Podling Report - September 2019

2019-08-30 Thread Beckerle, Mike
+1 from me. Do you want to add that we're participating at ApacheCon NA again, doing both a talk and the shark-tank as part of our community building efforts. From: Steve Lawrence Sent: Friday, August 30, 2019 8:58 AM To: dev@daffodil.apache.org Subject:

Java Jigsaw Modules

2019-09-04 Thread Beckerle, Mike
I learned recently of this new (to me) Java modules system code name Jigsaw, which is part of Java 9 and beyond. This may be helpful to us in isolating dependencies so that Daffodil can export only its own API, and not force users to align all their library jars with the same versions Daffodil

Re: [GENERATION] Code generation with Daffodil

2019-09-11 Thread Beckerle, Mike
Conceptually about right. Daffodil has 3 trees in the compiler First is the DSOM tree. This is the AST. Its structure matches that of the schema document (s). Second is the "Gram" or grammar tree. (So called because we tried to make them work sort of like Scala's combinator parser "rules" for

Re: Easy test to start

2019-09-11 Thread Beckerle, Mike
Hi Julian, Within the DFDL tree, probably the simplest test that runs end-to-end, search for this line: @Test def test_unsignedInt() { runner.runOneTest("TestUnsignedInt") } This will compile a schema, then use it to parse a trivial unsigned text integer, but you can walk through the code

Re: [DISCUSS] User Defined Functions Capability

2019-09-06 Thread Beckerle, Mike
lass path only during testing. Daffodil should be able to find them > just like it would find any other jar on the classpath, allowing us to > fully exercise UDF capabilities (including broken UDFs) without > polluting daffodil proper. > > On 9/6/19 11:31 AM, Beckerle, Mike wrote: >>

Re: [GENERATION] Code generation with Daffodil

2019-09-12 Thread Beckerle, Mike
bit about that... From: Beckerle, Mike Sent: Wednesday, September 11, 2019 10:51:39 AM To: dev@daffodil.apache.org Subject: Re: [GENERATION] Code generation with Daffodil Conceptually about right. Daffodil has 3 trees in the compiler

Re: [GENERATION] Code generation with Daffodil

2019-09-13 Thread Beckerle, Mike
I have been looking at the code branch Julian F created (while we were at ApacheCon NA 2019) based on modifying the BinaryIntegerKnownLengthParser class to have a code-generator. Previously I thought implementing this on the parser/unparser classes was ok, but having refreshed my knowledge of

Re: [DISCUSS] User Defined Functions Capability

2019-09-06 Thread Beckerle, Mike
re: Design-for-Test for UDF facility When this code gets to code review with tests one of the key things will be that we get decent diagnostics from this for all the things users can do wrong in creating the UDF jar, such as not having it be found, not having all the classes in it, not having

Re: [DISCUSS] User Defined Functions Capability

2019-09-06 Thread Beckerle, Mike
(I added comments to the wiki page, but doing the discussion on this list is probably better from Apache-process considerations: "If it didn't happen on the mailing list, it didn't happen.") The code seems to infer the argument types, but the examples all have explicit DPath cast functions

design note on changes for DAFFODIL-2192 fix

2019-11-05 Thread Beckerle, Mike
I have stopped using confluence wiki for design notes because well, it converts everything into wiki pages that aren't particularly maintainable. I am trying asciidoc. The attached file is an asciidoc design note with diagrams that are automatically rendered. Get the asciidoc plugin for

Re: design note on changes for DAFFODIL-2192 fix

2019-11-05 Thread Beckerle, Mike
There is also a slick asciidoc mode for Eclipse. You get side by side panels with source on left and rendering (including diagrams) on right. From: Beckerle, Mike Sent: Tuesday, November 5, 2019 8:37 PM To: dev@daffodil.apache.org Subject: design note on changes

updated design note: DAFFODIL-2192 fix: unparser next element resolver rewrite

2019-11-08 Thread Beckerle, Mike
See this PR: https://github.com/apache/incubator-daffodil-site/pull/12 This is just one file of asciidoc. You can access the file via checking out the branch: git fetch asf refs/pull/12/head:pr/asf/12 git checkout pr/asf/12 I'm striving to create docs that can eventually be modified into

Re: design note on changes for DAFFODIL-2192 fix

2019-11-06 Thread Beckerle, Mike
for the website as well as all the diagrams. Works nicely. Also IntelliJ has an asciidoctor view too. Chris Am 06.11.19, 03:24 schrieb "Beckerle, Mike" : There is also a slick asciidoc mode for Eclipse. You get side by side panels with source on left and rendering (including diagrams

Re: design note on changes for DAFFODIL-2192 fix

2019-11-06 Thread Beckerle, Mike
ccordingly if not. From: Steve Lawrence Sent: Wednesday, November 6, 2019 9:08 AM To: dev@daffodil.apache.org Subject: Re: design note on changes for DAFFODIL-2192 fix Sill missing. Maybe a mail server is removing it? On 11/6/19 8:32 AM, Beckerle, Mike wrote:

Re: 2.5.0 branch performance looks problematic currently (2019-12-02)

2019-12-03 Thread Beckerle, Mike
. Particularly if we are going to be focusing optimizing. It would be much easier to do so with the quick feedback on an integrated performance test-suite. From: Beckerle, Mike Sent: Monday, December 2, 2019 1:29 PM To: dev@daffodil.apache.org Subject: 2.5.0 branch

Re: DFDL spec "Known to Exist"

2019-12-09 Thread Beckerle, Mike
Brandon, you said "highlighted portion" but no highlighting came through. Can you mark it somehow that doesn't require highlighting. From: Sloane, Brandon Sent: Thursday, December 5, 2019 4:56 PM To: dev@daffodil.apache.org Subject: DFDL spec "Known to Exist"

Re: Developing against Scala 2.11

2019-12-17 Thread Beckerle, Mike
Try cleaning everything, separately compiling just the daffodil-macrolib module, then compile the rest. The need for this went away some time ago, not sure why but may have been associated with the 2.11 to 2.12 move. That's the theory I'm suggesting here anyway.

merge without squash - fix or leave it?

2019-12-11 Thread Beckerle, Mike
I neglected to squash the two commits together before merging the daffodil-2242-tunable branch, which is the standard for our workflow. Should I fix and force push, or just leave it? I.e., which is the greater sin, to not squash and litter the history, or force push to master?

Re: merge without squash - fix or leave it?

2019-12-12 Thread Beckerle, Mike
re more than one commit in > the PR. > > On 12/11/19 8:02 AM, Beckerle, Mike wrote: >> I neglected to squash the two commits together before merging the >> daffodil-2242-tunable branch, which is the standard for our workflow. >> >> Should I fix and force pus

Progress on schema compilation time

2019-12-05 Thread Beckerle, Mike
I have been working on DAFFODIL-1444 which is about improving schema compilation space/speed. I wanted to report on progress. Two schemas that take a long time to compile are EDIFACT and HL7-v2.7. I show below the improvement in schema compilation performance. I am comparing the current

Re: [DISCUSS] comm...@daffodil.apache.org

2019-10-21 Thread Beckerle, Mike
Reserving dev for topical discussions from developers seems wise. Robot-created emails should always be separable. +1 from me. From: Steve Lawrence Sent: Monday, October 21, 2019 10:16 AM To: dev@daffodil.apache.org Subject: Re: [DISCUSS]

Re: [GENERATION] Code generation with Daffodil

2019-10-21 Thread Beckerle, Mike
Mike, as I'm currenty travelling to Apache Con this week is already reserved with "Do awesome stuff and hang around with great people" so it would fit me will this week : ) When would it work for you? Tomorrow, Wednesday, Thursday? Julian Am 16.10.19, 16:01 schrieb "

Re: [GENERATION] Code generation with Daffodil

2019-10-16 Thread Beckerle, Mike
08.10.19, 20:50 schrieb "Beckerle, Mike" : So I have made some progress on restructuring this code-gen stuff to where the structure makes sense to me and takes advantage of the way Daffodil works. It's not functioning yet. I don't have enough facility with Julian's Expressi

2.5.0 branch performance looks problematic currently (2019-12-02)

2019-12-02 Thread Beckerle, Mike
Tresys ran some performance tests on 2.5.0 branch recently. Nacha is faster relative to baseline (baseline is release 2.0.0) Pretty much everything else looks substantially slower. Compared to the last prior performance report I have (from April 2019), things then were more mixed relative to

Re: Experience with the Daffodil USMTF schema?

2019-11-25 Thread Beckerle, Mike
As USMTF is a FOUO format, probably best to take this discussion over to daffodil-fouo-supp...@tresys.com, which is non-public. From: Rob Rose Sent: Monday, November 25, 2019 10:03 AM To: dev@daffodil.apache.org Subject: Experience with the Daffodil USMTF

Performance coding memo updated

2019-11-01 Thread Beckerle, Mike
We have a wiki memo: https://cwiki.apache.org/confluence/display/DAFFODIL/Coding+for+Performance which is about coding style for things like daffodil runtimes, which have to go fast. I added a section "Avoid Destructuring by Pattern in Assignments" which is a guidance that goes along with the

Re: [GENERATION] Code generation with Daffodil

2019-10-08 Thread Beckerle, Mike
ven javascript (I will definetly NOT do that one __ ). >> >> Julian >> >> Am 13.09.19, 18:55 schrieb "Christofer Dutz" : >> >> Hi Mike, >> >> You might want to have a look how we use the expressions in plc4x as I >> have a dedicated Parser an

  1   2   3   4   >