So, there's different kinds of users.
Few people are data generalists who have the complete world of data in their purview. E.g., people who deal with MIL messaging and have to cope with issues like bit order and characters smaller than a byte don't often have to deal with Cobol data with its packed and zoned decimal numbers, as is common in financial services applications. And vice versa, the world of financial services apps never has to deal with bit order or fields smaller than 1 byte (other than boolean flags as 1 bit) typically. So if you want to know whether you've covered properties for "everything", well there are very few properties in DFDL that aren't needed for *some* format that is common in some area of the world of data. So a suggestion is to come up with a short bullet list of characteristics of the data formats you are trying to cover properties for - e.g., what industries or areas of concern? Another way to size up coverage: take a look at these schemas on DFDLSchemas github site: ISO8583, NACHA, (credit card transaction related), IBM4690TLOG (point-of-sale terminal data), and EDIFACT (an EDI standard). We also badly need a Cobol example that exercises all those features. (packed/zoned, fixed-point scaling factors, choiceLengthKind 'explicit' etc. ) I wrote a blog post many years ago with a little cobol example in it. Maybe we can create an example from that: https://cboblog.typepad.com/cboblog/2008/07/dfdl-data-forma.html ________________________________ From: Costello, Roger L. <[email protected]> Sent: Friday, June 7, 2019 2:56:10 PM To: [email protected] Subject: Does my DFDL tutorial cover all the essential DFDL properties? Hello DFDL community, Below are the DFDL properties covered in my tutorial. Note that some properties have an enumerated set of allowable values; I typically cover some but not all the allowable values. At one point, Steve Lawrence says that I’ve covered only about 30% of all the properties specified in the DFDL specification. I have a question. But first, to best express my question let me first give you an analogy. The SGML specification (SGML is the precursor to XML) is very large, over 500 pages. However, most users use only a fraction of all the SGML standard; perhaps 20% of the features are sufficient for 90% of all users. Now to my question. The DFDL specification is large. It specifies many properties. However, I suspect that most users will use only a fraction of all the DFDL properties. Do you agree? Is the set of properties currently covered in my tutorial sufficient for 90% of the users? Stated another way, is the set of properties currently covered in my tutorial sufficient for parsing and unparsing 90% of all the data formats that most people will typically ever encounter? Or, am I missing some properties that will absolutely be needed by most users? If so, what are those properties, please? 1. representation (text, binary) 2. byteOrder (littleEndian, bigEndian) 3. binaryNumberRep 4. bitOrder (leastSignficantBitFirst, mostSignificantBitFirst) 5. alignment, alignmentUnits 6. hexBinary 7. outputNewLine 8. lengthKind (implicit, explicit, delimited, pattern) 9. separator (whitespace-separated list of separators) 10. separatorPosition (infix, prefix) 11. initiator 12. terminator (whitespace-separated list of terminators) 13. lengthKind (delimited, explicit, pattern) 14. lengthUnits (characters, bits, bytes) 15. lengthPattern 16. length (integer, expression) 17. hiddenGroupRef 18. inputValueCalc 19. outputValueCalc 20. occursCountKind (fixed, implicit, expression) 21. occursCount 22. DFDL expressions, using XPath functions 23. textTrimKind, textStringPadCharacter, textStringJustification 24. binaryCalendarRep, binaryCalendarEpoch 25. assert 26. checkConstraints 27. ref 28. nilValue 29. separatorSuppressionPolicy 30. encoding 31. choice: * discriminator * initiatedContent * choiceDispatchKey, choiceBranchKey * encodeDFDLEntities 32. defineFormat, format 33. defineEscapeScheme, 34. escapeScheme (escapeKind (escapeCharacter|escapeBlock), escapeCharacter, escapeEscapeCharacter, extraEscapedCharacters, generateEscapeBlock, escapeBlockStart, escapeBlockEnd) 35. escapeSchemeRef 36. defineVariable 37. textStandardDecimalSeparator 38. documentFinalTerminatorCanBeMissing 39. dollar symbol ($) denotes end-of-file 40. DFDL schema 41. parsing vs unparsing 42. 2-pass parsing 43. DFDL entities 44. Flags that may be set when running Daffodil (--Validate on|off|limited, -D, -l scala-xml) 45. DFDL vs ANTLR
