[GitHub] jadams-tresys commented on a change in pull request #173: Allow implicit conversion of string to HexBinary
jadams-tresys commented on a change in pull request #173: Allow implicit
conversion of string to HexBinary
URL: https://github.com/apache/incubator-daffodil/pull/173#discussion_r251588290
##
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/dpath/ConverterOps2.scala
##
@@ -78,3 +78,15 @@ case object StringToTime extends Converter {
result
}
}
+
+case object StringToHexBinary extends Converter with HexBinaryKind{
+ val name = "StringToHexBinary"
+
+ override def computeValue(a: AnyRef, dstate: DState): AnyRef = {
+val result = a match {
+ case s: String => hexStringToByteArray(s)
+ case x => throw new NumberFormatException("%s cannot be cast to
dfdl:hexBinary\ndfdl:hexBinary received an unrecognized type! Must be String or
HexBinary.".format(x.toString))
+}
+result
+ }
+}
Review comment:
I wasn't sure that it would, since the XSHexBinary computeValue also accepts
reading in a Byte array, so I thought it might be confusing if the
StringToHexBinary converter also allows reading in both a String or a Byte
array.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stevedlawrence commented on a change in pull request #173: Allow implicit conversion of string to HexBinary
stevedlawrence commented on a change in pull request #173: Allow implicit
conversion of string to HexBinary
URL: https://github.com/apache/incubator-daffodil/pull/173#discussion_r251584674
##
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/dpath/ConverterOps2.scala
##
@@ -78,3 +78,15 @@ case object StringToTime extends Converter {
result
}
}
+
+case object StringToHexBinary extends Converter with HexBinaryKind{
+ val name = "StringToHexBinary"
+
+ override def computeValue(a: AnyRef, dstate: DState): AnyRef = {
+val result = a match {
+ case s: String => hexStringToByteArray(s)
+ case x => throw new NumberFormatException("%s cannot be cast to
dfdl:hexBinary\ndfdl:hexBinary received an unrecognized type! Must be String or
HexBinary.".format(x.toString))
+}
+result
+ }
+}
Review comment:
Does it make sense to modify XSHexBinary to call this function now so as to
minimize duplicate code? So XSHexBInary basically just calls
StringToHexBinary.computeValue(...), and all the actual conversion logic is
only in this object? This looks like how we do other conversions like XSTime
vs StringToTime
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ltapilot commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
ltapilot commented on a change in pull request #170: Fixed CLI integration
tests so that they run properly on Windows.
URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251569553
##
File path:
daffodil-cli/src/it/scala/org/apache/daffodil/tunables/TestCLITunables2.scala
##
@@ -58,7 +58,7 @@ class TestCLITunables2 {
try {
// note: 2>&1 is shell-speak for "connect stderr into stdout"
- val cmd = String.format("""echo "a,b" | %s parse -s %s 2>&1""",
Util.binPath, testSchemaFile)
+ val cmd = String.format("""echo a,b| %s parse -s %s 2>&1""",
Util.binPath, testSchemaFile)
Review comment:
As stated, above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ltapilot commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
ltapilot commented on a change in pull request #170: Fixed CLI integration
tests so that they run properly on Windows.
URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251569241
##
File path:
daffodil-cli/src/it/scala/org/apache/daffodil/tunables/TestCLITunables2.scala
##
@@ -34,13 +34,13 @@ class TestCLITunables2 {
try {
// note: 2>&1 is shell-speak for "connect stderr into stdout"
- val cmd = String.format("""echo "a,b" | %s parse -s %s
-TsuppressSchemaDefinitionWarnings=all 2>&1""", Util.binPath, testSchemaFile)
+ val cmd = String.format("""echo a,b| %s parse -s %s
-TsuppressSchemaDefinitionWarnings=all 2>&1""", Util.binPath, testSchemaFile)
Review comment:
For the record, I concur with Steve. The 'Util.echoN' function is only
intended to cover the difference in 'echo' behavior
between Windows and Linux when the stated conditions are met.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] stevedlawrence commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
stevedlawrence commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows. URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251569006 ## File path: build.sbt ## @@ -139,6 +139,7 @@ lazy val commonSettings = Seq( sourceManaged := baseDirectory.value / "src_managed", resourceManaged := baseDirectory.value / "resource_managed", libraryDependencies ++= Dependencies.common, + parallelExecution in IntegrationTest := false Review comment: Seems reasonble in that case. The change only disables parallel executiong for integration tests (i.e. CLI tests), so it shouldn't affect the time it takes to run all our normal tests. As Josh mentions, we could disable/enable parallel execution based on the operating system, but I think I'd rather evertyhing be as similar as possible on different OS. So if it needs to be disabled on Windows, I'd vote for also disabling on Linux too. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ltapilot commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
ltapilot commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows. URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251567248 ## File path: build.sbt ## @@ -139,6 +139,7 @@ lazy val commonSettings = Seq( sourceManaged := baseDirectory.value / "src_managed", resourceManaged := baseDirectory.value / "resource_managed", libraryDependencies ++= Dependencies.common, + parallelExecution in IntegrationTest := false Review comment: I spent a couple of hours messing with this today. It turns out that the platform doesn't do a good enough job of predicting memory usage. Subsequently it starts too many tests in parallel and when the tests all ask for large blocks of RAM serious memory pressure builds at the system level. In my case the system even ran out of swap space. My system has 16GB of RAM and it is clear that this should still be able to run on systems with less than that. For these reasons I'm going to leave parallel execution OFF in this pull request. I wish it were not so, but I think that's the only way we're going to get this to run consistently and without failure. Sorry. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (DAFFODIL-2047) Create Windows MSI Installer
[ https://issues.apache.org/jira/browse/DAFFODIL-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark C. Otto reassigned DAFFODIL-2047: -- Assignee: Mark C. Otto > Create Windows MSI Installer > > > Key: DAFFODIL-2047 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2047 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Mark C. Otto >Priority: Major > > Windows users are usually more familiar with MSI installers rather than > dealing with zips. And the sbt-native-packager plugin we to create RPMs also > has the abilty to create MSI's. So we should modify our sbt config and > release script to create an MSI installer–could be a good way grow the > community beyond primiarly linux users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Release Apache Daffodil 2.3.0
I know of nothing that should be blocking the release. I closed PR 158. I will update our roadmap on the project wiki. From: Steve Lawrence Sent: Monday, January 28, 2019 7:55:55 AM To: dev@daffodil.apache.org Subject: [DISCUSS] Release Apache Daffodil 2.3.0 All, Alot of significant changes have been made since Daffodil 2.2.0, and it's been a while since we've had a release. I think it's time to get 2.3.0 out the door. There are currently 5 pull requests open: 172: Make XML comparisons more memory efficient 171: Added additional note to UPA violation errors 170: Fixed CLI integration tests so that they run properly on Windows. 158: Daffodil 1080 sequences and separators - preliminary review 133: Enabled a few Calendar tests that now work Of those, I think 158 and 133 are not ready to be merged, and perhaps should be closed, with new pull requests eventually added when updates are made? The remaining three look simple enough and will probably be merged this week. So, I propose that once those three PRs are merged, we start the release process for 2.3.0, unless anyone knows of any critical issues that should be resolved for the next release. I'll volunteer to be the release manager for 2.3.0. Thanks, - Steve
[GitHub] mbeckerle commented on issue #158: Daffodil 1080 sequences and separators - preliminary review
mbeckerle commented on issue #158: Daffodil 1080 sequences and separators - preliminary review URL: https://github.com/apache/incubator-daffodil/pull/158#issuecomment-458179666 Closing. This was never intended to be merged. It's just to gather comments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mbeckerle closed pull request #158: Daffodil 1080 sequences and separators - preliminary review
mbeckerle closed pull request #158: Daffodil 1080 sequences and separators - preliminary review URL: https://github.com/apache/incubator-daffodil/pull/158 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Defining Protocol State-Machines
Christofer, Your idea here sounds to me like a toolkit using on an XML-based language, for describing communicating state machines, or protocol handshakes. This is a good idea. From a systems-security perspective, creating these by starting with a Turing-complete programming language is unwise. See langsec.org for why etc. I don't know of such a kit, but I have not looked for one. if you search and find something that actually works when you try it out, please post back here or better,... on users list because many users of DFDL/Daffodil have this same issue. I think this does go outside the scope of the Daffodil project, which we need to keep narrowly focused on implementing DFDL, but I for one would be really interested in this. -Mike Beckerle Tresys From: Christofer Dutz Sent: Monday, January 28, 2019 9:07:11 AM To: dev@daffodil.apache.org Subject: Defining Protocol State-Machines Hi all, so now I seem to have managed describing the format of the messages of an industry protocol. Still I have to manually implement the protocol itself. I have to construct messages in a pre-defined format, send that to a remote station, wait for a response, extract data from that. Depending on the returned data, I construct other messages and send them again … I would think that it should be possible to sort of describe this in a universal way too. I’m sort of thinking of something like this: * I have some format (Preferably XML) where I define a state-machine for every protocol. * In this state machine I can attach message templates to the transitional-edges (similar to the XMl representation in the TDML) * In the states I define the options of what next edge I should take: * Sort of: * If “initial State” -> Send “COTP Connect” message -> Wait for Response -> if response/returnCode = 0xFF (all ok) -> Send “S7SetupConnectionRequest” message -> Wait For Response -> If … * If /S7Response/Parameter/ReturnCode == 0xFF -> Go to the “Initialized” state * If != 0xFF -> Go to some “Error handling” state So now my question: * Does Something like this already exist? * Would it make sense to define something here in the Daffodil project, or should I do that in PLC4X? Chris
Defining Protocol State-Machines
Hi all, so now I seem to have managed describing the format of the messages of an industry protocol. Still I have to manually implement the protocol itself. I have to construct messages in a pre-defined format, send that to a remote station, wait for a response, extract data from that. Depending on the returned data, I construct other messages and send them again … I would think that it should be possible to sort of describe this in a universal way too. I’m sort of thinking of something like this: * I have some format (Preferably XML) where I define a state-machine for every protocol. * In this state machine I can attach message templates to the transitional-edges (similar to the XMl representation in the TDML) * In the states I define the options of what next edge I should take: * Sort of: * If “initial State” -> Send “COTP Connect” message -> Wait for Response -> if response/returnCode = 0xFF (all ok) -> Send “S7SetupConnectionRequest” message -> Wait For Response -> If … * If /S7Response/Parameter/ReturnCode == 0xFF -> Go to the “Initialized” state * If != 0xFF -> Go to some “Error handling” state So now my question: * Does Something like this already exist? * Would it make sense to define something here in the Daffodil project, or should I do that in PLC4X? Chris
[jira] [Resolved] (DAFFODIL-2059) Computing the diff of large XML results in a stack overflow
[
https://issues.apache.org/jira/browse/DAFFODIL-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2059.
--
Resolution: Fixed
Fixed in commit cdf956b113e1eea2d7eddac87ffb11e66b872a41
> Computing the diff of large XML results in a stack overflow
> ---
>
> Key: DAFFODIL-2059
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2059
> Project: Daffodil
> Issue Type: Bug
>Reporter: Steve Lawrence
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 2.3.0
>
>
> We changed how we compare expected and actual infosets to always use
> computeDiffOne. However, for very large XML it results in the following stack
> overflow:
> {code}
> [error] at java.lang.String.valueOf(String.java:2994)
> [error] at
> scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
> [error] at
> scala.collection.TraversableOnce.$anonfun$addString$1(TraversableOnce.scala:359)
> [error] at scala.collection.immutable.List.foreach(List.scala:389)
> [error] at
> scala.collection.TraversableOnce.addString(TraversableOnce.scala:357)
> [error] at
> scala.collection.TraversableOnce.addString$(TraversableOnce.scala:353)
> [error] at
> scala.collection.AbstractTraversable.addString(Traversable.scala:104)
> [error] at
> scala.collection.TraversableOnce.mkString(TraversableOnce.scala:323)
> [error] at
> scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:322)
> [error] at
> scala.collection.AbstractTraversable.mkString(Traversable.scala:104)
> [error] at
> scala.collection.TraversableOnce.mkString(TraversableOnce.scala:325)
> [error] at
> scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:325)
> [error] at
> scala.collection.AbstractTraversable.mkString(Traversable.scala:104)
> [error] at
> org.apache.daffodil.xml.XMLUtils$.zPath$lzycompute$1(XMLUtils.scala:784)
> [error] at org.apache.daffodil.xml.XMLUtils$.zPath$1(XMLUtils.scala:784)
> [error] at
> org.apache.daffodil.xml.XMLUtils$.computeDiffOne(XMLUtils.scala:833)
> [error] at
> org.apache.daffodil.xml.XMLUtils$.computeDiffOne(XMLUtils.scala:825)
> [error] at
> org.apache.daffodil.xml.XMLUtils$.computeDiffOne(XMLUtils.scala:826)
> [error] at
> org.apache.daffodil.xml.XMLUtils$.computeDiffOne(XMLUtils.scala:826)
> ...
> {code}
> This causes tests on existing real world schemas to fail during TDML testing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] stevedlawrence merged pull request #172: Make XML comparisions more memory efficient
stevedlawrence merged pull request #172: Make XML comparisions more memory efficient URL: https://github.com/apache/incubator-daffodil/pull/172 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stevedlawrence commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
stevedlawrence commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows. URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251411702 ## File path: build.sbt ## @@ -139,6 +139,7 @@ lazy val commonSettings = Seq( sourceManaged := baseDirectory.value / "src_managed", resourceManaged := baseDirectory.value / "resource_managed", libraryDependencies ++= Dependencies.common, + parallelExecution in IntegrationTest := false Review comment: Agreed, I'm not even sure this change is needed at all. I know Mark was running into memory issues when runing on windows and this was one change he made early on. But I'm not sure it really helped. I think the actual fix was changes to DAFFODIL_JAVA_OPTS in TestCLIDebugger. Mark, can you remove this change and see if the memory issues come back? If not, I think we should remove this change completely. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (DAFFODIL-2036) poor diagnostic - choice with sequences as branches with all optional content.
[
https://issues.apache.org/jira/browse/DAFFODIL-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Adams resolved DAFFODIL-2036.
--
Resolution: Fixed
Assignee: Dave Thompson (was: Josh Adams)
Fix Version/s: 2.3.0
This has been resolved in commit dfc40612d27eb891bb8bcad7fc2c653413187d78
> poor diagnostic - choice with sequences as branches with all optional content.
> --
>
> Key: DAFFODIL-2036
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2036
> Project: Daffodil
> Issue Type: Bug
> Components: Middle End
>Affects Versions: 2.2.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> Suppose you have this choice:
>
>
>
>
>
>
>
>
>
> You get an SDE with this message:
> Schema Definition Error: UPA violation. Multiple choice branches begin with
> {http://example.com}foo
> Note that elements with dfdl:outputValueCalc cannot be used to distinguish
> choice branches.
> The offending choice branches are:
> sequence[1] at Location in file:/tmp/s1_5963264522094411444.dfdl.xsd
> sequence[2] at Location in file:/tmp/s1_5963264522094411444.dfdl.xsd
> This occurs because the two branches of the choice meet the letter of the law
> by bing sequences which are considered "required", but as their content is
> all optional and they have no framing, they are conceptually optional.
> So there's really no way to look at the infoset and decide which arm of the
> choice.
> The error here should be that the content of a choice branch is entirely
> optional which isn't allowed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] jadams-tresys merged pull request #171: Added additional note to UPA violation errors
jadams-tresys merged pull request #171: Added additional note to UPA violation errors URL: https://github.com/apache/incubator-daffodil/pull/171 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[DISCUSS] Release Apache Daffodil 2.3.0
All, Alot of significant changes have been made since Daffodil 2.2.0, and it's been a while since we've had a release. I think it's time to get 2.3.0 out the door. There are currently 5 pull requests open: 172: Make XML comparisons more memory efficient 171: Added additional note to UPA violation errors 170: Fixed CLI integration tests so that they run properly on Windows. 158: Daffodil 1080 sequences and separators - preliminary review 133: Enabled a few Calendar tests that now work Of those, I think 158 and 133 are not ready to be merged, and perhaps should be closed, with new pull requests eventually added when updates are made? The remaining three look simple enough and will probably be merged this week. So, I propose that once those three PRs are merged, we start the release process for 2.3.0, unless anyone knows of any critical issues that should be resolved for the next release. I'll volunteer to be the release manager for 2.3.0. Thanks, - Steve
[GitHub] stevedlawrence commented on a change in pull request #170: Fixed CLI integration tests so that they run properly on Windows.
stevedlawrence commented on a change in pull request #170: Fixed CLI
integration tests so that they run properly on Windows.
URL: https://github.com/apache/incubator-daffodil/pull/170#discussion_r251393696
##
File path:
daffodil-cli/src/it/scala/org/apache/daffodil/tunables/TestCLITunables2.scala
##
@@ -34,13 +34,13 @@ class TestCLITunables2 {
try {
// note: 2>&1 is shell-speak for "connect stderr into stdout"
- val cmd = String.format("""echo "a,b" | %s parse -s %s
-TsuppressSchemaDefinitionWarnings=all 2>&1""", Util.binPath, testSchemaFile)
+ val cmd = String.format("""echo a,b| %s parse -s %s
-TsuppressSchemaDefinitionWarnings=all 2>&1""", Util.binPath, testSchemaFile)
Review comment:
The new ``Util.echoN`` function is actually only needed when the ``-n`` flag
is used. The echo command behaves exactly the same on linux in windows if the
following conditions are met:
1. No flags are provided (i.e. ``-n``)
1. No quotes are needed
1. No spaces exists at the end of the echo string before the pipe
If these are all true, then the only difference between windows and linux
echo is CRLF vs LF, which the schemes in most of these tests handle with
``%NL;``. In this particular test, we needed to remove the quotes and the space
before the pipe to make it work on linux and windows, but the schema is fine
with the newline differences. In cases where the ``-n`` flag was used for
linux, the new ``Util.echoN`` function was created, which creates the right
command to echo without a newline at the end based on the operating system. So
only tests that do not expect a ``%NL;`` need use use the new function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
