Thanks John and Don. Speeding up parsing by outputting EXI is an interesting idea. I guess the theory behind it is: If it takes a long time to output a huge text XML file, then instead output a much smaller binary EXI file. Interesting idea!
Okay, I gave it a go. Here are the results: Parsing to a huge text XML file took 226 seconds. Parsing to a binary EXI file took 206 seconds. Parsing to EXI was a little faster. Not a big difference. The difference in the size of the output was remarkable: The size of the text XML file is 4 GB. The size of the EXI file is 287 MB. /Roger From: Interrante, John A (GE Aerospace, US) <john.interra...@ge.com> Sent: Tuesday, December 26, 2023 5:08 PM To: users@daffodil.apache.org Subject: [EXT] RE: Parsing 5 million lines of input is taking 4 minutes - too slow! Hi Roger, If you are using Daffodil 3.4.0 or later (3.6.0 would be better), then the Daffodil CLI has two new infoset types (-I exi and -I exisa) which will output non-schema aware EXI infosets and schema aware EXI infosets, respectively. EXI (binary XML) infosets are significantly smaller in size than normal XML infosets, often even smaller than the original data format when made schema aware. Daffodil uses the Exificient library to support these infoset types. Try adding -I exi or -I exisa to your “daffodil parse” command to see how much it can speed up your parsing. You will not end up with a 4GB XML infoset file; you will end up with hopefully a lot smaller EXI infoset file which can be used in place of the XML infoset depending on how the end user application reads the infoset (the end user application also needs to use an EXI library). John From: Roger L Costello <coste...@mitre.org<mailto:coste...@mitre.org>> Sent: Tuesday, December 26, 2023 11:04 AM To: users@daffodil.apache.org<mailto:users@daffodil.apache.org> Subject: EXT: Parsing 5 million lines of input is taking 4 minutes - too slow! Hi Folks, My input file contains 5 million 132-character records. I have done everything that I can think of to make the parsing faster: 1. I precompiled the schema and used it to do the parsing 2. I set Java -Xmx40960m 3. I used a bunch of dfdl:choiceDispatchKey to divide-and-conquer And yet it still takes 4 minutes before the (4 GB) XML file is produced. Waiting 4 minutes is not acceptable for my clients. A couple of questions: 1. Is there anything else that I can do to speed things up? 2. I believe there is time needed to do the parsing and generate an in-memory parse tree, and there is time needed to serialize the in-memory parse tree to an XML file. Is there a way to find those two times? I suspect the former is a lot quicker than the latter. /Roger
