Sorry, here is a direct link to the thread https://lists.apache.org/thread/rkryds030qyt5c2lq95g5mrnc87cp010.
MARSHALL MILLER Principal Software Security Engineer marshall.mil...@sealingtech.com<mailto:marshall.mil...@sealingtech.com> SealingTech | A Parsons Company 6750 Alexander Bell Drive, Suite, 200 Columbia, MD 21046 www.sealingtech.com<http://www.sealingtech.com/?utm_source=outlook.com&utm_medium=signature_link&utm_content=associate_email> Join us on social media | #BuildSecureSolve Facebook<https://www.facebook.com/sealingtech/> | LinkedIn<https://www.linkedin.com/company/sealing-technologies> | Twitter<https://twitter.com/sealing_tech> NOTICE: This message may contain privileged information intended solely for the use of the intended recipient. If you are not the intended recipient, please contact the sender of this message immediately. Any dissemination, distribution, copying, or other use of information in this message is prohibited. From: Marshall Miller <marshall.mil...@sealingtech.com> Date: Friday, December 8, 2023 at 10:04 AM To: users@daffodil.apache.org <users@daffodil.apache.org> Subject: Re: Bits are streamed into an application ... the bits are put in memory ... is the first bit received at the lowest memory address? When discussing Link-16 it is also important to be aware of exactly which “flavor” you are referring to. The original standard is based on RF transmission and there are many different standards that extend and modify it to support different forms of communication. A conversation about one such variant commonly called SISO-J can be found at https://lists.apache.org/list?d...@daffodil.apache.org:2018-1. MARSHALL MILLER Principal Software Security Engineer marshall.mil...@sealingtech.com<mailto:marshall.mil...@sealingtech.com> SealingTech | A Parsons Company 6750 Alexander Bell Drive, Suite, 200 Columbia, MD 21046 www.sealingtech.com<http://www.sealingtech.com/?utm_source=outlook.com&utm_medium=signature_link&utm_content=associate_email> Join us on social media | #BuildSecureSolve Facebook<https://www.facebook.com/sealingtech/> | LinkedIn<https://www.linkedin.com/company/sealing-technologies> | Twitter<https://twitter.com/sealing_tech> NOTICE: This message may contain privileged information intended solely for the use of the intended recipient. If you are not the intended recipient, please contact the sender of this message immediately. Any dissemination, distribution, copying, or other use of information in this message is prohibited. From: Steve Lawrence <slawre...@apache.org> Date: Friday, December 8, 2023 at 8:55 AM To: users@daffodil.apache.org <users@daffodil.apache.org> Subject: Re: Bits are streamed into an application ... the bits are put in memory ... is the first bit received at the lowest memory address? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Yep! That's the right idea. On 2023-12-08 08:46 AM, Roger L Costello wrote: > I am pretty certain that I will continue using a hex editor to create > binary documents. > > So, I need a simple, step-by-step procedure for creating binary > documents that have a bit order = leastSignificantBitFirst. > > Below is the procedure that I’ve devised. I would appreciate it if you > would let me know if it is correct or incorrect. > > Step 1: Write on a piece of paper the bits that are to go into the > document. If the first field of your data format is two bits and the > field is to have the decimal value 2 (binary 10), then on the paper > write 10 on the right side: > > If the next field is five bits and the field is to have the decimal > value 9 (binary 01001), then write 01001 to the left of the first field: > > If the next field is three bits and the field is to have the decimal > value 7 (binary 111), then write 111 to the left of the second field: > > Proceed in the same manner for the remaining fields. Now we have the > first field on the right side and the last field on the left side: > > Step 2: Starting from the right side moving leftward, add a space every > four bits: > > Step 3: Starting from the right side moving leftward, add a couple more > spaces every eight bits: > > Step 4: Under each four-bit group, show the group’s hex value: > > Step 5: Label the rightmost pair of hex digits Byte 1, the pair of hex > digits to its left Byte 2, and so forth: > > Step 6: Open a hex editor. At address (offset) 00 type in the hex pair > for Byte 1: > > Step 7: At address (offset) 01 type in the hex pair for Byte 2: > > Step 8: At address (offset) 02 type in the hex pair for Byte 3: > > Step 9: Proceed in the same manner for the remaining bytes. The hex > editor displays the first byte on the left side and the last byte on the > right side (just the opposite of how the bytes appear on paper!): > > Step 10: Save the hex editor content to a file. > > TaDa! > > You have created a binary document where the bit order is > leastSignificantBitFirst. > > *From:*Roger L Costello <coste...@mitre.org> > *Sent:* Friday, December 8, 2023 4:04 AM > *To:* users@daffodil.apache.org > *Subject:* Re: Bits are streamed into an application ... the bits are > put in memory ... is the first bit received at the lowest memory address? > > Steve, Mike, > > Your explanations of this bit order stuff are exceptional. I mean > really, really exceptional. > > * hex editors which don't understand the idea of right-to-left data > > Are there hex editors that do understand the idea of right-to-left data? > > * I believe the data viewer in the Daffodil VS Code extension is > planning (or already has?) the ability to show bytes in the reverse > order at bit resolution > > I will look into this. > > /Roger > > *From:*Steve Lawrence <slawre...@apache.org <mailto:slawre...@apache.org>> > *Sent:* Thursday, December 7, 2023 3:28 PM > *To:* users@daffodil.apache.org <mailto:users@daffodil.apache.org> > *Subject:* Re: Bits are streamed into an application ... the bits are > put in memory ... is the first bit received at the lowest memory address? > > Lin16/VMF is definitely confusing. A hex editor will actually display > the data like this: 4E .. . . . 21 Another way to think about how to > read link16/vmf data (which is more useful when using hex editors which > don't understand the idea of right-to-left > > ZjQcmQRYFpfptBannerEnd > > Lin16/VMF is definitely confusing. A hex editor will actually display > > the data like this: > > 4E ..... 21 > > Another way to think about how to read link16/vmf data (which is more > > useful when using hex editors which don't understand the idea of > > right-to-left data), is to read each *byte* left-to-right, but read the > > *bits within each byte* right-to-left. So if you look at your example > > data in a hex editor you'll see something like this > > low memory address high memory address > > 4 E . . . 2 1 > > 0100 1110 ........ ........ ........ 0010 0001 > > If we read the bytes left-to-right like a hex editor would show, our > > first byte is 01001110 (0x4E). But within that byte, we read the bits > > right-to-left, which means our first 2-bit field would be "10" (taken > > from the right of the 0xE, not left of the 0x4). As an further example, > > if there was a 3-bit field following our 2-bit field, it would be 011. > > Note that once we consume the entire first byte, we start reading from > > the right-most bit of the 2nd byte, and so on. This can make things > > tricky when fields overlap byte boundaries (which is very common in > > VMF/Link16). For example, let's say we've already consumed 5 bits in the > > first byte (marked in x's below), and the next field is 6-bits long > > (marked from 1-6 in the order to read the bits below), this is where the > > bits would come from for that field: > > 321x xxxx .... .654 > > So this field crosses a byte boundary, and some bits come from the left > > of the first byte and others from the right of the second byte. In this > > example, the the value of this 6-bit field 654321. > > It can definitely be confusing to read *bytes* left-to-right but *bits* > > right-to-left. So it is sometimes helpful to mentally (or even manually) > > reorder all the bytes so they are reversed (without reversing the bits), > > and then just read the bits from the end of the file, right-to-left and > > bottom-to-top. > > In fact, I believe the data viewer in the Daffodil VS Code extension is > > planning (or already has?) the ability to show bytes in the reverse > > order at bit resolution, which would make this a little easier so you > > wouldn't have to do it yourself. > > On 2023-12-07 02:38 PM, Roger L Costello wrote: > >> Thanks Steve. > >> > >> As it turns out, I am working with the Link-16 data format. You are > >> saying that the stream of received bits – 0 then 1 then 1 then 1 then 0 > >> then 0 then 1 then 0 then … - are viewed in this fashion: > >> > >> High memory address Low memory address > >> > >> 0010 0001 ................................................ 0100 1110 > >> > >> Hex: 21 ……………………………………. 4E > >> > >> That is consistent with the Link-16 specification. > >> > >> Here’s where I get confused. Suppose I store those bits into a file. > >> Then I open the file in a hex editor. How will the hex editor display > >> the data? > >> > >> Will the hex editor display the four bits at the lowest memory address > >> (1110) in hex digit form (E), followed by the four bits at the > >> next-to-lowest memory address (0100) in hex digit form (4): > >> > >> E4 ……………………………… 12 > >> > >> Or will the hex editor see the bits in reverse order: > >> > >> Low memory address High memory address > >> > >> 0111 0010 ………………………………………… 1000 0100 > >> > >> Hex: 72 ………………………………………………. 84 > >> > >> And display this: > >> > >> 72 ………………………………………….. 84 > >> > >> That is wildly different! > >> > >> Eek! I am so confused! Help! > >> > >> *From:*Steve Lawrence <slawre...@apache.org <mailto:slawre...@apache.org>> > >> *Sent:* Thursday, December 7, 2023 1:16 PM > >> *To:* users@daffodil.apache.org <mailto:users@daffodil.apache.org> > >> *Subject:* [EXT] Re: Bits are streamed into an application ... the bits > >> are put in memory ... is the first bit received at the lowest memory > >> address? > >> > >> The most likely interpretation is that the 2-bit field will have a value > >> of "01", with a decimal value of 1. This is what DFDL calls dfdl: > >> bitOrder="mostSignificantBitFirst" and is what most data formats use. > >> DFDL also has dfdl: bitOrder="leastSignificantBitFirst", > >> > >> The most likely interpretation is that the 2-bit field will have a value > >> > >> of "01", with a decimal value of 1. This is what DFDL calls > >> > >> dfdl:bitOrder="mostSignificantBitFirst" and is what most data formats use. > >> > >> DFDL also has dfdl:bitOrder="leastSignificantBitFirst", which is common > >> > >> in some old military formats like VMF and Link16. One way to imagine > >> > >> this is as if the memory addresses and bits were ordered and read > >> > >> right-to-left. So, your example would look like this: > >> > >> High memory address Low memory address > >> > >> 0 0 ................................................ 1 1 0 > >> > >> So we have the same data, and you still read the low memory address bits > >> > >> first, it's just all backwards. Because we read right-to-left, the first > >> > >> two bits are now "10". Note that interpreting the value of the bits is > >> > >> the same as normal, so "10" evaluates to 2--all that changes is the > >> > >> order in which we read bits. > >> > >> I haven't seen any formats where the high memory address bits would be > >> > >> read first and would lead to the 2-bit field being "00". DFDL doesn't > >> > >> have a way to model this. > >> > >> On 2023-12-07 12:28 PM, Roger L Costello wrote: > >> > >>> Hi Folks, > >> > >>> > >> > >>> A basic question about bits. > >> > >>> > >> > >>> Scenario: an application is receiving a message. The message arrives as a >>> stream of bits. The first bit received is 0. The second bit received is 1. >>> The third bit received is another 1. ... The second-to-last bit received is >>> 0. The last bit received is 0. > >> > >>> > >> > >>> The application stores the message in memory. > >> > >>> > >> > >>> Will the first bit received be in the lowest memory address and the last >>> bit received in the highest memory address? I.e., > >> > >>> > >> > >>> Low memory address High memory address > >> > >>> 0 1 1 ............................................................. 0 0 > >> > >>> > >> > >>> The message contains a series of bit fields. The specification for the >>> message says the first field is two bits. > >> > >>> > >> > >>> Is the first field the bits 0 1 or the bits 00? > >> > >>> > >> > >>> /Roger > >> >
