Hi Martin, I heed your suggestions and I think I am very close. The QA works perfectly with various message lengths ranging from 3 bytes to 30 bytes and various spurious byte lengths before the preamble (which is 4 bytes). The spurious samples range from 4 to 9 bytes. Sadly, the OOT implemented block did not work as I expected it to. In the GRC system, I implemented the spurious bytes by using a Gnuradio Delay block which inputs zeros ahead of the start of the preamble with subsequent data bytes following it. .
When I use a delay of 5 units it produces 4 zeros before the preamble and with a delay of 16, it again showed 4 zeros before the preamble. I will need to run more tests at different delays to verify consistency before trying any fix. It would have been easier if the QA test failed, then I could debug by single stepping through the code to determine what is wrong. Breaking at the in-system block level is more difficult to fix. Have a great evening! Best Regards, George On Tue, Sep 15, 2020 at 5:46 AM George Edwards <[email protected]> wrote: > Thank you very much Martin! George > > On Mon, Sep 14, 2020, 11:28 PM Martin Luelf <[email protected]> wrote: > >> Dear George, >> >> please keep the list in CC so other people can read this discussion in >> the future. >> >> general_work can be called with any number of input parameters. This >> could be just 5, or 500 because it can buffer the repeated result from >> your repeating source. You can somewhat steer this with the forecast >> method by telling GNURadio that you need 100 inputs. In that case >> GNURadio will not call general_work with less than 100 inputs, but it >> might still call it with more than 100. Also for unittests that do not >> repeat keep in mind that GNURadio will silently drop any excess inputs >> that are less than the requested size. >> >> I suggest you put a print statement both in forecast and general_work to >> and then run your code multiple times (also under different loads) to >> get a better idea of what GNURadio is doing under the hood. >> >> Yours >> Martin >> >> On 15.09.20 06:28, George Edwards wrote: >> > Hi Martin, >> > >> > I was out of town for the weekend, so I am just getting at my email. >> > >> > Thanks for your detailed reply as usual. I really appreciate your help >> > and I know it is a lot of effort and time on your part so I really >> > appreciate it. >> > : >> > The param cnt works well and is used to synch up the indices for the >> > in and out arrays. For example, suppose the input data was >> > {1,2,3,0,1,3,2,5,6} and the preamble pattern is {3,0,1,3,2}. >> > Then after initialization, cnt = 7 and i = 7 and of course the number >> of >> > sync bytes = buf_size = 5. Before leaving the initialization loop I >> > memcpy the 5 preamble bytes to the out array, which means the next >> index in >> > "out" to write on the next input data sample is out[5], however, this >> > sample comes from in[i] = in[7]. Immediately after initialization, >> hence >> > the relationship: >> > out[i-cnt+buf_size] = in[i] is correct because out[5] = in[7] and as i >> > increases, we get out[6] = in[8], etc. >> > >> > However, I see one big problem that will cause my logic to fail and it >> > is based on your explanation that each time the general_work method is >> > called it brings in a new set of data and i starts over in the >> "for loop". >> > I will need to change my logic, but please allow me to ask another >> > question. Let's say, I have a source which generates 100 bytes and it >> > Repeats. In Gnuradio, does this mean that each time the general_work >> > method is called it receives a block of100 bytes of data? Thus, in the >> C >> > logic test "for (int i = 0; i < ninput_items[0]; i++)", can I assume >> the >> > Gnuradio parameter ninput_items[0] is equal to 100 and each function >> call? >> > >> > Thanks for the help. >> > >> > Best Regards, >> > George >> > >> > >> > On Sun, Sep 13, 2020 at 2:51 AM Martin Luelf <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi George, >> > >> > noutput_items is a number given to you by GNURadio. I would not >> > overwrite this variable, otherwise you no longer know how many items >> > you >> > can write into the output buffer. Create a new variable >> nitems_written >> > or similar to track how many items you have written. >> > >> > Also have a look at what each variable is tracking exactly. cnt >> > seems to >> > be the number of input bytes you have checked for the preamble. But >> you >> > also use it to compute the position in the out array, which does not >> > make sense to me. cnt would increase with more spurious bytes in >> front >> > of the preamble, but the output position should not depend on the >> > number >> > of spurious bytes. From the code you showed me cnt also seems to >> > persist >> > between multiple calls to general_work, but the out array is always >> > clean when general_work is called again (meaning that on every >> > general_work call you should always start writing to element 0). >> > >> > You loop over your entire input array and if the preamble was found >> you >> > copy the entire preamble and then you copy every following byte one >> by >> > one. That means you copy the entire preamble again (without the >> first >> > byte) and if your input buffer is longer than your message size you >> > also >> > copy more bytes to the output than your message is long which could >> > result in missing the next preamble. Also you are potentially >> writing >> > more items into the out array than GNURadio has space for, which is >> why >> > I recommend you not to overwrite noutput_items and while you are >> still >> > learning check that your computed indices for out and in are within >> the >> > allowed bounds before every single write and read (i.e. by using >> > assertions). >> > >> > What messages are you using in your unittest and what kind of >> spurious >> > bytes? Is is all zeros, or all ones? I would recommend you use >> > different >> > bytes, e.g. an increasing counter for your message and a decreasing >> > counter starting from 255 for your spurious bytes so you can quickly >> > spot if your messages are cropped, or off in some other way. Because >> > the >> > way you describe your results it seems like one message is copied >> > twice. >> > I could not find any obvious bug in your code that would output a >> > message exactly twice, so I suppose it copies some data that look >> > like a >> > message on the first glance. >> > >> > Yours >> > Martin >> > >> > >> > On 11.09.20 03:06, George Edwards wrote: >> > > Hi Martin, >> > > >> > > Thanks for your detailed answer. I really appreciate the great >> > effort >> > > you put into explaining how things work. I am still on the >> > learning curve. >> > > >> > > I used your suggestions to the best of my understanding and it >> > worked >> > > beautifully for the one sync pattern test vector in the QA test. >> > > >> > > Then, I took your suggestion for repeated sync patterns using an >> > init >> > > flag which I reset to 0 to restart the process. For the QA test, >> I >> > > repeated the original input data twice (now I have 2 >> > sync patterns), so >> > > the expected QA output should be the original output repeated >> > twice. I >> > > modified the C code by adding an if statement at the end to >> check if >> > > noutput_items == Buf_size+message_size (buf_size is the length of >> > the >> > > pattern, which I call preamble) and if it is, I reset the init >> > flag to >> > > zero as well as other params used in the initialization section >> > of the >> > > code. The QA test failed by producing an output with 3 repeated >> > copies >> > > of the original output rather than the expected 2 copies. I do >> not >> > > expect you to send too much time looking at my code below, >> > however, I >> > > would appreciate it very much, if you would glance at it to see >> > if you >> > > can spot what I am doing wrong. The test to un-initialize >> > (setting init >> > > to 0) was done towards the end of the code block after the >> > consume method. >> > > >> > > int kk = 0;____ >> > > >> > > for (int i = 0; i < ninput_items[0]; i++)____ >> > > >> > > {____ >> > > >> > > if(!init){__ >> > > >> > > cnt += 1; // cnt number of passing >> > bytes____ >> > > >> > > kk = initialize(in[i]);____ >> > > >> > > if (kk == 0){____ >> > > >> > > noutput_items = cnt;____ >> > > >> > > }else{____ >> > > >> > > memcpy((void*)out, (const void*)preamble, >> > buf_size);____ >> > > >> > > noutput_items = cnt;____ >> > > >> > > } >> > > >> > > } else { >> > > >> > > out[i-cnt+buf_size] = in[i];____ >> > > >> > > noutput_items = buf_size + message_size;____ >> > > >> > > }____ >> > > >> > > } ____ >> > > >> > > consume_each (noutput_items);____ >> > > >> > > if (noutput_items == buf_size + message_size){____ >> > > >> > > init = 0; // re-initialize all____ >> > > >> > > cnt = 0;____ >> > > >> > > kk = 0;____ >> > > >> > > }____ >> > > >> > > return noutput_items;____ >> > > >> > > } >> > > >> > > >> > > Thanks very much for the help. >> > > >> > > Regards, >> > > George >> > > >> > > On Thu, Sep 10, 2020 at 12:06 AM Martin Luelf <[email protected] >> > <mailto:[email protected]> >> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > > >> > > Dear George, >> > > >> > > this also caused me a lot of headache when I started with >> > GNURadio, so >> > > here is what I learned about it. >> > > >> > > Let's start with the forecast method. This tells GNURadio how >> > many >> > > input >> > > samples you need on each input stream to generate the >> > requested number >> > > of output items. Usually GNURadio will run your forecast >> > function a >> > > couple of times with different output numbers to find a good >> > data chunk >> > > size to give to the general work function. Keep in mind that >> > the number >> > > of required input items you give here is a minimum number and >> > GNURadio >> > > might decide to give you more input data than you requested. >> > It is also >> > > important to know that the number of samples you request >> here is >> > > just an >> > > estimate. You are not forced to actually process that much >> data. >> > > >> > > Now to the general_work function. noutput_items tells you how >> > many >> > > samples GNURadio would like you to produce (this is a maximum >> > number >> > > you >> > > may produce less). It also tells you how much memory is >> > allocated in >> > > every array in the output_items vector. If you have only one >> > output and >> > > you used the default <out type> *out = (<out type> *) >> > output_items[0]; >> > > definition this tells you how many items you can place (at >> > most) into >> > > the out array. >> > > >> > > The ninput_items[] array tells you how many input items are >> > > available in >> > > the input arrays. Again if you just have one input and you >> > use const >> > > <in >> > > type> *in = (const <in type> *) input_items[0]; >> > ninput_items[0] is the >> > > number of inputs available in the in array. You may not read >> > more items >> > > than that from the array. >> > > >> > > Within the given input symbols you can start looking for your >> > sync >> > > pattern. If you generate output you have to write (in your >> case >> > > copy) it >> > > to the out array. At the end of general_work you call >> > consume(0, K) >> > > with >> > > the number of input items K that you have consumed on input >> > 0. That is >> > > how many of the input items you have used and do not need to >> > see again. >> > > If you consume 0 symbols the next call to general_work will >> > show you >> > > the >> > > exact same input samples again. If you consume >> > ninput_items[0] you will >> > > see completely new input samples on the first input the next >> time >> > > general_work is called. And then you return the number of >> > samples you >> > > generated (i.e. how much samples you put into the out array). >> > This >> > > number must be smaller or equal noutput_items, because >> > otherwise you >> > > would have written out of the allocated memory which might >> > result in a >> > > segfault/core dump. Note that you don't have to call consume >> > at the >> > > very >> > > end of work and there is also another way of telling GNURadio >> > how many >> > > samples you have produced, but let's leave that for another >> day. >> > > >> > > So a very easy (but not the most efficient) setup for your >> > problem >> > > could be: >> > > Add a boolean flag to your _impl class that both forecast and >> > > general_work can read/write to. This flag will indicate >> > whether or not >> > > you have found the sync pattern or not. You initialize this >> flag >> > > with false. >> > > Assume you have a sync pattern of length L and a message with >> > M data >> > > symbols afterwards. >> > > >> > > In forecast if the flag is set (meaning you have found the >> sync >> > > pattern) >> > > you need L+M symbols of input. If the flag is not set you >> > need L input >> > > samples, regardless of how many output samples GNURadio wants >> > you to >> > > generate. >> > > >> > > In general work if the flag is false you search the input for >> > the sync >> > > pattern. If you found it at position i (counting from 0) you >> > set the >> > > flag to true, consume i samples (i.e. everything before the >> sync >> > > marker). If the sync marker is not found you keep the flag to >> > false and >> > > consume the inputs that you have searched so far. In both >> > cases you >> > > return 0 since you have not generated any output yet. >> > > >> > > If the flag is true you copy the first L+M samples from the >> > input to >> > > the >> > > output, you set the flag to false (because after the data you >> > have to >> > > start searching for the sync marker again) you consume L+M >> > samples and >> > > return L+M samples. >> > > >> > > Note: This is a very easy to understand scheme, but >> > unfortunately not >> > > very efficient. You only process a single block of either >> > unwanted >> > > spurious symbols, or one sync marker and data at a time. So >> > once you >> > > have a good understanding of how this works you should tweak >> > that block >> > > to be able to process multiple blocks of spurious symbols and >> > sync >> > > patterns/data within once call to general_work. It uses the >> same >> > > kind of >> > > logic, but requires more housekeeping of counters and >> indices. >> > > >> > > If your input is symbols rather than bits/bytes you should >> > also look at >> > > the paper from J. Massey "Optimum Frame Synchronization" from >> > 1972 on >> > > how to perform optimum sync marker detection, which performs >> > better >> > > than >> > > the intuitive correlation search. >> > > >> > > Hope that gets you started. >> > > >> > > Yours >> > > Martin >> > > >> > > >> > > On 10.09.20 04:34, George Edwards wrote: >> > > > Hello, >> > > > >> > > > I am writing an OOT block in C++ that receives a sequence >> of >> > > numbers and >> > > > searches through for a sync pattern and passes to its >> > output the >> > > sync >> > > > pattern and the bytes of data which follows it. The QA >> test >> > > shows the >> > > > block recognizes the pattern and will pass the pattern >> > along with >> > > the >> > > > data that follow it, but there is a problem. The block >> > does not >> > > know a >> > > > priori the number of spurious bytes preceding the sync >> > pattern of >> > > bytes, >> > > > so I cannot set up the true relationship between the >> > ninput_items >> > > and >> > > > noutput_items. However, the block can count the number of >> > bytes that >> > > > came in before the pattern. This is my problem: >> > > > >> > > > 1) In the OOT general_work method: If I set noutput_items >> = >> > > > ninput_items[0], then in addition to passing the correct >> > data to the >> > > > output, it passes trailing zeros equal to the number of >> > spurious >> > > bytes >> > > > that entered before the pattern. >> > > > >> > > > 2) If I set the return noutput_items = ninput_items - cnt >> > (where >> > > cnt is >> > > > the number of spurious bytes before the pattern) >> depending on >> > > where I >> > > > put noutput_items in the code, it either throws a core >> dump or >> > > cuts off >> > > > the number of true data. >> > > > >> > > > Also, within the forecast method, I simply use a _for_ >> > loop and set >> > > > ninput_items_required[i]=noutput_items; >> > > > >> > > > I will appreciate any help to point me in the right >> > direction in >> > > dealing >> > > > with this non-regular output/ inputs relationship. >> > > > >> > > > Thank you! >> > > > >> > > > George >> > > >> > >> >
