Got! I had to apply the mask of 0xff because the inputStream.read() function returns ints while i was getting bytes from the bytes[] array.
On 26 February 2017 at 15:41, Jameson Lopp <jameson.l...@gmail.com> wrote: > The logic looks correct to me; it blows up on the very first block? Have > you tried any other .dat files? > > I hope that this import tool hasn't become obsolete due to any changes in > how bitcoind stores blocks on disk... > > On Sun, Feb 26, 2017 at 3:00 PM, Nishil Shah <nishil.sha...@gmail.com> > wrote: > >> Oh, that's perfect. I'm very, very close. Do you mind taking at a short >> snippet of my code I put into this test file: >> >> https://gist.github.com/anonymous/8b748d8a3810515be8d9349768eb92e4 >> >> So, I replicated most of the logic found in BlockFileLoader.java. Rather >> than reading from input stream, I am trying to keep and index to iterate >> through the actual bytes themselves. >> >> Line 66 yields the error: NegativeArraySizeException at >> org.bitcoinj.core.Message.readBytes. I'm guessing that my index is wrong >> at some point. Can't spot the bug, however. >> >> Keep in mind, I want to build from the bytes array, because in another >> situation I do not want to use the FileInputStream. >> >> Any ideas? Thank you so much. >> >> On 25 February 2017 at 16:34, Jameson Lopp <jameson.l...@gmail.com> >> wrote: >> >>> Aha, if you can somehow get access to the bytes then you should be able >>> to duplicate the byte logic here: https://github.com/bitco >>> inj/bitcoinj/blob/3177bd52a2bfa491c5902e95b8840030e1a31159/c >>> ore/src/main/java/org/bitcoinj/utils/BlockFileLoader.java#L129 >>> >>> Basically, you need to do some low level byte parsing so that you can >>> eventually read in what the size of the next Block is, then you can take >>> the next N bytes and pass them to the deserializer in order to actually >>> construct the Block object: https://github.com/bit >>> coinj/bitcoinj/blob/3177bd52a2bfa491c5902e95b8840030e1a31159 >>> /core/src/main/java/org/bitcoinj/utils/BlockFileLoader.java#L152 >>> >>> - Jameson >>> >>> On Sat, Feb 25, 2017 at 5:20 PM, Nishil Shah <nishil.sha...@gmail.com> >>> wrote: >>> >>>> Figured it out. I'm using hadoop and the files are stored on the >>>> distributed file system. Need to find a workaround. If I read all the bytes >>>> from the file, can I construct a block with it by passing it in to the >>>> Block constructor? What exactly is the "payloadBytes" parameter? >>>> >>>> Thanks. >>>> >>>> On 22 February 2017 at 16:49, Jameson Lopp <jameson.l...@gmail.com> >>>> wrote: >>>> >>>>> That's quite odd; it sounds like you didn't actually add the File(s) >>>>> to the List. Did you do a .exists() check on them like it does here? >>>>> https://github.com/bitcoinj/bitcoinj/blob/3177bd52a2bf >>>>> a491c5902e95b8840030e1a31159/core/src/main/java/org/bitcoinj >>>>> /utils/BlockFileLoader.java#L67 >>>>> >>>>> I don't think the different styles of getting the params should >>>>> matter. Feel free to share your code. >>>>> >>>>> On Wed, Feb 22, 2017 at 3:14 PM, Nishil Shah <nishil.sha...@gmail.com> >>>>> wrote: >>>>> >>>>>> Ah, I can't check right now, but I remember I was using new >>>>>> MainNetParams() as opposed to MainNetParams.get(). Could that be the >>>>>> difference? >>>>>> >>>>>> On 22 February 2017 at 12:53, Nishil Shah <nishil.sha...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> next() just complains that there is no next item in the iterator. >>>>>>> Does this mean the I/O failed? According to the docs, the >>>>>>> BlockFileLoader >>>>>>> swallows I/O exceptions, so I can't see the I/O error. >>>>>>> >>>>>>> On 21 February 2017 at 10:31, Jameson Lopp <jameson.l...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> You should definitely be able to build your own List<File> list >>>>>>>> that you pass to the loader - I've done this myself in the past. >>>>>>>> >>>>>>>> What error are you seeing? >>>>>>>> >>>>>>>> On Mon, Feb 20, 2017 at 4:20 PM, Nishil Shah < >>>>>>>> nishil.sha...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I would use that List to initialize the block file loader. >>>>>>>>> >>>>>>>>> On 20 February 2017 at 15:19, Nishil Shah <nishil.sha...@gmail.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> I see, thanks. One last question: So shouldn't I be able to >>>>>>>>>> simply build a List<File> and append paths to my block files >>>>>>>>>> "/path/to/blk00000.dat"? I tried this and when I do >>>>>>>>>> blockFileLoader.next() >>>>>>>>>> I get an error. Don't think this works. >>>>>>>>>> >>>>>>>>>> On 19 February 2017 at 03:25, Jameson Lopp < >>>>>>>>>> jameson.l...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Right, getReferenceClientList >>>>>>>>>>> <https://github.com/bitcoinj/bitcoinj/blob/3177bd52a2bfa491c5902e95b8840030e1a31159/core/src/main/java/org/bitcoinj/utils/BlockFileLoader.java#L53> >>>>>>>>>>> just tries to be smart and figure out the location of your block >>>>>>>>>>> files >>>>>>>>>>> automatically. If it doesn't work for your system, you can model >>>>>>>>>>> its logic >>>>>>>>>>> to create your own function that builds a list of your block files. >>>>>>>>>>> >>>>>>>>>>> For your purposes, I'm assuming you're analyzing Bitcoin on >>>>>>>>>>> mainnet, so you'd pass MainNetParams.get(). The only other thing >>>>>>>>>>> for you to >>>>>>>>>>> figure out then would be your data storage, which would be your >>>>>>>>>>> "chain" >>>>>>>>>>> object or whatever function to which you pass the blocks to be >>>>>>>>>>> processed. >>>>>>>>>>> >>>>>>>>>>> - Jameson >>>>>>>>>>> >>>>>>>>>>> On Sun, Feb 19, 2017 at 2:00 AM, Nishil Shah < >>>>>>>>>>> nishil.sha...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> It doesn't take in a file name? >>>>>>>>>>>> >>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>> >>>>>>>>>>>> On Feb 19, 2017, at 12:33 AM, Jameson Lopp < >>>>>>>>>>>> jameson.l...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I think what you want is the BlockFileLoader, as seen here in >>>>>>>>>>>> the BlockImporter: https://github. >>>>>>>>>>>> com/bitcoinj/bitcoinj/blob/master/tools/src/main/java/org/bi >>>>>>>>>>>> tcoinj/tools/BlockImporter.java#L67 >>>>>>>>>>>> >>>>>>>>>>>> - Jameson >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Feb 18, 2017 at 11:51 PM, Nishil Shah < >>>>>>>>>>>> nishil.sha...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Is there anyway to import blk?????.dat files individually and >>>>>>>>>>>>> parse them into Block objects? I'm not trying to use any network >>>>>>>>>>>>> connections. I want to get everything from disk if possible to >>>>>>>>>>>>> make my >>>>>>>>>>>>> computations/processing as fast as possible because I want parse >>>>>>>>>>>>> many files >>>>>>>>>>>>> in parallel. I've read about some "block importer" tool but can't >>>>>>>>>>>>> really >>>>>>>>>>>>> see how to use it in my situation. Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "bitcoinj" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "bitcoinj" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/bitcoinj/4wB6-i7ys8E/unsub >>>>>>>>>>>> scribe. >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "bitcoinj" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "bitcoinj" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/bitcoinj/4wB6-i7ys8E/unsub >>>>>>>>>>> scribe. >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "bitcoinj" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "bitcoinj" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/bitcoinj/4wB6-i7ys8E/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> bitcoinj+unsubscr...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "bitcoinj" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to bitcoinj+unsubscr...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "bitcoinj" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>> pic/bitcoinj/4wB6-i7ys8E/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> bitcoinj+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "bitcoinj" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to bitcoinj+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "bitcoinj" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/bitcoinj/4wB6-i7ys8E/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> bitcoinj+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "bitcoinj" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to bitcoinj+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "bitcoinj" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/bitcoinj/4wB6-i7ys8E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > bitcoinj+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "bitcoinj" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoinj+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
