Hi, Sorry for the slow reply.
Just going to throw out a guess here, but seeing that you're using windows, did you make sure to use O_BINARY when opening the file descriptor? If you didn't, then the C library will silently corrupt your data by converting line endings. If that's not it, then could you provide a complete self-contained program (with main() function and all parameters filled in) that demonstrates the problem? -Kenton On Tue, Feb 16, 2021 at 7:15 AM [email protected] < [email protected]> wrote: > Hi > > I could do a little more things. > > Interestingly, capnp.exe built under x64 can do decode message. > I bet it does different processing than MessageReader's do and simply > follow message format structure. > > I made a test code using MapViewOfFile and FlatArrayMessageReader,. > It can read messages that I couldnt by using StreamFdMessageReader. > But easily fails with a little large data, say 32 entries in first level > List and 1 entry in the second level List, > getting following exception: > > Exception: capnp\serialize.c++:51: failed: expected array.size() >= offset > + segmentSize; Message ends prematurely in first segment. > stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23 > 7ffc1b18d720 > > So I think I'm stuck. Any advice on how to proceed will be greatly > appreciated. > > Thank you in advance. > > > 2021年2月16日火曜日 16:23:38 UTC+9 [email protected]: > >> Hi >> >> So, trying to understand what is happening. >> >> using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be >> read correctly. >> capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin >> By looking at hex dump of the message file, it looks like a proper >> message format. >> >> So, writing new message is okay but reading it is not working properly. >> Using capnproto-0.8.0 built with VS2017/x64 , and create a new message, >> save it to a file, and then reading it by my program fails. >> Since all happens when I do message.getRoot<Dir4D>(), this "Premature >> EOF" situation happens inside capnp library. >> >> Is there any way to fix this? or to narrow down what is wrong? >> >> Thank you. >> >> 2021年2月16日火曜日 4:56:44 UTC+9 [email protected]: >> >>> Hello >>> >>> I have been learning this wonderful software. >>> So I have been looking for an opportunity to use capnproto for my >>> project, >>> and now I have it but I need to use it with MSVC 2017(recently updated >>> to latest version 15.9.33), x64. >>> I only need serialization and no need for RPC or async. >>> This is my very first time posting a message to this community, please >>> let me know if I'm not doing right here. >>> >>> I have a schema whose basic construct is List of List of struct ( x,y >>> points in 4d expressed by two level lists). Second level will have a few >>> hundreds of entries and first level will have lots of entries. >>> And I am now stuck with a issue. If I put many entries in either of >>> lists (more than a few hundreds) and write that to a file, then when I read >>> the message, I get exception like this: >>> Any help would be very appreciated. >>> >>> printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >= >>> minBytes; Premature EOF >>> stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308 7ff672fb15d3 >>> 7ffc1aa97c23 7ffc1b18d720 >>> >>> Now I found that writing 4 entries in first level, two entries in second >>> level (two x,y points each in 4 frames), I get the same error. >>> >>> I'm putting my schema and my read/write code below. I'm using these >>> codes with capnproto0.8.0. But actually this schema is a little striped >>> version than the one I want to use. When I used that a little more complex >>> schema, I faced similar issue with capnproto 0.7.0 and 0.8.0. >>> >>> I built capnproto following the steps in the website, had to skip >>> heavy-tests project and kj-async (0.8.0), hoping the build was ok. >>> >>> Is there anything missing or done wrong? >>> It would be very much appreciated for any help. >>> Thank you. >>> >>> >>> sparse4d.capnp : >>> >>> # unique file ID, generated by `capnp id` >>> @0xae7dda29a38d355f; >>> >>> struct Sparse4DHeader { >>> tSize @0 :Int32; >>> zSize @1 :Int32; >>> ySize @2 :Int32; >>> xSize @3 :Int32; >>> } >>> >>> struct Dir4D { >>> header @0 :Sparse4DHeader; >>> planes @1 :List(SparsePlane); >>> >>> struct SparsePlane { >>> t @0 :UInt16; >>> z @1 :UInt16; >>> points @2 :List(Point); >>> } >>> } >>> >>> struct Point { >>> x @0 :UInt16; >>> y @1 :UInt16; >>> intensity @2 :Float32; >>> } >>> >>> >>> And read/write code: >>> >>> /** >>> * Copyright (c) 2021 ASTOM R&D >>> * Author: Yojiro Nobukuni >>> */ >>> //#include "pch.h" >>> >>> #include "../sparse4d.capnp.h" >>> #include <capnp/message.h> >>> #include <capnp/serialize-packed.h> >>> #include <iostream> >>> #include <ios> >>> >>> #include <sys/types.h> >>> #include <sys/stat.h> >>> #include <fcntl.h> >>> //#include <unistd.h> >>> >>> /** >>> * g++ writer.cpp photodeco_kernel.capnp.c++ -o writer -lcapnp -lkj >>> * ./writer > test.message >>> */ >>> >>> /** >>> * Write PhotodecoKernel message to fd >>> * @param int fd fd to read message in >>> */ >>> void writeSparse4D(int fd, unsigned int nofPlanes, unsigned int >>> nofPoints, bool packed) >>> { >>> ::capnp::MallocMessageBuilder message; >>> >>> auto dir4d = message.initRoot<Dir4D>(); >>> auto header = dir4d.initHeader(); >>> header.setTSize(7); >>> header.setZSize(7); >>> header.setXSize(7); >>> header.setYSize(7); >>> >>> auto planes = dir4d.initPlanes(nofPlanes); >>> for (auto plane : planes) { >>> plane.setT(9); >>> plane.setZ(8); >>> auto points = plane.initPoints(nofPoints); >>> #if 0 >>> // This one or the other one in #else block, same >>> behavior. >>> for (unsigned int i = 0; i < nofPoints; i++) { >>> points[i].setX(1); >>> points[i].setY(2); >>> points[i].setIntensity(3); >>> } >>> #else >>> for (auto p : points) { >>> p.setX(1); >>> p.setY(2); >>> p.setIntensity(3); >>> } >>> #endif >>> } >>> >>> try { >>> if (packed) >>> writePackedMessageToFd(fd, message); >>> else >>> writeMessageToFd(fd, message); >>> } >>> catch (std::exception& e) { >>> std::cout << "writeSparse4D: std::exception: " << e.what() << std::endl; >>> return; >>> } >>> } >>> >>> >>> /** >>> * Print PhotodecoKernerl kernel in text >>> * @param PhotodecoKernel::Reader kernel >>> */ >>> void printDir4D(Dir4D::Reader dir4d, bool printPoints) >>> { >>> auto header = dir4d.getHeader(); >>> std::cout << " sizes[T,Z,Y,X] = " >>> << header.getTSize() << ", " << header.getZSize() << ", " << >>> header.getYSize() << ", " << header.getXSize() << ", " >>> << std::endl; >>> //std::cout << " nofPoints = " << header.getNofPoints() << std::endl; >>> >>> if (!printPoints) return; >>> >>> size_t nofPoints = 0; >>> auto planes = dir4d.getPlanes(); >>> for (auto plane : planes) { >>> const auto t = plane.getT(); >>> const auto z = plane.getZ(); >>> std::cout << "plane(t,z): " << t << ", " << z << std::endl; >>> for (auto p : plane.getPoints()) { >>> std::cout << t << ", " << z << ", " << p.getY() << ", " << p.getX() << >>> ", " << p.getIntensity() << std::endl; >>> nofPoints++; >>> } >>> } >>> >>> std::cout << "Total nof KernelPoints = " << nofPoints << std::endl; >>> } >>> >>> /** >>> * Read PhotodecoKernerl message from fd and print it in text >>> * @param int fd >>> * @param bool packed >>> */ >>> void printSparseDir4D(int fd, bool printPoints, bool packed) >>> { >>> std::cout << "printSparseDir4D" << std::endl; >>> >>> // readerOptions for reading message. >>> capnp::ReaderOptions readerOptions; >>> readerOptions.traversalLimitInWords = 1000 * 1000 * 1000 * >>> readerOptions.traversalLimitInWords; >>> //readerOptions.nestingLimit; >>> >>> ::capnp::MessageReader *message; >>> try { >>> if (packed) >>> message = new ::capnp::PackedFdMessageReader(fd, readerOptions); >>> else >>> message = new ::capnp::StreamFdMessageReader(fd, readerOptions); >>> >>> auto dir4d = message->getRoot<Dir4D>(); >>> printDir4D(dir4d, printPoints); >>> } >>> catch (std::exception& e) { >>> std::cout << "printSparseDir4D: std::exception: " << e.what() << >>> std::endl; >>> return; >>> } >>> >>> >>> delete message; >>> } >>> >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com > <https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnSX5tBvM9M7WUKr0YMzuXyqSOz%2BmVFG3O1oH1iu8rfyQ%40mail.gmail.com.
