Hi Yojiro, It would be best if you could reduce the problem to a minimal self-contained test case. If your program is multiple files then it is probably too big for us to look at. All I really need is for you to add a main() function to the code you sent before, so I can see exactly the code that demonstrates the problem.
-Kenton On Sun, Feb 21, 2021 at 9:07 AM [email protected] < [email protected]> wrote: > Hi Kenton > > I prepared my source for sending to you. > Can I send it to your email address? or reply to this group with it as > attachment? > > Thank you. > > > 2021年2月21日日曜日 20:09:41 UTC+9 [email protected]: > >> Hi Kenton, >> >> Glad to see your reply. >> >> Yes I am aware of open mode. I do, for writing, >> return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_CREAT | _O_RDWR | >> _O_TRUNC), _SH_DENYNO, (_S_IREAD | _S_IWRITE)); >> and for reading, >> return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_RDWR), (_SH_DENYNO), >> (_S_IREAD | _S_IWRITE)); >> >> Using _O_WRONLY for writing, _O_RDONLY for reading did not change the >> situation. >> For shared flag, I have tried SH_DENYNO or SH_DENYRW but made no >> difference, as I had expected. >> >> I can provide my code. I will send it to you when its ready, have to >> clean it a little I guess. >> >> Thank you very much for the message. >> >> 2021年2月20日土曜日 1:55:08 UTC+9 [email protected]: >> >>> Hi, >>> >>> Sorry for the slow reply. >>> >>> Just going to throw out a guess here, but seeing that you're using >>> windows, did you make sure to use O_BINARY when opening the file >>> descriptor? If you didn't, then the C library will silently corrupt your >>> data by converting line endings. >>> >>> If that's not it, then could you provide a complete self-contained >>> program (with main() function and all parameters filled in) that >>> demonstrates the problem? >>> >>> -Kenton >>> >>> On Tue, Feb 16, 2021 at 7:15 AM [email protected] < >>> [email protected]> wrote: >>> >>>> Hi >>>> >>>> I could do a little more things. >>>> >>>> Interestingly, capnp.exe built under x64 can do decode message. >>>> I bet it does different processing than MessageReader's do and simply >>>> follow message format structure. >>>> >>>> I made a test code using MapViewOfFile and FlatArrayMessageReader,. >>>> It can read messages that I couldnt by using StreamFdMessageReader. >>>> But easily fails with a little large data, say 32 entries in first >>>> level List and 1 entry in the second level List, >>>> getting following exception: >>>> >>>> Exception: capnp\serialize.c++:51: failed: expected array.size() >= >>>> offset + segmentSize; Message ends prematurely in first segment. >>>> stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23 >>>> 7ffc1b18d720 >>>> >>>> So I think I'm stuck. Any advice on how to proceed will be greatly >>>> appreciated. >>>> >>>> Thank you in advance. >>>> >>>> >>>> 2021年2月16日火曜日 16:23:38 UTC+9 [email protected]: >>>> >>>>> Hi >>>>> >>>>> So, trying to understand what is happening. >>>>> >>>>> using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be >>>>> read correctly. >>>>> capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin >>>>> By looking at hex dump of the message file, it looks like a proper >>>>> message format. >>>>> >>>>> So, writing new message is okay but reading it is not working properly. >>>>> Using capnproto-0.8.0 built with VS2017/x64 , and create a new >>>>> message, save it to a file, and then reading it by my program fails. >>>>> Since all happens when I do message.getRoot<Dir4D>(), this "Premature >>>>> EOF" situation happens inside capnp library. >>>>> >>>>> Is there any way to fix this? or to narrow down what is wrong? >>>>> >>>>> Thank you. >>>>> >>>>> 2021年2月16日火曜日 4:56:44 UTC+9 [email protected]: >>>>> >>>>>> Hello >>>>>> >>>>>> I have been learning this wonderful software. >>>>>> So I have been looking for an opportunity to use capnproto for my >>>>>> project, >>>>>> and now I have it but I need to use it with MSVC 2017(recently >>>>>> updated to latest version 15.9.33), x64. >>>>>> I only need serialization and no need for RPC or async. >>>>>> This is my very first time posting a message to this community, >>>>>> please let me know if I'm not doing right here. >>>>>> >>>>>> I have a schema whose basic construct is List of List of struct ( x,y >>>>>> points in 4d expressed by two level lists). Second level will have a few >>>>>> hundreds of entries and first level will have lots of entries. >>>>>> And I am now stuck with a issue. If I put many entries in either of >>>>>> lists (more than a few hundreds) and write that to a file, then when I >>>>>> read >>>>>> the message, I get exception like this: >>>>>> Any help would be very appreciated. >>>>>> >>>>>> printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >= >>>>>> minBytes; Premature EOF >>>>>> stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308 >>>>>> 7ff672fb15d3 7ffc1aa97c23 7ffc1b18d720 >>>>>> >>>>>> Now I found that writing 4 entries in first level, two entries in >>>>>> second level (two x,y points each in 4 frames), I get the same error. >>>>>> >>>>>> I'm putting my schema and my read/write code below. I'm using these >>>>>> codes with capnproto0.8.0. But actually this schema is a little striped >>>>>> version than the one I want to use. When I used that a little more >>>>>> complex >>>>>> schema, I faced similar issue with capnproto 0.7.0 and 0.8.0. >>>>>> >>>>>> I built capnproto following the steps in the website, had to skip >>>>>> heavy-tests project and kj-async (0.8.0), hoping the build was ok. >>>>>> >>>>>> Is there anything missing or done wrong? >>>>>> It would be very much appreciated for any help. >>>>>> Thank you. >>>>>> >>>>>> >>>>>> sparse4d.capnp : >>>>>> >>>>>> # unique file ID, generated by `capnp id` >>>>>> @0xae7dda29a38d355f; >>>>>> >>>>>> struct Sparse4DHeader { >>>>>> tSize @0 :Int32; >>>>>> zSize @1 :Int32; >>>>>> ySize @2 :Int32; >>>>>> xSize @3 :Int32; >>>>>> } >>>>>> >>>>>> struct Dir4D { >>>>>> header @0 :Sparse4DHeader; >>>>>> planes @1 :List(SparsePlane); >>>>>> >>>>>> struct SparsePlane { >>>>>> t @0 :UInt16; >>>>>> z @1 :UInt16; >>>>>> points @2 :List(Point); >>>>>> } >>>>>> } >>>>>> >>>>>> struct Point { >>>>>> x @0 :UInt16; >>>>>> y @1 :UInt16; >>>>>> intensity @2 :Float32; >>>>>> } >>>>>> >>>>>> >>>>>> And read/write code: >>>>>> >>>>>> /** >>>>>> * Copyright (c) 2021 ASTOM R&D >>>>>> * Author: Yojiro Nobukuni >>>>>> */ >>>>>> //#include "pch.h" >>>>>> >>>>>> #include "../sparse4d.capnp.h" >>>>>> #include <capnp/message.h> >>>>>> #include <capnp/serialize-packed.h> >>>>>> #include <iostream> >>>>>> #include <ios> >>>>>> >>>>>> #include <sys/types.h> >>>>>> #include <sys/stat.h> >>>>>> #include <fcntl.h> >>>>>> //#include <unistd.h> >>>>>> >>>>>> /** >>>>>> * g++ writer.cpp photodeco_kernel.capnp.c++ -o writer -lcapnp -lkj >>>>>> * ./writer > test.message >>>>>> */ >>>>>> >>>>>> /** >>>>>> * Write PhotodecoKernel message to fd >>>>>> * @param int fd fd to read message in >>>>>> */ >>>>>> void writeSparse4D(int fd, unsigned int nofPlanes, unsigned int >>>>>> nofPoints, bool packed) >>>>>> { >>>>>> ::capnp::MallocMessageBuilder message; >>>>>> >>>>>> auto dir4d = message.initRoot<Dir4D>(); >>>>>> auto header = dir4d.initHeader(); >>>>>> header.setTSize(7); >>>>>> header.setZSize(7); >>>>>> header.setXSize(7); >>>>>> header.setYSize(7); >>>>>> >>>>>> auto planes = dir4d.initPlanes(nofPlanes); >>>>>> for (auto plane : planes) { >>>>>> plane.setT(9); >>>>>> plane.setZ(8); >>>>>> auto points = plane.initPoints(nofPoints); >>>>>> #if 0 >>>>>> // This one or the other one in #else block, same >>>>>> behavior. >>>>>> for (unsigned int i = 0; i < nofPoints; i++) { >>>>>> points[i].setX(1); >>>>>> points[i].setY(2); >>>>>> points[i].setIntensity(3); >>>>>> } >>>>>> #else >>>>>> for (auto p : points) { >>>>>> p.setX(1); >>>>>> p.setY(2); >>>>>> p.setIntensity(3); >>>>>> } >>>>>> #endif >>>>>> } >>>>>> >>>>>> try { >>>>>> if (packed) >>>>>> writePackedMessageToFd(fd, message); >>>>>> else >>>>>> writeMessageToFd(fd, message); >>>>>> } >>>>>> catch (std::exception& e) { >>>>>> std::cout << "writeSparse4D: std::exception: " << e.what() << >>>>>> std::endl; >>>>>> return; >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> /** >>>>>> * Print PhotodecoKernerl kernel in text >>>>>> * @param PhotodecoKernel::Reader kernel >>>>>> */ >>>>>> void printDir4D(Dir4D::Reader dir4d, bool printPoints) >>>>>> { >>>>>> auto header = dir4d.getHeader(); >>>>>> std::cout << " sizes[T,Z,Y,X] = " >>>>>> << header.getTSize() << ", " << header.getZSize() << ", " << >>>>>> header.getYSize() << ", " << header.getXSize() << ", " >>>>>> << std::endl; >>>>>> //std::cout << " nofPoints = " << header.getNofPoints() << std::endl; >>>>>> >>>>>> if (!printPoints) return; >>>>>> >>>>>> size_t nofPoints = 0; >>>>>> auto planes = dir4d.getPlanes(); >>>>>> for (auto plane : planes) { >>>>>> const auto t = plane.getT(); >>>>>> const auto z = plane.getZ(); >>>>>> std::cout << "plane(t,z): " << t << ", " << z << std::endl; >>>>>> for (auto p : plane.getPoints()) { >>>>>> std::cout << t << ", " << z << ", " << p.getY() << ", " << p.getX() >>>>>> << ", " << p.getIntensity() << std::endl; >>>>>> nofPoints++; >>>>>> } >>>>>> } >>>>>> >>>>>> std::cout << "Total nof KernelPoints = " << nofPoints << std::endl; >>>>>> } >>>>>> >>>>>> /** >>>>>> * Read PhotodecoKernerl message from fd and print it in text >>>>>> * @param int fd >>>>>> * @param bool packed >>>>>> */ >>>>>> void printSparseDir4D(int fd, bool printPoints, bool packed) >>>>>> { >>>>>> std::cout << "printSparseDir4D" << std::endl; >>>>>> >>>>>> // readerOptions for reading message. >>>>>> capnp::ReaderOptions readerOptions; >>>>>> readerOptions.traversalLimitInWords = 1000 * 1000 * 1000 * >>>>>> readerOptions.traversalLimitInWords; >>>>>> //readerOptions.nestingLimit; >>>>>> >>>>>> ::capnp::MessageReader *message; >>>>>> try { >>>>>> if (packed) >>>>>> message = new ::capnp::PackedFdMessageReader(fd, readerOptions); >>>>>> else >>>>>> message = new ::capnp::StreamFdMessageReader(fd, readerOptions); >>>>>> >>>>>> auto dir4d = message->getRoot<Dir4D>(); >>>>>> printDir4D(dir4d, printPoints); >>>>>> } >>>>>> catch (std::exception& e) { >>>>>> std::cout << "printSparseDir4D: std::exception: " << e.what() << >>>>>> std::endl; >>>>>> return; >>>>>> } >>>>>> >>>>>> >>>>>> delete message; >>>>>> } >>>>>> >>>>>> >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Cap'n Proto" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>> >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/854ef4ca-17b6-4776-bd4d-03340c75dbbcn%40googlegroups.com > <https://groups.google.com/d/msgid/capnproto/854ef4ca-17b6-4776-bd4d-03340c75dbbcn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnL2Brgu8ETiYL06DzBrwzDxKq7TZpGyHnnMWqNwy8d%3DQ%40mail.gmail.com.
