Hi Kenton, Glad to see your reply.
Yes I am aware of open mode. I do, for writing, return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_CREAT | _O_RDWR | _O_TRUNC), _SH_DENYNO, (_S_IREAD | _S_IWRITE)); and for reading, return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_RDWR), (_SH_DENYNO), (_S_IREAD | _S_IWRITE)); Using _O_WRONLY for writing, _O_RDONLY for reading did not change the situation. For shared flag, I have tried SH_DENYNO or SH_DENYRW but made no difference, as I had expected. I can provide my code. I will send it to you when its ready, have to clean it a little I guess. Thank you very much for the message. 2021年2月20日土曜日 1:55:08 UTC+9 [email protected]: > Hi, > > Sorry for the slow reply. > > Just going to throw out a guess here, but seeing that you're using > windows, did you make sure to use O_BINARY when opening the file > descriptor? If you didn't, then the C library will silently corrupt your > data by converting line endings. > > If that's not it, then could you provide a complete self-contained program > (with main() function and all parameters filled in) that demonstrates the > problem? > > -Kenton > > On Tue, Feb 16, 2021 at 7:15 AM [email protected] <[email protected]> > wrote: > >> Hi >> >> I could do a little more things. >> >> Interestingly, capnp.exe built under x64 can do decode message. >> I bet it does different processing than MessageReader's do and simply >> follow message format structure. >> >> I made a test code using MapViewOfFile and FlatArrayMessageReader,. >> It can read messages that I couldnt by using StreamFdMessageReader. >> But easily fails with a little large data, say 32 entries in first level >> List and 1 entry in the second level List, >> getting following exception: >> >> Exception: capnp\serialize.c++:51: failed: expected array.size() >= >> offset + segmentSize; Message ends prematurely in first segment. >> stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23 >> 7ffc1b18d720 >> >> So I think I'm stuck. Any advice on how to proceed will be greatly >> appreciated. >> >> Thank you in advance. >> >> >> 2021年2月16日火曜日 16:23:38 UTC+9 [email protected]: >> >>> Hi >>> >>> So, trying to understand what is happening. >>> >>> using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be >>> read correctly. >>> capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin >>> By looking at hex dump of the message file, it looks like a proper >>> message format. >>> >>> So, writing new message is okay but reading it is not working properly. >>> Using capnproto-0.8.0 built with VS2017/x64 , and create a new message, >>> save it to a file, and then reading it by my program fails. >>> Since all happens when I do message.getRoot<Dir4D>(), this "Premature >>> EOF" situation happens inside capnp library. >>> >>> Is there any way to fix this? or to narrow down what is wrong? >>> >>> Thank you. >>> >>> 2021年2月16日火曜日 4:56:44 UTC+9 [email protected]: >>> >>>> Hello >>>> >>>> I have been learning this wonderful software. >>>> So I have been looking for an opportunity to use capnproto for my >>>> project, >>>> and now I have it but I need to use it with MSVC 2017(recently updated >>>> to latest version 15.9.33), x64. >>>> I only need serialization and no need for RPC or async. >>>> This is my very first time posting a message to this community, please >>>> let me know if I'm not doing right here. >>>> >>>> I have a schema whose basic construct is List of List of struct ( x,y >>>> points in 4d expressed by two level lists). Second level will have a few >>>> hundreds of entries and first level will have lots of entries. >>>> And I am now stuck with a issue. If I put many entries in either of >>>> lists (more than a few hundreds) and write that to a file, then when I >>>> read >>>> the message, I get exception like this: >>>> Any help would be very appreciated. >>>> >>>> printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >= >>>> minBytes; Premature EOF >>>> stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308 7ff672fb15d3 >>>> 7ffc1aa97c23 7ffc1b18d720 >>>> >>>> Now I found that writing 4 entries in first level, two entries in >>>> second level (two x,y points each in 4 frames), I get the same error. >>>> >>>> I'm putting my schema and my read/write code below. I'm using these >>>> codes with capnproto0.8.0. But actually this schema is a little striped >>>> version than the one I want to use. When I used that a little more complex >>>> schema, I faced similar issue with capnproto 0.7.0 and 0.8.0. >>>> >>>> I built capnproto following the steps in the website, had to skip >>>> heavy-tests project and kj-async (0.8.0), hoping the build was ok. >>>> >>>> Is there anything missing or done wrong? >>>> It would be very much appreciated for any help. >>>> Thank you. >>>> >>>> >>>> sparse4d.capnp : >>>> >>>> # unique file ID, generated by `capnp id` >>>> @0xae7dda29a38d355f; >>>> >>>> struct Sparse4DHeader { >>>> tSize @0 :Int32; >>>> zSize @1 :Int32; >>>> ySize @2 :Int32; >>>> xSize @3 :Int32; >>>> } >>>> >>>> struct Dir4D { >>>> header @0 :Sparse4DHeader; >>>> planes @1 :List(SparsePlane); >>>> >>>> struct SparsePlane { >>>> t @0 :UInt16; >>>> z @1 :UInt16; >>>> points @2 :List(Point); >>>> } >>>> } >>>> >>>> struct Point { >>>> x @0 :UInt16; >>>> y @1 :UInt16; >>>> intensity @2 :Float32; >>>> } >>>> >>>> >>>> And read/write code: >>>> >>>> /** >>>> * Copyright (c) 2021 ASTOM R&D >>>> * Author: Yojiro Nobukuni >>>> */ >>>> //#include "pch.h" >>>> >>>> #include "../sparse4d.capnp.h" >>>> #include <capnp/message.h> >>>> #include <capnp/serialize-packed.h> >>>> #include <iostream> >>>> #include <ios> >>>> >>>> #include <sys/types.h> >>>> #include <sys/stat.h> >>>> #include <fcntl.h> >>>> //#include <unistd.h> >>>> >>>> /** >>>> * g++ writer.cpp photodeco_kernel.capnp.c++ -o writer -lcapnp -lkj >>>> * ./writer > test.message >>>> */ >>>> >>>> /** >>>> * Write PhotodecoKernel message to fd >>>> * @param int fd fd to read message in >>>> */ >>>> void writeSparse4D(int fd, unsigned int nofPlanes, unsigned int >>>> nofPoints, bool packed) >>>> { >>>> ::capnp::MallocMessageBuilder message; >>>> >>>> auto dir4d = message.initRoot<Dir4D>(); >>>> auto header = dir4d.initHeader(); >>>> header.setTSize(7); >>>> header.setZSize(7); >>>> header.setXSize(7); >>>> header.setYSize(7); >>>> >>>> auto planes = dir4d.initPlanes(nofPlanes); >>>> for (auto plane : planes) { >>>> plane.setT(9); >>>> plane.setZ(8); >>>> auto points = plane.initPoints(nofPoints); >>>> #if 0 >>>> // This one or the other one in #else block, same >>>> behavior. >>>> for (unsigned int i = 0; i < nofPoints; i++) { >>>> points[i].setX(1); >>>> points[i].setY(2); >>>> points[i].setIntensity(3); >>>> } >>>> #else >>>> for (auto p : points) { >>>> p.setX(1); >>>> p.setY(2); >>>> p.setIntensity(3); >>>> } >>>> #endif >>>> } >>>> >>>> try { >>>> if (packed) >>>> writePackedMessageToFd(fd, message); >>>> else >>>> writeMessageToFd(fd, message); >>>> } >>>> catch (std::exception& e) { >>>> std::cout << "writeSparse4D: std::exception: " << e.what() << std::endl; >>>> return; >>>> } >>>> } >>>> >>>> >>>> /** >>>> * Print PhotodecoKernerl kernel in text >>>> * @param PhotodecoKernel::Reader kernel >>>> */ >>>> void printDir4D(Dir4D::Reader dir4d, bool printPoints) >>>> { >>>> auto header = dir4d.getHeader(); >>>> std::cout << " sizes[T,Z,Y,X] = " >>>> << header.getTSize() << ", " << header.getZSize() << ", " << >>>> header.getYSize() << ", " << header.getXSize() << ", " >>>> << std::endl; >>>> //std::cout << " nofPoints = " << header.getNofPoints() << std::endl; >>>> >>>> if (!printPoints) return; >>>> >>>> size_t nofPoints = 0; >>>> auto planes = dir4d.getPlanes(); >>>> for (auto plane : planes) { >>>> const auto t = plane.getT(); >>>> const auto z = plane.getZ(); >>>> std::cout << "plane(t,z): " << t << ", " << z << std::endl; >>>> for (auto p : plane.getPoints()) { >>>> std::cout << t << ", " << z << ", " << p.getY() << ", " << p.getX() << >>>> ", " << p.getIntensity() << std::endl; >>>> nofPoints++; >>>> } >>>> } >>>> >>>> std::cout << "Total nof KernelPoints = " << nofPoints << std::endl; >>>> } >>>> >>>> /** >>>> * Read PhotodecoKernerl message from fd and print it in text >>>> * @param int fd >>>> * @param bool packed >>>> */ >>>> void printSparseDir4D(int fd, bool printPoints, bool packed) >>>> { >>>> std::cout << "printSparseDir4D" << std::endl; >>>> >>>> // readerOptions for reading message. >>>> capnp::ReaderOptions readerOptions; >>>> readerOptions.traversalLimitInWords = 1000 * 1000 * 1000 * >>>> readerOptions.traversalLimitInWords; >>>> //readerOptions.nestingLimit; >>>> >>>> ::capnp::MessageReader *message; >>>> try { >>>> if (packed) >>>> message = new ::capnp::PackedFdMessageReader(fd, readerOptions); >>>> else >>>> message = new ::capnp::StreamFdMessageReader(fd, readerOptions); >>>> >>>> auto dir4d = message->getRoot<Dir4D>(); >>>> printDir4D(dir4d, printPoints); >>>> } >>>> catch (std::exception& e) { >>>> std::cout << "printSparseDir4D: std::exception: " << e.what() << >>>> std::endl; >>>> return; >>>> } >>>> >>>> >>>> delete message; >>>> } >>>> >>>> >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/838636b5-60ec-4fdf-8649-b6743683e104n%40googlegroups.com.
