Hi, you fixed it! It’s working beautifully now, thank you both :)
Best regards Marius Steffen mar...@mariussteffen.de > Am 07.01.2019 um 09:19 schrieb Marcus Meissner <mar...@jet.franken.de>: > > Hi, > > can you try test patch below? (did not test myself as i have to leave for > work ;) > > Ciao, Marcus > On Mon, Jan 07, 2019 at 08:53:04AM +0100, Marcus Meissner wrote: >> Hi, >> >> Hmm, I was hoping this was not the case, but well. >> >> I implemented related code for Canon EOS, I will do such a thing for Nikon >> too. >> >> Ciao, Marcus >> On Mon, Jan 07, 2019 at 07:29:04AM +0000, Horshack wrote: >>> I didn't realize libgphoto2 issued such large MTP object read requests. >>> That is most likely the issue. When I was developing airnef >>> (http://testcams.com/airnef/) I discovered an intermittent bug on Nikon's >>> MTP implementation that caused it to randomly hang during large MPT object >>> read requests. The first symptom was the transfer would start to slow, and >>> then eventually would stop and time out. I attributed the issue to a heap >>> issue inside the camera's firmware, based on some experimentation trying >>> varying-sized MTP requests. The solution I implemented was a configurable >>> per-transfer size limit, with a default of 1MB - based on my testing that >>> yielded the max throughput (same as larger sizes) while still avoiding the >>> Nikon firmware issue. >>> >>> Here's the comment from my airnef code: >>> >>> # >>> # When I first wrote this routine I was experiencing very flaky wireless >>> behavior from >>> # all Nikon bodies I tested it with (D7200, J4, D750, WU-1a). About halfway >>> through a >>> # NEF download via MTP_OP_GetObject the transfer rate would become erratic >>> and sometimes >>> # the camera would completely stop transferring and then eventually time >>> out on a socket >>> # receive. It wouldn't do this on every file but about once every few >>> files. To handle >>> # this behavior I added significant amounts of recovery logic, both in the >>> low-levels >>> # of mtpwifi.py and the upper levels of this module (appMain), both of >>> which were designed >>> # to retain any partially transferred data and allow the retry of >>> downloads. Luckily when >>> # a Nikon body hits this condition it could be revived by dropping the >>> TCP/IP connection >>> # and restarting everything over again, from the opening of the TCP/IP >>> socket through the >>> # MTP start session and back to this routine to retrieve the portion of the >>> file we have >>> # left to download via MTP_OP_GetPartialObject. >>> # >>> # It later occurred to me that the issue might be specific to Nikon's >>> implementation of >>> # MTP_OP_GetObject, as it appeared the camera's erratic behavior was >>> related to how large >>> # the object was - JPEGs were fine but larger NEFs should issues and very >>> large MOV >>> # files were the worst. It seemed the camera has some type of memory leak >>> associated with >>> # the command, perhaps committing internal resources to the entire file >>> rather than to each >>> # payload piece. So I experimented with transfers that relied solely >>> MTP_OP_GetPartialObject, >>> # using various transfer sizes, and confirmed my suspicion - there's a bug >>> in Nikon's firmware >>> # related to large transfers, both from the atomic MTP_OP_GetObject request >>> and also >>> # large MTP_OP_GetPartialObject requests. Based on these results I rewrote >>> this method >>> # to use only MTP_OP_GetPartialObject and with smaller transfer sizes and >>> all the erratic >>> # Nikon behavior disappeared. Fortunately performance did not suffer - I >>> found that using >>> # a transfer size of 1MB (vs the full object size in the orig >>> implementation) yielded the >>> # same performance as the latter, and actually much better when you >>> consider that we didn't >>> # have to go through time-consuming connection tear-downs and >>> re-establishment cycles. On >>> # both my D7200 and J4 I see approx 2.3 MB/s of sustained xfer performance >>> on the adhoc wifi >>> # conneciton when the camera is next to the computer. >>> # >>> >>> Adam >>> >>> >>> ________________________________ >>> From: Marius Steffen <mar...@mariussteffen.de> >>> Sent: Sunday, January 6, 2019 4:38 PM >>> To: Horshack >>> Cc: Marcus Meissner; gphoto-devel@lists.sourceforge.net; Marcus Meissner >>> Subject: Re: [gphoto-devel] PTP/IP RAW download stalling gphoto2 >>> >>> Hi, >>> >>> You’re right, of course - the size seems to be the problem indeed. >>> Even with the patch applied, the problem persists. But the results are >>> interesting: neither is the file completely downloaded, nor does it always >>> stall at the same position, as you can see in the following lines acquired >>> from multiple test runs: >>> >>> 40.717975 ptp_ptpip_generic_read (2): reading at offset 31297528, >>> reading 13983496 bytes >>> 41.232571 ptp_ptpip_generic_read (2): reading at offset 38903800, >>> reading 6360451 bytes >>> 36.434165 ptp_ptpip_generic_read (2): reading at offset 37052408, >>> reading 8150851 bytes >>> 35.296001 ptp_ptpip_generic_read (2): reading at offset 30875640, >>> reading 14042982 bytes >>> >>> Best regards >>> >>> Marius Steffen >>> mar...@mariussteffen.de >>> >>> >>> >>>> Am 06.01.2019 um 19:35 schrieb Horshack <horsh...@live.com>: >>>> >>>> The fact it fails with uncompressed or lossless compressed NEFs but not >>>> lossy compressed NEFs hints the issue may be related to files above a >>>> certain size, as the lossy version will be smallest of the three options. >>>> You can prove/disprove this by trying a lossless compressed NEF shot in DX >>>> area mode, which will be smaller than the FX version. >>>> >>>> Adam >>>> >>>> On Jan 6, 2019, at 10:28 AM, Marcus Meissner <meiss...@suse.de> wrote: >>>> >>>>> Hi, >>>>> >>>>> Yes, it is very likely an issue in libgphoto2. >>>>> >>>>> If you download the NEF over USB, is the end bytes also the bytes below? >>>>> >>>>> One other things is that the camera might expect us polling the event fd >>>>> too, >>>>> can you try the following patch, it has both more debug and also checking >>>>> for events inbetween reads. >>>>> >>>>> Ciao, Marcus >>>>> >>>>> >>>>> >>>>> On Sun, Jan 06, 2019 at 06:07:15PM +0100, Marius Steffen wrote: >>>>>> Hi, >>>>>> >>>>>> strange - though, IIRC on my old D5500 everything was working, too. >>>>>> I suspected SnapBridge at first, but using „qDslrDashboard“, which isn’t >>>>>> using libgphoto2 (at least it doesn’t link against the dylib), I’m able >>>>>> to successfully download the RAW file. >>>>>> >>>>>> It’s hanging after the 0490 line - though there were more of this data >>>>>> dump blocks for the .NEF, this is the last one. I can’t really tell how >>>>>> much of the .NEF is already received, but looking the time elapsed and >>>>>> the size of the log, I suspect it’s (almost) completely read. >>>>>> >>>>>> The camera is turned on the whole time. >>>>>> >>>>>> I’m using libgphoto2 in an application, with exactly the same problem >>>>>> (so the problem’s probably not in gphoto2 CLI, but libgphoto2) - >>>>>> according to the debugger, it’s stalled at read. >>>>>> >>>>>> Best regards >>>>>> >>>>>> Marius Steffen >>>>>> mar...@mariussteffen.de >>>>>> >>>>>> >>>>>> >>>>>>> Am 06.01.2019 um 17:19 schrieb Marcus Meissner <mar...@jet.franken.de>: >>>>>>> >>>>>>> On Fri, Jan 04, 2019 at 04:39:27AM +0100, Marius Steffen wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I’m having problems downloading RAW images from my Nikon D850, >>>>>>>> connected via PTP/IP. >>>>>>>> The camera is configured in RAW+JPEG(Basic) S mode, so gphoto2 >>>>>>>> transfers the JPEG file first, which succeeds. >>>>>>>> Afterwards, it is downloading the RAW file, but never finishes - >>>>>>>> instead, at some point, it freezes, the camera finally disconnects >>>>>>>> (without gphoto2 noticing). >>>>>>>> Only manually aborting gphoto2 is shutting it down, without ever >>>>>>>> saving the RAW file. >>>>>>>> >>>>>>>> I’ve done some tests, and only ever observed this behavior when using >>>>>>>> „Lossless compressed“ or „Uncompressed“ RAW file compression settings. >>>>>>>> „Compressed“ (= lossy compression) seems to work though. >>>>>>>> >>>>>>>> I’d like to have this fixed, and am willing to help of course, as this >>>>>>>> leads to crashing my cameras WiFi, which then has to be turned on >>>>>>>> again using SnapBridge. >>>>>>>> >>>>>>>> I’ve attached the relevant log file, without the image data; the last >>>>>>>> part in the full log is something like: >>>>>>>> >>>>>>>> 36.773805 ptp_ptpip_generic_read (3): ptpip/generic_read data: >>>>>>>> (hexdump of 1176 bytes) >>>>>>>> 0000 f3 ef df df cf cf 6f df-5f 4f 4f df 7f df bd 7f ......o._OO..... >>>>>>>> 0010 7f 7e fb f3 db ef 4f bd-ff 7e fc f4 fd fd fc fb .~....O..~...... >>>>>>>> 0020 d3 db 9b d3 8f ef 6f bf-3d fe fb f7 f3 db f3 8f ......o.=....... >>>>>>>> 0030 f7 db f3 ef cf cf df bd-ff 7d 3e fb 93 ef bd b8 .........}>..... >>>>>>>> 0040 bd 3e f7 fb f3 f3 f3 f7-f7 f7 f3 ef df bf 3e f6 .>............>. >>>>>>>> 0050 fc f5 fb db f7 d7 f7 db-d3 f3 93 ef 5f cf df bf ............_... >>>>>>>> 0060 3f 7f 3e fb f3 f7 f7 ef-bd 7f 3d bf 7f 3f 7f 7d ?.>.......=..?.} >>>>>>>> 0070 7f 7d bd 7d 7f 3f 7f 7f-7f 7f 3d 7e fd fd f7 fb .}.}.?....=~.... >>>>>>>> 0080 f3 f3 f3 f3 ef df bf 3d-7f 7d bf 7e fc fd fb ef .......=.}.~.... >>>>>>>> 0090 bd 7f 3e fb db f7 d7 f3-ef cf b9 3f 7f 3e f5 fd ..>........?.>.. >>>>>>>> 00a0 fb f3 f7 ef bf 7e fc fd-fb d7 ef be e2 fd e5 fc .....~.......... >>>>>>>> 00b0 fb ef bd bf 7d 7d 3e e4-fc fb f7 f3 ef cf df 6f ....}}>........o >>>>>>>> 00c0 cf 4f bd bf 38 3f 72 38-fe e5 fb f3 d7 ef be fb .O..8?r8........ >>>>>>>> 00d0 8f ef bf 7d fd 7f 7f 7e-fb f3 ef df be fb ef df ...}...~........ >>>>>>>> 00e0 be fc fd f7 f5 fc fd f5-f5 fb f7 f7 ef df bd 3e ……………> >>>>>>>> <SNIP> >>>>>>>> 0490 f3 9f f7 d7 ef df 4f 4f- ......OO >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have tried reproducing this with the Nikon D750 and Nikon Z6 over >>>>>>> Wifi, >>>>>>> but failed... it downloads both JPG and uncompressed RAW. >>>>>>> >>>>>>> How much data has been read from the NEF file at above step? >>>>>>> >>>>>>> Is there another try to read after the 0490 ... line, or does it hang >>>>>>> right there? >>>>>>> >>>>>>> Does it hang there with the camera still on? >>>>>>> >>>>>>> >>>>>>> Can you attach gdb and check if it is hanging somewhere besides read? >>>>>>> >>>>>>> Ciao, Marcus >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gphoto-devel mailing list >>>>>> Gphoto-devel@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/gphoto-devel >>>>> >>>>> -- >>>>> Marcus Meissner,SUSE LINUX GmbH; Maxfeldstrasse 5; D-90409 Nuernberg; Zi. >>>>> 3.1-33,+49-911-740 53-432,,serv=loki,mail=wotan,type=real >>>>> <meiss...@suse.de> >>>>> <xx.pat> >>>>> _______________________________________________ >>>>> Gphoto-devel mailing list >>>>> Gphoto-devel@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/gphoto-devel >>> >> >> >> _______________________________________________ >> Gphoto-devel mailing list >> Gphoto-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/gphoto-devel > <xx.pat>_______________________________________________ > Gphoto-devel mailing list > Gphoto-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/gphoto-devel _______________________________________________ Gphoto-devel mailing list Gphoto-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gphoto-devel
