Hi,
can you try test patch below? (did not test myself as i have to leave for work
;)
Ciao, Marcus
On Mon, Jan 07, 2019 at 08:53:04AM +0100, Marcus Meissner wrote:
> Hi,
>
> Hmm, I was hoping this was not the case, but well.
>
> I implemented related code for Canon EOS, I will do such a thing for Nikon
> too.
>
> Ciao, Marcus
> On Mon, Jan 07, 2019 at 07:29:04AM +0000, Horshack wrote:
> > I didn't realize libgphoto2 issued such large MTP object read requests.
> > That is most likely the issue. When I was developing airnef
> > (http://testcams.com/airnef/) I discovered an intermittent bug on Nikon's
> > MTP implementation that caused it to randomly hang during large MPT object
> > read requests. The first symptom was the transfer would start to slow, and
> > then eventually would stop and time out. I attributed the issue to a heap
> > issue inside the camera's firmware, based on some experimentation trying
> > varying-sized MTP requests. The solution I implemented was a configurable
> > per-transfer size limit, with a default of 1MB - based on my testing that
> > yielded the max throughput (same as larger sizes) while still avoiding the
> > Nikon firmware issue.
> >
> > Here's the comment from my airnef code:
> >
> > #
> > # When I first wrote this routine I was experiencing very flaky wireless
> > behavior from
> > # all Nikon bodies I tested it with (D7200, J4, D750, WU-1a). About halfway
> > through a
> > # NEF download via MTP_OP_GetObject the transfer rate would become erratic
> > and sometimes
> > # the camera would completely stop transferring and then eventually time
> > out on a socket
> > # receive. It wouldn't do this on every file but about once every few
> > files. To handle
> > # this behavior I added significant amounts of recovery logic, both in the
> > low-levels
> > # of mtpwifi.py and the upper levels of this module (appMain), both of
> > which were designed
> > # to retain any partially transferred data and allow the retry of
> > downloads. Luckily when
> > # a Nikon body hits this condition it could be revived by dropping the
> > TCP/IP connection
> > # and restarting everything over again, from the opening of the TCP/IP
> > socket through the
> > # MTP start session and back to this routine to retrieve the portion of the
> > file we have
> > # left to download via MTP_OP_GetPartialObject.
> > #
> > # It later occurred to me that the issue might be specific to Nikon's
> > implementation of
> > # MTP_OP_GetObject, as it appeared the camera's erratic behavior was
> > related to how large
> > # the object was - JPEGs were fine but larger NEFs should issues and very
> > large MOV
> > # files were the worst. It seemed the camera has some type of memory leak
> > associated with
> > # the command, perhaps committing internal resources to the entire file
> > rather than to each
> > # payload piece. So I experimented with transfers that relied solely
> > MTP_OP_GetPartialObject,
> > # using various transfer sizes, and confirmed my suspicion - there's a bug
> > in Nikon's firmware
> > # related to large transfers, both from the atomic MTP_OP_GetObject request
> > and also
> > # large MTP_OP_GetPartialObject requests. Based on these results I rewrote
> > this method
> > # to use only MTP_OP_GetPartialObject and with smaller transfer sizes and
> > all the erratic
> > # Nikon behavior disappeared. Fortunately performance did not suffer - I
> > found that using
> > # a transfer size of 1MB (vs the full object size in the orig
> > implementation) yielded the
> > # same performance as the latter, and actually much better when you
> > consider that we didn't
> > # have to go through time-consuming connection tear-downs and
> > re-establishment cycles. On
> > # both my D7200 and J4 I see approx 2.3 MB/s of sustained xfer performance
> > on the adhoc wifi
> > # conneciton when the camera is next to the computer.
> > #
> >
> > Adam
> >
> >
> > ________________________________
> > From: Marius Steffen <mar...@mariussteffen.de>
> > Sent: Sunday, January 6, 2019 4:38 PM
> > To: Horshack
> > Cc: Marcus Meissner; gphoto-devel@lists.sourceforge.net; Marcus Meissner
> > Subject: Re: [gphoto-devel] PTP/IP RAW download stalling gphoto2
> >
> > Hi,
> >
> > You’re right, of course - the size seems to be the problem indeed.
> > Even with the patch applied, the problem persists. But the results are
> > interesting: neither is the file completely downloaded, nor does it always
> > stall at the same position, as you can see in the following lines acquired
> > from multiple test runs:
> >
> > 40.717975 ptp_ptpip_generic_read (2): reading at offset 31297528,
> > reading 13983496 bytes
> > 41.232571 ptp_ptpip_generic_read (2): reading at offset 38903800,
> > reading 6360451 bytes
> > 36.434165 ptp_ptpip_generic_read (2): reading at offset 37052408,
> > reading 8150851 bytes
> > 35.296001 ptp_ptpip_generic_read (2): reading at offset 30875640,
> > reading 14042982 bytes
> >
> > Best regards
> >
> > Marius Steffen
> > mar...@mariussteffen.de
> >
> >
> >
> > > Am 06.01.2019 um 19:35 schrieb Horshack <horsh...@live.com>:
> > >
> > > The fact it fails with uncompressed or lossless compressed NEFs but not
> > > lossy compressed NEFs hints the issue may be related to files above a
> > > certain size, as the lossy version will be smallest of the three options.
> > > You can prove/disprove this by trying a lossless compressed NEF shot in
> > > DX area mode, which will be smaller than the FX version.
> > >
> > > Adam
> > >
> > > On Jan 6, 2019, at 10:28 AM, Marcus Meissner <meiss...@suse.de> wrote:
> > >
> > >> Hi,
> > >>
> > >> Yes, it is very likely an issue in libgphoto2.
> > >>
> > >> If you download the NEF over USB, is the end bytes also the bytes below?
> > >>
> > >> One other things is that the camera might expect us polling the event fd
> > >> too,
> > >> can you try the following patch, it has both more debug and also checking
> > >> for events inbetween reads.
> > >>
> > >> Ciao, Marcus
> > >>
> > >>
> > >>
> > >> On Sun, Jan 06, 2019 at 06:07:15PM +0100, Marius Steffen wrote:
> > >>> Hi,
> > >>>
> > >>> strange - though, IIRC on my old D5500 everything was working, too.
> > >>> I suspected SnapBridge at first, but using „qDslrDashboard“, which
> > >>> isn’t using libgphoto2 (at least it doesn’t link against the dylib),
> > >>> I’m able to successfully download the RAW file.
> > >>>
> > >>> It’s hanging after the 0490 line - though there were more of this data
> > >>> dump blocks for the .NEF, this is the last one. I can’t really tell how
> > >>> much of the .NEF is already received, but looking the time elapsed and
> > >>> the size of the log, I suspect it’s (almost) completely read.
> > >>>
> > >>> The camera is turned on the whole time.
> > >>>
> > >>> I’m using libgphoto2 in an application, with exactly the same problem
> > >>> (so the problem’s probably not in gphoto2 CLI, but libgphoto2) -
> > >>> according to the debugger, it’s stalled at read.
> > >>>
> > >>> Best regards
> > >>>
> > >>> Marius Steffen
> > >>> mar...@mariussteffen.de
> > >>>
> > >>>
> > >>>
> > >>>> Am 06.01.2019 um 17:19 schrieb Marcus Meissner <mar...@jet.franken.de>:
> > >>>>
> > >>>> On Fri, Jan 04, 2019 at 04:39:27AM +0100, Marius Steffen wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> I’m having problems downloading RAW images from my Nikon D850,
> > >>>>> connected via PTP/IP.
> > >>>>> The camera is configured in RAW+JPEG(Basic) S mode, so gphoto2
> > >>>>> transfers the JPEG file first, which succeeds.
> > >>>>> Afterwards, it is downloading the RAW file, but never finishes -
> > >>>>> instead, at some point, it freezes, the camera finally disconnects
> > >>>>> (without gphoto2 noticing).
> > >>>>> Only manually aborting gphoto2 is shutting it down, without ever
> > >>>>> saving the RAW file.
> > >>>>>
> > >>>>> I’ve done some tests, and only ever observed this behavior when using
> > >>>>> „Lossless compressed“ or „Uncompressed“ RAW file compression
> > >>>>> settings. „Compressed“ (= lossy compression) seems to work though.
> > >>>>>
> > >>>>> I’d like to have this fixed, and am willing to help of course, as
> > >>>>> this leads to crashing my cameras WiFi, which then has to be turned
> > >>>>> on again using SnapBridge.
> > >>>>>
> > >>>>> I’ve attached the relevant log file, without the image data; the last
> > >>>>> part in the full log is something like:
> > >>>>>
> > >>>>> 36.773805 ptp_ptpip_generic_read (3): ptpip/generic_read data:
> > >>>>> (hexdump of 1176 bytes)
> > >>>>> 0000 f3 ef df df cf cf 6f df-5f 4f 4f df 7f df bd 7f
> > >>>>> ......o._OO.....
> > >>>>> 0010 7f 7e fb f3 db ef 4f bd-ff 7e fc f4 fd fd fc fb
> > >>>>> .~....O..~......
> > >>>>> 0020 d3 db 9b d3 8f ef 6f bf-3d fe fb f7 f3 db f3 8f
> > >>>>> ......o.=.......
> > >>>>> 0030 f7 db f3 ef cf cf df bd-ff 7d 3e fb 93 ef bd b8
> > >>>>> .........}>.....
> > >>>>> 0040 bd 3e f7 fb f3 f3 f3 f7-f7 f7 f3 ef df bf 3e f6
> > >>>>> .>............>.
> > >>>>> 0050 fc f5 fb db f7 d7 f7 db-d3 f3 93 ef 5f cf df bf
> > >>>>> ............_...
> > >>>>> 0060 3f 7f 3e fb f3 f7 f7 ef-bd 7f 3d bf 7f 3f 7f 7d
> > >>>>> ?.>.......=..?.}
> > >>>>> 0070 7f 7d bd 7d 7f 3f 7f 7f-7f 7f 3d 7e fd fd f7 fb
> > >>>>> .}.}.?....=~....
> > >>>>> 0080 f3 f3 f3 f3 ef df bf 3d-7f 7d bf 7e fc fd fb ef
> > >>>>> .......=.}.~....
> > >>>>> 0090 bd 7f 3e fb db f7 d7 f3-ef cf b9 3f 7f 3e f5 fd
> > >>>>> ..>........?.>..
> > >>>>> 00a0 fb f3 f7 ef bf 7e fc fd-fb d7 ef be e2 fd e5 fc
> > >>>>> .....~..........
> > >>>>> 00b0 fb ef bd bf 7d 7d 3e e4-fc fb f7 f3 ef cf df 6f
> > >>>>> ....}}>........o
> > >>>>> 00c0 cf 4f bd bf 38 3f 72 38-fe e5 fb f3 d7 ef be fb
> > >>>>> .O..8?r8........
> > >>>>> 00d0 8f ef bf 7d fd 7f 7f 7e-fb f3 ef df be fb ef df
> > >>>>> ...}...~........
> > >>>>> 00e0 be fc fd f7 f5 fc fd f5-f5 fb f7 f7 ef df bd 3e ……………>
> > >>>>> <SNIP>
> > >>>>> 0490 f3 9f f7 d7 ef df 4f 4f- ......OO
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I have tried reproducing this with the Nikon D750 and Nikon Z6 over
> > >>>> Wifi,
> > >>>> but failed... it downloads both JPG and uncompressed RAW.
> > >>>>
> > >>>> How much data has been read from the NEF file at above step?
> > >>>>
> > >>>> Is there another try to read after the 0490 ... line, or does it hang
> > >>>> right there?
> > >>>>
> > >>>> Does it hang there with the camera still on?
> > >>>>
> > >>>>
> > >>>> Can you attach gdb and check if it is hanging somewhere besides read?
> > >>>>
> > >>>> Ciao, Marcus
> > >>>
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Gphoto-devel mailing list
> > >>> Gphoto-devel@lists.sourceforge.net
> > >>> https://lists.sourceforge.net/lists/listinfo/gphoto-devel
> > >>
> > >> --
> > >> Marcus Meissner,SUSE LINUX GmbH; Maxfeldstrasse 5; D-90409 Nuernberg;
> > >> Zi. 3.1-33,+49-911-740 53-432,,serv=loki,mail=wotan,type=real
> > >> <meiss...@suse.de>
> > >> <xx.pat>
> > >> _______________________________________________
> > >> Gphoto-devel mailing list
> > >> Gphoto-devel@lists.sourceforge.net
> > >> https://lists.sourceforge.net/lists/listinfo/gphoto-devel
> >
>
>
> _______________________________________________
> Gphoto-devel mailing list
> Gphoto-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gphoto-devel
diff --git a/camlibs/ptp2/library.c b/camlibs/ptp2/library.c
index 24e69ffb6..4fddfd48d 100644
--- a/camlibs/ptp2/library.c
+++ b/camlibs/ptp2/library.c
@@ -7554,6 +7554,26 @@ get_file_func (CameraFilesystem *fs, const char *folder,
const char *filename,
}
goto done;
}
+ if (
(ptp_operation_issupported(params,PTP_OC_GetPartialObject)) &&
+ (size > BLOBSIZE)
+ ) {
+ unsigned char *ximage = NULL;
+ uint32_t offset = 0;
+
+ while (offset < size) {
+ uint32_t xsize = size - offset;
+ uint32_t xlen;
+
+ if (xsize > BLOBSIZE)
+ xsize = BLOBSIZE;
+ C_PTP_REP (ptp_getpartialobject
(params, oid, offset, xsize, &ximage, &xlen));
+ gp_file_append (file, (char*)ximage,
xlen);
+ free (ximage);
+ ximage = NULL;
+ offset += xlen;
+ }
+ goto done;
+ }
#undef BLOBSIZE
if (size) {
uint16_t ret;
_______________________________________________
Gphoto-devel mailing list
Gphoto-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
