Re: Retrieving the EXIF date/time from 250k images
There is a perl program called exiftool that can load and set exif tool without
loading the image data (or at least it doesn’t decode the image data). I don’t
know whether it would be faster than loading image data/properties with
ImageIO. You could write a perl script that used your bundled exiftool to load
the exif data and output the results for many files in a format your program
could handle, because instantiating perl/exiftool repeatedly for each image in
a separate NSTask would probably be pretty slow.
Jim Crate
> On Jan 7, 2023, at 2:07 PM, Alex Zavatone via Cocoa-dev
> wrote:
>
> Hi Gabe. I’d add basic logging before you start each image and after you
> complete each image to see how much each is taking on each of problem tests
> so you can see the extent of how slow it is on your problem platforms.
>
> Then you can add more logging to expose the problems and start to address
> them once you see where the bottlenecks are.
>
> I wonder if there is a method to load the EXIF data out of the files without
> opening them completely. That would seem like the ideal approach.
>
> Cheers,
> Alex Zavatone
>
>> On Jan 7, 2023, at 12:36 PM, Gabriel Zachmann wrote:
>>
>> Hi Alex, hi everyone,
>>
>> thanks a lot for the many suggestions!
>> And sorry for following up on this so late!
>> I hope you are still willing to engage in this discussion.
>>
>> Yes, Alex, I agree in that the main question is:
>> how can I get the metadata of a large amount of images (say, 100k-300k)
>> *without* actually loading the whole image files.
>> (For your reference: I am interested in the date tags embedded in the EXIF
>> dictionary, and those dates will be read just once per image, then cached in
>> a dictionary containing filename & dates, and that dictionary will get
>> stored on disk for future use by the app.)
>>
>>> CGImageSourceRef imageSourceRef =
>>> CGImageSourceCreateWithURL((CFURLRef)imageUrl, NULL);
>>
>> I have tried this:
>>
>> for ( NSString* filename in imagefiles )
>> {
>> NSURL * imgurl = [NSURL fileURLWithPath: filename isDirectory: NO];
>>CGImageSourceRef sourceref = CGImageSourceCreateWithURL( (__bridge
>> CFURLRef) imgurl, NULL );
>> }
>>
>> This takes 1 minute for around 300k images stored on my internal SSD.
>> That would be OK.
>>
>> However! .. if performed on a folder stored on an external hard disk, I get
>> the following timings:
>>
>> - 20 min for 150k images (45 GB)
>> - 12 min for 150k images (45 GB), second time
>> - 150 sec for 25k images (18 GB)
>> - 170 sec for 25k images (18 GB), with the lines below (*)
>> - 80 sec for 22k (3 GB) images
>> - 80 sec for 22k (3 GB) images, with the lines below (*)
>>
>> All experiments were done on different folders on the same hard disk, WD
>> MyPassport Ultra, 1 TB, USB-A connector to Macbook Air M2.
>> Timings with the same number of files/GB were the same folders, resp.
>>
>> (*): these were timings where I added the following lines to the loop:
>>
>> CFDictionaryRef fileProps = CGImageSourceCopyPropertiesAtIndex( image,
>> 0, NULL );
>> bool success = CFDictionaryGetValueIfPresent( fileProps,
>> kCGImagePropertyExifDictionary, (const void **) & exif_dict );
>> CFDictionaryGetValueIfPresent( exif_dict,
>> kCGImagePropertyExifDateTimeDigitized, (const void **) & dateref );
>> iso_date = [isoDateFormatter_ dateFromString: (__bridge NSString *
>> _Nonnull)(dateref) ];
>> [datesAndTimes_ addObject: iso_date ];
>>
>> (Plus some error checking, which I omit here.)
>>
>> First of all, we can see that the vast majority of time is spent on
>> CGImageSourceCreateWithURL().
>> Second, there seem to be some caching effects, although I have a hard time
>> understanding that, but that is not the point.
>> Third, the durations are not linear; I guess it might have something to do
>> with the sizes of the files, too, but again, didn't investigate further.
>>
>> So, it looks to me like CGImageSourceCreateWithURL() really loads the
>> complete image file.
>>
>> I don't see why Ole Begemann (ref'ed in Alex' post) can claim his approach
>> does not load the whole image.
>>
>>
>> Some people suggested parallelizing the whole task, using
>> dispatch_queue_create or NSOperationQueue.
>> (Thanks Steve, Gary, Jack!)
>> Before restructuring my code for that, I would like to better understand why
>> you think that will speed up things.
>> The code above pretty much does no computations, so most of the time is, I
>> guess, spent on waiting for the data to arrive from hard disk.
>> So, why would would several threads loading those images in parallel help
>> here? In my thinking, they will just compete for the same resource, i.e.,
>> hard disk.
>>
>>
>> I also googled quite a bit, to no avail.
>>
>> Any and all hints, suggestions, and insights will be highly appreciated!
>> Best, Gab
>>
>>
>>>
>>
>>
>>> if (!imageSourceRef)
Re: Retrieving the EXIF date/time from 250k images
On Aug 18, 2022, at 7:47 AM, Mike Abdullah wrote: > > It’s not a very good fit, but when you say a “GCD concurrent queue”, you’d > need to be more specific. There are several configs possible. Do you mean a > global queue, or one you made yourself? If you made it yourself, how did you > configure it? Like Alex, for me this was 10-12 years ago so I don’t remember much about exactly how I tried to use GCD. It spun up too many threads, I re-read the docs and assumed it was spinning up new threads based on file IO blocking, and killing itself. I didn’t see a handy way to limit GCD concurrent queueing, and NSOperationQueue had maxConcurrentOperationCount documented. NSOperationQueue was also more idiomatic Objective-C and easier to use/understand in an Obj-C program. > The tricky problem is that GCD aims to optimise CPU usage. Reading image > metadata from disk is almost certainly not CPU bound; it’s going to be > limited by disk speed and the file system instead. GCD will have a tendency > to see tasks blocked on doing this, not occupying the CPU, and so spin up > another task in the hope that one _will_ then use the CPU, eventually > grinding to a halt as a huge number of threads contend for access to the file > system. In my case the app reads hundreds images from disk or network file share, performs operations (scale/rotate/crop/etc), rendering on CPU (better accuracy for printing) and writing to disk or network share. Perfect fit for NSOperationQueue with maxConcurrentOperationCount, so I was grateful someone at Apple had already done the hard work and made concurrent programming much easier for people with tasks like mine. The OP is just grabbing image metadata but would almost certainly run into the same problem using GCD's dispatch_async so they’ll also be much better off using NSOperationQueue with maxConcurrentOperationCount. Jim Crate > On 17 Aug 2022, at 20:32, James Crate via Cocoa-dev > wrote: >> >> I have an app that does some image processing, and when I tried to use GCD >> it created several hundred threads which didn’t work very well. >> NSOperationQueue allows you to set the max concurrent operations, and the >> batch exporting process fully utilizes all logical cores on the CPU. >> >> opsQueue.maxConcurrentOperationCount = >> NSProcessInfo.processInfo.processorCount; >> >> Maybe I was using GCD wrong, or maybe reading, processing, and writing >> several hundred images is not a good fit for GCD concurrent queue? In any >> case NSOperationQueue is easy to use and works well. >> >> Jim Crate ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Retrieving the EXIF date/time from 250k images
I have an app that does some image processing, and when I tried to use GCD it created several hundred threads which didn’t work very well. NSOperationQueue allows you to set the max concurrent operations, and the batch exporting process fully utilizes all logical cores on the CPU. opsQueue.maxConcurrentOperationCount = NSProcessInfo.processInfo.processorCount; Maybe I was using GCD wrong, or maybe reading, processing, and writing several hundred images is not a good fit for GCD concurrent queue? In any case NSOperationQueue is easy to use and works well. Jim Crate > On Aug 16, 2022, at 3:37 PM, Jack Brindle via Cocoa-dev > wrote: > > Instead of using NSOperationQueue, I would use GCD to handle the tasks. > Create a new Concurrent queue > (dispatch_queue_create(DISPATCH_QUEUE_CONCURRENT)), then enqueue the > individual items to the queue for processing (dispatch_async(), using the > queue created above). Everything can be handled in blocks, including the > completion routines. As Christian says the problem then is that data may not > be in the original order so you will probably want to sort the returned > objects when done. This should significantly speed up the time to do the > whole task. > > Jack > > >> On Aug 16, 2022, at 12:26 PM, Steve Christensen via Cocoa-dev >> wrote: >> >> You mentioned creating and managing threads on your own, but that’s what >> NSOperationQueue —and the lower-level DispatchQueue— does. It also will be >> more efficient with thread management since it has an intimate understanding >> of the capabilities of the processor, etc., and will work to do the “right >> thing” on a per-device basis. >> >> By leveraging NSOperationQueue and then keeping each of the queue operations >> focused on a single file then you’re not complicating the management of what >> to do next since most of that is handled for you. Let NSManagedObjectQueue >> do the heavy lifting (scheduling work) and focus on your part of the task >> (performing the work). >> >> Steve >> >>> On Aug 16, 2022, at 8:41 AM, Gabriel Zachmann wrote: >>> >>> That is a good idea. Thanks a lot! >>> >>> Maybe, I can turn this into more fine-grained, dynamic load balancing (or >>> latency hiding), as follows: >>> create a number of threads (workers); >>> as soon as a worker is finished with their "current" image, it gets the >>> next one (a piece of work) out of the list, processes it, and stores the >>> iso_date in the output array (dates_and_times). >>> Both accesses to the pointer to the currently next piece of work, and the >>> output array would need to be made exclusive, of course. >>> >>> Best regards, Gabriel >> ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Scaling a UIImage
On Nov 2, 2020, at 5:59 PM, Carl Hoefs via Cocoa-dev
wrote:
> I have an iOS app that interacts with a macOS server process. The iOS app
> takes a 3264x2448 camera image, scales it to 640x480 pixels, and makes a JPEG
> representation of it to send to the server:
I have code that does pretty much the same thing, in Swift though so you’ll
need to convert the API calls to ObjC. Since you’re taking a picture, you could
use the AVCapturePhoto directly.
let capture : AVCapturePhoto
private lazy var context = CIContext()
lazy var remotePreviewImage: Data? = {
guard let cgImage =
self.capture.cgImageRepresentation()?.takeRetainedValue() else { return nil }
var baseImg = CIImage(cgImage: cgImage)
if let orientation = self.capture.metadata[
String(kCGImagePropertyOrientation) ] as? Int32 {
baseImg = baseImg.oriented(forExifOrientation: orientation)
}
let scalePct = [800.0 / baseImg.extent.size.width, 800.0 /
baseImg.extent.size.height].max() ?? 0.3
let transformedImg = baseImg.transformed(by: CGAffineTransform(scaleX:
scalePct, y: scalePct))
print("generated remote preview image \(transformedImg.extent.size)")
let colorspace : CGColorSpace = baseImg.colorSpace ??
CGColorSpace(name: CGColorSpace.sRGB)!
let compressionKey = CIImageRepresentationOption(rawValue:
kCGImageDestinationLossyCompressionQuality as String)
let data = self.context.jpegRepresentation(of: transformedImg,
colorSpace: colorspace,
options: [compressionKey :
0.6])
print("photo generated preview \(data?.count ?? 0) bytes")
return data
}()
I had a previous version that used ImageIO. I don’t remember why I switched but
I still had the commented code hanging around.
//lazy var remotePreviewImage: Data? = {
//guard let data = self.capture.fileDataRepresentation() else { return
nil }
//guard let src = CGImageSourceCreateWithData(data as NSData, nil) else
{ return nil }
//let thumbOpts = [
//kCGImageSourceCreateThumbnailFromImageAlways: true,
//kCGImageSourceCreateThumbnailWithTransform: true,
//kCGImageSourceThumbnailMaxPixelSize: 800,
//] as [CFString : Any]
//
//if let cgImage = CGImageSourceCreateThumbnailAtIndex(src, 0,
thumbOpts as CFDictionary) {
//// create jpg data
//let data = NSMutableData()
//
//if let dest = CGImageDestinationCreateWithData(data, kUTTypeJPEG,
1, nil) {
//CGImageDestinationAddImage(dest, cgImage,
[kCGImageDestinationLossyCompressionQuality: 0.6] as CFDictionary)
//CGImageDestinationFinalize(dest)
//}
//print("getPhoto generated preview \(data.count) bytes for
RemoteCapture")
//return data as Data
//}
//return nil
//}()
Jim Crate
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Launching another app from mine
On Jun 15, 2020, at 2:30 PM, Gabriel Zachmann via Cocoa-dev wrote: > > I would like to launch application B from my application A using Swift. > Both applications are created, compiled, signed, and notarized by me. > Challenge: no user intervention should be necessary when launching B from A. > > Is that possible? NSWorkspace does this. Look at the launchApplication method. You can get the app name from the bundle identifier using the NSWorkSpace URLForApplicationWithBundleIdentifier method. Jim Crate ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Screensaver can capture mouse events under Catalina
On May 25, 2020, at 3:43 PM, Gabriel Zachmann via Cocoa-dev wrote: > I can confirm: this is indeed critical. Otherwise, the .saver does not > receive mouse events. > > Unfortunately, it seems that I still cannot get key events. If you are trying to get events from arrow or modifier keys, you have to use keyDown and not keyUp. But you may not be able to get any key events in a screensaver for security reasons. Jim Crate ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Concurrent loading of images ?
On May 14, 2020, at 11:13 AM, Gabriel Zachmann via Cocoa-dev
wrote:
>
> I thought i'm doing that with this code:
>
> -animateOneframe:
> // essentially, this gets called when the next image is to be displayed
> current_image_ = prefetched_image_; // these are CGImageRef's
> dispatch_async( prefetch_queue_, ^{
> [self loadNextImage];// stores next image in
> prefetched_image_
> } );
> create a new CALayer,
> assign the current_image_ as content,
> create an animation for the layer,
> add the layer to the layer hierarchy
Since this stutter does not occur when cycling back through images that have
already been assigned to the layer, it’s possible that if the size of the
CGImageRef does not match the size of the CALayer, there may be some kind of
render occuring, especially if the image is being scaled to fit the layer. Do
images much larger than the CALayer stutter more?
Also, if you’re resizing, you’ll want to use CIImage’s CILanczosScaleTransform
filter. Simply scaling a using a transform will give you a much blurrier image.
Jim Crate
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Concurrent loading of images ?
On May 8, 2020, at 6:00 PM, Gabriel Zachmann via Cocoa-dev wrote: > >> Sure. Just be aware that if you're using NSImage, simply loading an NSImage >> does not rasterize it; the class tries to be 'lazy' about doing work. So >> your background task should explicitly render it, e.g. into an >> NSBitmapImageRep. > > I am using CGImageSourceCreateImageAtIndex and related functions. > But I am still unclear which function actually takes the time. > Maybe it is only when I create the CALayer like this? > > CALayer * imgLayer= [CALayer layer]; > imgLayer.contents = nsimage; Reading a 50MB image will definitely take some time, so preloading the image will definitely help. I’ve usually used an NSOperationQueue but my apps working with images can sometimes be loading hundreds of images and NSOperationQueue has an option to limit the number of operations run concurrently. Since you’ll only be loading the next image then a single GCD dispatch would work just as well. Also, if you’re already getting a CGImageRef using CGImageSourceCreateImageAtIndex, why not just set imgLayer.contents to the CGImageRef? Jim Crate ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Customizing the Notarization Workflow fails
On Apr 25, 2020, at 10:12 AM, Gabriel Zachmann via Cocoa-dev
wrote:
>
> I am trying to follow these instructions:
>
> https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow?language=objc
>
> However, I can't even achieve the first step ("Export a Package for
> Notarization").
I’m not sure how you would script the export, but if you go to menu option
Window -> Organizer, you can see your archives. From there you can export.
Apps have the button “Distribute App”, and a command line program has the
button “Distrubute Content”. Clicking that button lets you export the archive
or the app/built products. Those are exported into a folder named with the date
by default.
Once you have the product exported, the directions on that page work fine for
building a script to handle signing/notarizing.
Jim Crate
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
