- Revision
- 283592
- Author
- akeer...@apple.com
- Date
- 2021-10-05 17:42:35 -0700 (Tue, 05 Oct 2021)
Log Message
[iOS] Transcode videos selected from UIImagePickerController
https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678
Reviewed by Tim Horton.
Source/WebCore:
* en.lproj/Localizable.strings:
Add a localizable string for the message displayed while transcoding
video.
Source/WebCore/PAL:
Add AVFoundation API needed to transcode video.
* pal/cocoa/AVFoundationSoftLink.h:
* pal/cocoa/AVFoundationSoftLink.mm:
Source/WebKit:
File inputs on iOS allow users to choose images/videos from the system
photo picker, using UIImagePickerController. In single selection mode,
UIImagePickerController transcodes the selected video to H.264. However,
in multiple selection mode, video is not transcoded and is left in its
original format.
Today, videos on most iOS devices are encoded with HEVC by default.
However, some sites, such as Twitter, only accept H.264 encoded video.
Thus, the current video upload behavior is problematic, as users may be
unable to upload video.
Unfortunately, the photo picking functionality of UIImagePickerController
is deprecated. The best solution would be to adopt PHPickerViewController,
the replacement API, which performs transcoding when retrieving selected
items (regardless of single/multiple selection). However,
PHPickerViewController currently lacks other functionality that WebKit
requires, preventing adoption.
Consequently, the short term solution is to transcode the videos in
WebKit, ensuring H.264 encoded video is always provided to sites. See
below for implementation details.
* Platform/spi/ios/PhotosUISPI.h: Added.
* UIProcess/ios/forms/WKFileUploadPanel.mm:
(-[_WKFileUploadItem setFileURL:]):
Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
(-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
and the display of progress UI. Transcoding is performed serially
(one video at a time), but occurs off the main thread.
(-[WKFileUploadMediaTranscoder start]):
Begin transcoding. Run a timer to update the progress UI, as
AVAssetExportSession does not provide progress updates on its own.
The progress UI is implemented using PUActivityProgressController, to
match system Photos UI.
(-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
Transcode a single video, using AVAssetExportSession. If transcoding
fails for any reason, the original video is used as a fallback.
Transcoding can also be cancelled using the progress UI, in which case
no more videos are processed.
(-[WKFileUploadMediaTranscoder _finishedProcessing]):
(-[WKFileUploadMediaTranscoder _dismissProgress]):
(-[WKFileUploadMediaTranscoder _updateProgress:]):
(-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
(-[WKFileUploadPanel _chooseMediaItems:]):
Refactor the common aspects of single/multiple media selection into a
single method.
(-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
(-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
(-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
Remove redundant platform conditional.
(-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
If any videos were selected, transcode them prior to uploading.
* WebKit.xcodeproj/project.pbxproj:
Modified Paths
Added Paths
Diff
Modified: trunk/Source/WebCore/ChangeLog (283591 => 283592)
--- trunk/Source/WebCore/ChangeLog 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebCore/ChangeLog 2021-10-06 00:42:35 UTC (rev 283592)
@@ -1,3 +1,16 @@
+2021-10-05 Aditya Keerthi <akeer...@apple.com>
+
+ [iOS] Transcode videos selected from UIImagePickerController
+ https://bugs.webkit.org/show_bug.cgi?id=230639
+ rdar://79665678
+
+ Reviewed by Tim Horton.
+
+ * en.lproj/Localizable.strings:
+
+ Add a localizable string for the message displayed while transcoding
+ video.
+
2021-10-05 Chris Dumez <cdu...@apple.com>
ASSERT(m_callback->hasCallback()) under IntersectionObserver::notify()
Modified: trunk/Source/WebCore/PAL/ChangeLog (283591 => 283592)
--- trunk/Source/WebCore/PAL/ChangeLog 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebCore/PAL/ChangeLog 2021-10-06 00:42:35 UTC (rev 283592)
@@ -1,3 +1,16 @@
+2021-10-05 Aditya Keerthi <akeer...@apple.com>
+
+ [iOS] Transcode videos selected from UIImagePickerController
+ https://bugs.webkit.org/show_bug.cgi?id=230639
+ rdar://79665678
+
+ Reviewed by Tim Horton.
+
+ Add AVFoundation API needed to transcode video.
+
+ * pal/cocoa/AVFoundationSoftLink.h:
+ * pal/cocoa/AVFoundationSoftLink.mm:
+
2021-10-05 Ayumi Kojima <ayumi_koj...@apple.com>
Unreviewed, reverting r283339.
Modified: trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h (283591 => 283592)
--- trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h 2021-10-06 00:42:35 UTC (rev 283592)
@@ -38,6 +38,7 @@
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCache)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCollection)
+SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetExportSession)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetImageGenerator)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReader)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReaderSampleReferenceOutput)
@@ -190,6 +191,8 @@
#define AVVideoCodecTypeHEVCWithAlpha PAL::get_AVFoundation_AVVideoCodecTypeHEVCWithAlpha()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeMPEG4, NSString *)
#define AVFileTypeMPEG4 PAL::get_AVFoundation_AVFileTypeMPEG4()
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *)
+#define AVFileTypeQuickTimeMovie PAL::get_AVFoundation_AVFileTypeQuickTimeMovie()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecKey, NSString *)
#define AVVideoCodecKey PAL::get_AVFoundation_AVVideoCodecKey()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecH264, NSString *)
@@ -336,4 +339,7 @@
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioPCMBuffer)
#endif // PLATFORM(COCOA)
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *)
+#define AVAssetExportPresetHighestQuality PAL::get_AVFoundation_AVAssetExportPresetHighestQuality()
+
#endif // USE(AVFOUNDATION)
Modified: trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm (283591 => 283592)
--- trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm 2021-10-06 00:42:35 UTC (rev 283592)
@@ -63,6 +63,7 @@
#endif
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetCache, PAL_EXPORT)
+SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportSession, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetImageGenerator, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetReader, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetWriter, PAL_EXPORT)
@@ -133,6 +134,7 @@
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasConnectedNotification, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasDisconnectedNotification, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeMPEG4, NSString *, PAL_EXPORT)
+SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResize, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspect, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspectFill, NSString *, PAL_EXPORT)
@@ -253,4 +255,6 @@
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioPCMBuffer, PAL_EXPORT)
#endif
+SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *, PAL_EXPORT)
+
#endif // USE(AVFOUNDATION)
Modified: trunk/Source/WebCore/en.lproj/Localizable.strings (283591 => 283592)
--- trunk/Source/WebCore/en.lproj/Localizable.strings 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebCore/en.lproj/Localizable.strings 2021-10-06 00:42:35 UTC (rev 283592)
@@ -754,6 +754,9 @@
/* Description of the PostScript type supported by the PDF pseudo plug-in. Visible in the Installed Plug-ins page in Safari. */
"PostScript" = "PostScript";
+/* Title for file upload progress view */
+"Preparing (file upload)" = "Preparing…";
+
/* Title for Quick Look action button */
"Quick Look" = "Quick Look";
Modified: trunk/Source/WebKit/ChangeLog (283591 => 283592)
--- trunk/Source/WebKit/ChangeLog 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebKit/ChangeLog 2021-10-06 00:42:35 UTC (rev 283592)
@@ -1,3 +1,83 @@
+2021-10-05 Aditya Keerthi <akeer...@apple.com>
+
+ [iOS] Transcode videos selected from UIImagePickerController
+ https://bugs.webkit.org/show_bug.cgi?id=230639
+ rdar://79665678
+
+ Reviewed by Tim Horton.
+
+ File inputs on iOS allow users to choose images/videos from the system
+ photo picker, using UIImagePickerController. In single selection mode,
+ UIImagePickerController transcodes the selected video to H.264. However,
+ in multiple selection mode, video is not transcoded and is left in its
+ original format.
+
+ Today, videos on most iOS devices are encoded with HEVC by default.
+ However, some sites, such as Twitter, only accept H.264 encoded video.
+ Thus, the current video upload behavior is problematic, as users may be
+ unable to upload video.
+
+ Unfortunately, the photo picking functionality of UIImagePickerController
+ is deprecated. The best solution would be to adopt PHPickerViewController,
+ the replacement API, which performs transcoding when retrieving selected
+ items (regardless of single/multiple selection). However,
+ PHPickerViewController currently lacks other functionality that WebKit
+ requires, preventing adoption.
+
+ Consequently, the short term solution is to transcode the videos in
+ WebKit, ensuring H.264 encoded video is always provided to sites. See
+ below for implementation details.
+
+ * Platform/spi/ios/PhotosUISPI.h: Added.
+ * UIProcess/ios/forms/WKFileUploadPanel.mm:
+ (-[_WKFileUploadItem setFileURL:]):
+
+ Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
+
+ (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
+
+ Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
+ and the display of progress UI. Transcoding is performed serially
+ (one video at a time), but occurs off the main thread.
+
+ (-[WKFileUploadMediaTranscoder start]):
+
+ Begin transcoding. Run a timer to update the progress UI, as
+ AVAssetExportSession does not provide progress updates on its own.
+
+ The progress UI is implemented using PUActivityProgressController, to
+ match system Photos UI.
+
+ (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
+
+ Transcode a single video, using AVAssetExportSession. If transcoding
+ fails for any reason, the original video is used as a fallback.
+ Transcoding can also be cancelled using the progress UI, in which case
+ no more videos are processed.
+
+ (-[WKFileUploadMediaTranscoder _finishedProcessing]):
+ (-[WKFileUploadMediaTranscoder _dismissProgress]):
+ (-[WKFileUploadMediaTranscoder _updateProgress:]):
+ (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
+ (-[WKFileUploadPanel _chooseMediaItems:]):
+
+ Refactor the common aspects of single/multiple media selection into a
+ single method.
+
+ (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
+ (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
+ (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
+ (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
+ (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
+
+ Remove redundant platform conditional.
+
+ (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
+
+ If any videos were selected, transcode them prior to uploading.
+
+ * WebKit.xcodeproj/project.pbxproj:
+
2021-10-05 Alex Christensen <achristen...@webkit.org>
Add an entitlement check to only allow AdAttributionDaemon to be connected to by the network process
Added: trunk/Source/WebKit/Platform/spi/ios/PhotosUISPI.h (0 => 283592)
--- trunk/Source/WebKit/Platform/spi/ios/PhotosUISPI.h (rev 0)
+++ trunk/Source/WebKit/Platform/spi/ios/PhotosUISPI.h 2021-10-06 00:42:35 UTC (rev 283592)
@@ -0,0 +1,49 @@
+/*
+ * Copyright (C) 2021 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#if USE(APPLE_INTERNAL_SDK)
+
+#import <PhotosUI/PUActivityProgressController.h>
+
+#else
+
+#import "UIKitSPI.h"
+
+@interface PUActivityProgressController : NSObject
+
+@property (nonatomic, copy) NSString *title;
+
+@property (nonatomic, copy) void (^cancellationHandler)(void);
+
+@property (nonatomic, readonly) BOOL isCancelled;
+
+- (void)setFractionCompleted:(double)fractionCompleted;
+
+- (void)showAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;
+- (void)hideAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;
+
+@end
+
+#endif // USE(APPLE_INTERNAL_SDK)
Modified: trunk/Source/WebKit/UIProcess/ios/forms/WKFileUploadPanel.mm (283591 => 283592)
--- trunk/Source/WebKit/UIProcess/ios/forms/WKFileUploadPanel.mm 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebKit/UIProcess/ios/forms/WKFileUploadPanel.mm 2021-10-06 00:42:35 UTC (rev 283592)
@@ -32,6 +32,7 @@
#import "APIData.h"
#import "APIOpenPanelParameters.h"
#import "APIString.h"
+#import "PhotosUISPI.h"
#import "UIKitSPI.h"
#import "UserInterfaceIdiom.h"
#import "WKContentViewInteraction.h"
@@ -45,11 +46,17 @@
#import <UniformTypeIdentifiers/UniformTypeIdentifiers.h>
#import <WebCore/LocalizedStrings.h>
#import <WebCore/MIMETypeRegistry.h>
+#import <wtf/MainThread.h>
#import <wtf/OptionSet.h>
#import <wtf/RetainPtr.h>
#import <wtf/WeakObjCPtr.h>
#import <wtf/text/StringView.h>
+#import <pal/cocoa/AVFoundationSoftLink.h>
+
+SOFT_LINK_FRAMEWORK(PhotosUI)
+SOFT_LINK_CLASS(PhotosUI, PUActivityProgressController)
+
using namespace WebKit;
enum class WKFileUploadPanelImagePickerType : uint8_t {
@@ -79,7 +86,6 @@
@interface _WKFileUploadItem : NSObject
- (instancetype)initWithFileURL:(NSURL *)fileURL;
@property (nonatomic, readonly, getter=isVideo) BOOL video;
-@property (nonatomic, readonly) NSURL *fileURL;
@property (nonatomic, readonly) RetainPtr<UIImage> displayImage;
@end
@@ -109,6 +115,11 @@
return _fileURL.get();
}
+- (void)setFileURL:(NSURL *)fileURL
+{
+ _fileURL = fileURL;
+}
+
- (RetainPtr<UIImage>)displayImage
{
ASSERT_NOT_REACHED();
@@ -153,10 +164,165 @@
@end
+#pragma mark - WKFileUploadMediaTranscoder
+@interface WKFileUploadMediaTranscoder : NSObject
+
+- (instancetype)initWithItems:(NSArray *)items videoCount:(NSUInteger)videoCount completionHandler:(WTF::Function<void(NSArray<_WKFileUploadItem *> *)>&&)completionHandler;
+
+- (void)start;
+
+@end
+
+@implementation WKFileUploadMediaTranscoder {
+ RetainPtr<NSTimer> _progressTimer;
+ RetainPtr<PUActivityProgressController> _progressController;
+ RetainPtr<AVAssetExportSession> _exportSession;
+ RetainPtr<NSArray<_WKFileUploadItem *>> _items;
+ RetainPtr<NSString> _temporaryDirectoryPath;
+
+ // Only called if the transcoding is not cancelled.
+ WTF::Function<void(NSArray<_WKFileUploadItem *> *)> _completionHandler;
+
+ NSUInteger _videoCount;
+ NSUInteger _processedVideoCount;
+}
+
+- (instancetype)initWithItems:(NSArray<_WKFileUploadItem *> *)items videoCount:(NSUInteger)videoCount completionHandler:(WTF::Function<void(NSArray<_WKFileUploadItem *> *)>&&)completionHandler
+{
+ if (!(self = [super init]))
+ return nil;
+
+ _items = items;
+ _processedVideoCount = 0;
+ _videoCount = videoCount;
+
+ _completionHandler = WTFMove(completionHandler);
+
+ return self;
+}
+
+- (void)start
+{
+ _progressController = adoptNS([allocPUActivityProgressControllerInstance() init]);
+ [_progressController setTitle:WEB_UI_STRING_KEY("Preparing…", "Preparing (file upload)", "Title for file upload progress view")];
+ [_progressController showAnimated:YES allowDelay:YES];
+
+ [_progressController setCancellationHandler:makeBlockPtr([weakSelf = WeakObjCPtr<WKFileUploadMediaTranscoder>(self)] {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf)
+ return;
+
+ [strongSelf->_exportSession cancelExport];
+ [strongSelf _dismissProgress];
+ }).get()];
+
+ _progressTimer = [NSTimer scheduledTimerWithTimeInterval:0.1f target:self selector:@selector(_updateProgress:) userInfo:nil repeats:YES];
+
+ [self _processItemAtIndex:0];
+}
+
+- (void)_processItemAtIndex:(NSUInteger)index
+{
+ if ([_progressController isCancelled])
+ return;
+
+ if (index >= [_items count]) {
+ [self _finishedProcessing];
+ return;
+ }
+
+ _WKFileUploadItem *item = [_items objectAtIndex:index];
+
+ while (!item.isVideo) {
+ index++;
+
+ if (index == [_items count]) {
+ [self _finishedProcessing];
+ return;
+ }
+
+ item = [_items objectAtIndex:index];
+ }
+
+ NSString *temporaryDirectory = [self _temporaryDirectoryCreateIfNecessary];
+ if (!temporaryDirectory) {
+ LOG_ERROR("WKFileUploadMediaTranscoder: Failed to make temporary directory");
+ [self _finishedProcessing];
+ return;
+ }
+
+ NSString *fileName = [item.fileURL.lastPathComponent.stringByDeletingPathExtension stringByAppendingPathExtension:UTTypeQuickTimeMovie.preferredFilenameExtension.uppercaseString];
+ NSString *filePath = [temporaryDirectory stringByAppendingPathComponent:fileName];
+ NSURL *outputURL = [NSURL fileURLWithPath:filePath isDirectory:NO];
+
+ RetainPtr<AVURLAsset> asset = adoptNS([PAL::allocAVURLAssetInstance() initWithURL:item.fileURL options:nil]);
+ _exportSession = adoptNS([PAL::allocAVAssetExportSessionInstance() initWithAsset:asset.get() presetName:AVAssetExportPresetHighestQuality]);
+ [_exportSession setOutputURL:outputURL];
+ [_exportSession setOutputFileType:AVFileTypeQuickTimeMovie];
+
+ [_exportSession exportAsynchronouslyWithCompletionHandler:makeBlockPtr([weakSelf = WeakObjCPtr<WKFileUploadMediaTranscoder>(self), index] () mutable {
+ ensureOnMainRunLoop([weakSelf = WTFMove(weakSelf), index] {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf)
+ return;
+
+ AVAssetExportSessionStatus status = [strongSelf->_exportSession status];
+
+ if (status == AVAssetExportSessionStatusCancelled)
+ return;
+
+ if (status == AVAssetExportSessionStatusCompleted) {
+ _WKFileUploadItem *item = [strongSelf->_items objectAtIndex:index];
+ [item setFileURL:[strongSelf->_exportSession outputURL]];
+ }
+
+ strongSelf->_exportSession = nil;
+
+ strongSelf->_processedVideoCount++;
+ [strongSelf _processItemAtIndex:index + 1];
+ });
+ }).get()];
+}
+
+- (void)_finishedProcessing
+{
+ [self _dismissProgress];
+
+ if (auto completionHandler = std::exchange(_completionHandler, nullptr))
+ completionHandler(_items.get());
+}
+
+- (void)_dismissProgress
+{
+ [_progressTimer invalidate];
+ [_progressController hideAnimated:NO allowDelay:NO];
+}
+
+- (void)_updateProgress:(NSTimer *)timer
+{
+ auto currentSessionProgress = [_exportSession progress];
+ [_progressController setFractionCompleted:(currentSessionProgress + _processedVideoCount) / _videoCount];
+}
+
+- (NSString *)_temporaryDirectoryCreateIfNecessary
+{
+ if (_temporaryDirectoryPath) {
+ BOOL isDirectory = NO;
+ BOOL exists = [[NSFileManager defaultManager] fileExistsAtPath:_temporaryDirectoryPath.get() isDirectory:&isDirectory];
+
+ if (exists && isDirectory)
+ return _temporaryDirectoryPath.get();
+ }
+
+ _temporaryDirectoryPath = FileSystem::createTemporaryDirectory(@"WKVideoUpload");
+ return _temporaryDirectoryPath.get();
+}
+
+@end
+
#pragma mark - WKFileUploadPanel
-
@interface WKFileUploadPanel () <UIPopoverControllerDelegate, UINavigationControllerDelegate, UIImagePickerControllerDelegate, UIDocumentPickerDelegate, UIAdaptivePresentationControllerDelegate
#if USE(UICONTEXTMENU)
, UIContextMenuInteractionDelegate
@@ -172,6 +338,7 @@
CGPoint _interactionPoint;
BOOL _allowMultipleFiles;
BOOL _usingCamera;
+ RetainPtr<WKFileUploadMediaTranscoder> _mediaTranscoder;
RetainPtr<UIImagePickerController> _imagePicker;
RetainPtr<UIViewController> _presentationViewController; // iPhone always. iPad for Fullscreen Camera.
ALLOW_DEPRECATED_DECLARATIONS_BEGIN
@@ -220,6 +387,30 @@
[self _dispatchDidDismiss];
}
+- (void)_chooseMediaItems:(NSArray<_WKFileUploadItem *> *)mediaItems
+{
+ RetainPtr<UIImage> iconImage = nil;
+ NSMutableArray *fileURLs = [NSMutableArray array];
+ NSUInteger videoCount = 0;
+
+ for (_WKFileUploadItem *item in mediaItems) {
+ [fileURLs addObject:item.fileURL];
+
+ if (!iconImage)
+ iconImage = item.displayImage;
+
+ if (item.isVideo)
+ videoCount++;
+ }
+
+ NSUInteger imageCount = mediaItems.count - videoCount;
+
+ NSString *displayString = (imageCount || videoCount) ? [NSString localizedStringWithFormat:WEB_UI_NSSTRING(@"%lu photo(s) and %lu video(s)", "label next to file upload control; parameters are the number of photos and the number of videos"), (unsigned long)imageCount, (unsigned long)videoCount] : nil;
+
+ [self _dismissDisplayAnimated:YES];
+ [self _chooseFiles:fileURLs displayString:displayString iconImage:iconImage.get()];
+}
+
- (void)_chooseFiles:(NSArray *)fileURLs displayString:(NSString *)displayString iconImage:(UIImage *)iconImage
{
NSUInteger count = [fileURLs count];
@@ -729,18 +920,16 @@
if ([self _willMultipleSelectionDelegateBeCalled])
return;
- [self _dismissDisplayAnimated:YES];
-
[self _processMediaInfoDictionaries:@[info]
- successBlock:^(NSArray *processedResults, NSString *displayString) {
- ASSERT([processedResults count] == 1);
- _WKFileUploadItem *result = [processedResults objectAtIndex:0];
- RunLoop::main().dispatch([self, strongSelf = retainPtr(self), result = retainPtr(result), displayString = retainPtr(displayString)] {
- [self _chooseFiles:@[result.get().fileURL] displayString:displayString.get() iconImage:result.get().displayImage.get()];
+ successBlock:^(NSArray<_WKFileUploadItem *> *items) {
+ ASSERT([items count] == 1);
+ ensureOnMainRunLoop([self, strongSelf = retainPtr(self), items = retainPtr(items)] {
+ [self _chooseMediaItems:items.get()];
});
}
failureBlock:^{
- RunLoop::main().dispatch([self, strongSelf = retainPtr(self)] {
+ ensureOnMainRunLoop([self, strongSelf = retainPtr(self)] {
+ [self _dismissDisplayAnimated:YES];
[self _cancel];
});
}
@@ -749,27 +938,13 @@
- (void)imagePickerController:(UIImagePickerController *)imagePicker didFinishPickingMultipleMediaWithInfo:(NSArray *)infos
{
- [self _dismissDisplayAnimated:YES];
-
[self _processMediaInfoDictionaries:infos
- successBlock:^(NSArray *processedResults, NSString *displayString) {
- RetainPtr<UIImage> iconImage = nil;
- NSMutableArray *fileURLs = [NSMutableArray array];
- for (_WKFileUploadItem *result in processedResults) {
- NSURL *fileURL = result.fileURL;
- if (!fileURL)
- continue;
- [fileURLs addObject:result.fileURL];
- if (!iconImage)
- iconImage = result.displayImage;
- }
-
- RunLoop::main().dispatch([self, strongSelf = retainPtr(self), fileURLs = retainPtr(fileURLs), displayString = retainPtr(displayString), iconImage] {
- [self _chooseFiles:fileURLs.get() displayString:displayString.get() iconImage:iconImage.get()];
- });
+ successBlock:^(NSArray<_WKFileUploadItem *> *items) {
+ [self _uploadMediaItemsTranscodingVideo:items];
}
failureBlock:^{
- RunLoop::main().dispatch([self, strongSelf = retainPtr(self)] {
+ ensureOnMainRunLoop([self, strongSelf = retainPtr(self)] {
+ [self _dismissDisplayAnimated:YES];
[self _cancel];
});
}
@@ -784,17 +959,16 @@
#pragma mark - Process UIImagePicker results
-- (void)_processMediaInfoDictionaries:(NSArray *)infos successBlock:(void (^)(NSArray *processedResults, NSString *displayString))successBlock failureBlock:(void (^)(void))failureBlock
+- (void)_processMediaInfoDictionaries:(NSArray *)infos successBlock:(void (^)(NSArray<_WKFileUploadItem *> *processedResults))successBlock failureBlock:(void (^)(void))failureBlock
{
- [self _processMediaInfoDictionaries:infos atIndex:0 processedResults:[NSMutableArray array] processedImageCount:0 processedVideoCount:0 successBlock:successBlock failureBlock:failureBlock];
+ [self _processMediaInfoDictionaries:infos atIndex:0 processedResults:[NSMutableArray array] successBlock:successBlock failureBlock:failureBlock];
}
-- (void)_processMediaInfoDictionaries:(NSArray *)infos atIndex:(NSUInteger)index processedResults:(NSMutableArray *)processedResults processedImageCount:(NSUInteger)processedImageCount processedVideoCount:(NSUInteger)processedVideoCount successBlock:(void (^)(NSArray *processedResults, NSString *displayString))successBlock failureBlock:(void (^)(void))failureBlock
+- (void)_processMediaInfoDictionaries:(NSArray *)infos atIndex:(NSUInteger)index processedResults:(NSMutableArray<_WKFileUploadItem *> *)processedResults successBlock:(void (^)(NSArray<_WKFileUploadItem *> *processedResults))successBlock failureBlock:(void (^)(void))failureBlock
{
NSUInteger count = [infos count];
if (index == count) {
- NSString *displayString = (processedImageCount || processedVideoCount) ? [NSString localizedStringWithFormat:WEB_UI_NSSTRING(@"%lu photo(s) and %lu video(s)", "label next to file upload control; parameters are the number of photos and the number of videos"), (unsigned long)processedImageCount, (unsigned long)processedVideoCount] : nil;
- successBlock(processedResults, displayString);
+ successBlock(processedResults);
return;
}
@@ -803,10 +977,8 @@
index++;
auto uploadItemSuccessBlock = ^(_WKFileUploadItem *uploadItem) {
- NSUInteger newProcessedVideoCount = processedVideoCount + (uploadItem.isVideo ? 1 : 0);
- NSUInteger newProcessedImageCount = processedImageCount + (uploadItem.isVideo ? 0 : 1);
[processedResults addObject:uploadItem];
- [self _processMediaInfoDictionaries:infos atIndex:index processedResults:processedResults processedImageCount:newProcessedImageCount processedVideoCount:newProcessedVideoCount successBlock:successBlock failureBlock:failureBlock];
+ [self _processMediaInfoDictionaries:infos atIndex:index processedResults:processedResults successBlock:successBlock failureBlock:failureBlock];
};
[self _uploadItemFromMediaInfo:info successBlock:uploadItemSuccessBlock failureBlock:failureBlock];
@@ -890,7 +1062,6 @@
return;
}
-#if PLATFORM(IOS_FAMILY)
if (NSURL *imageURL = info[UIImagePickerControllerImageURL]) {
if (!imageURL.isFileURL) {
LOG_ERROR("WKFileUploadPanel: Expected image URL to be a file path, it was not");
@@ -902,7 +1073,6 @@
successBlock(adoptNS([[_WKImageFileUploadItem alloc] initWithFileURL:imageURL]).get());
return;
}
-#endif
UIImage *originalImage = [info objectForKey:UIImagePickerControllerOriginalImage];
if (!originalImage) {
@@ -916,6 +1086,30 @@
[self _uploadItemForJPEGRepresentationOfImage:originalImage successBlock:successBlock failureBlock:failureBlock];
}
+- (void)_uploadMediaItemsTranscodingVideo:(NSArray<_WKFileUploadItem *> *)items
+{
+ auto videoCount = [[items indexesOfObjectsPassingTest:^(_WKFileUploadItem *item, NSUInteger, BOOL*) {
+ return item.isVideo;
+ }] count];
+
+ ensureOnMainRunLoop([self, strongSelf = retainPtr(self), items = retainPtr(items), videoCount] {
+ if (!videoCount) {
+ [self _chooseMediaItems:items.get()];
+ return;
+ }
+
+ _mediaTranscoder = adoptNS([[WKFileUploadMediaTranscoder alloc] initWithItems:items.get() videoCount:videoCount completionHandler:[weakSelf = WeakObjCPtr<WKFileUploadPanel>(self)] (NSArray<_WKFileUploadItem *> *items) {
+ auto strongSelf = weakSelf.get();
+ if (!strongSelf)
+ return;
+
+ [strongSelf _chooseMediaItems:items];
+ }]);
+
+ [_mediaTranscoder start];
+ });
+}
+
- (BOOL)platformSupportsPickerViewController
{
#if PLATFORM(WATCHOS)
Modified: trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj (283591 => 283592)
--- trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj 2021-10-06 00:20:55 UTC (rev 283591)
+++ trunk/Source/WebKit/WebKit.xcodeproj/project.pbxproj 2021-10-06 00:42:35 UTC (rev 283592)
@@ -2033,6 +2033,7 @@
E596DD6A251E71D400C275A7 /* WKContactPicker.h in Headers */ = {isa = PBXBuildFile; fileRef = E596DD68251E71D300C275A7 /* WKContactPicker.h */; };
E5BEF6822130C48000F31111 /* WebDataListSuggestionsDropdownIOS.h in Headers */ = {isa = PBXBuildFile; fileRef = E5BEF6802130C47F00F31111 /* WebDataListSuggestionsDropdownIOS.h */; };
E5CB07DC20E1678F0022C183 /* WKFormColorControl.h in Headers */ = {isa = PBXBuildFile; fileRef = E5CB07DA20E1678F0022C183 /* WKFormColorControl.h */; };
+ E5DEFA6826F8F42600AB68DB /* PhotosUISPI.h in Headers */ = {isa = PBXBuildFile; fileRef = E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */; };
ED82A7F2128C6FAF004477B3 /* WKBundlePageOverlay.h in Headers */ = {isa = PBXBuildFile; fileRef = 1A22F0FF1289FCD90085E74F /* WKBundlePageOverlay.h */; settings = {ATTRIBUTES = (Private, ); }; };
F4094CBD2553053D003D73E3 /* DisplayListReaderHandle.h in Headers */ = {isa = PBXBuildFile; fileRef = F4094CBB255304AF003D73E3 /* DisplayListReaderHandle.h */; };
F4094CBE25530540003D73E3 /* DisplayListWriterHandle.h in Headers */ = {isa = PBXBuildFile; fileRef = F4094CB92553047E003D73E3 /* DisplayListWriterHandle.h */; };
@@ -6092,6 +6093,7 @@
E5BEF6812130C47F00F31111 /* WebDataListSuggestionsDropdownIOS.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = WebDataListSuggestionsDropdownIOS.mm; path = ios/WebDataListSuggestionsDropdownIOS.mm; sourceTree = "<group>"; };
E5CB07DA20E1678F0022C183 /* WKFormColorControl.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = WKFormColorControl.h; path = ios/forms/WKFormColorControl.h; sourceTree = "<group>"; };
E5CB07DB20E1678F0022C183 /* WKFormColorControl.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; name = WKFormColorControl.mm; path = ios/forms/WKFormColorControl.mm; sourceTree = "<group>"; };
+ E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = PhotosUISPI.h; sourceTree = "<group>"; };
ECA680D31E6904B500731D20 /* ExtraPrivateSymbolsForTAPI.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ExtraPrivateSymbolsForTAPI.h; sourceTree = "<group>"; };
ECBFC1DB1E6A4D66000300C7 /* ExtraPublicSymbolsForTAPI.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ExtraPublicSymbolsForTAPI.h; sourceTree = "<group>"; };
F036978715F4BF0500C3A80E /* WebColorPicker.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebColorPicker.cpp; sourceTree = "<group>"; };
@@ -11805,6 +11807,7 @@
2D4AF0882044C3C4006C8817 /* FrontBoardServicesSPI.h */,
A13B3DA1207F39DE0090C58D /* MobileWiFiSPI.h */,
3178AF9720E2A7F80074DE94 /* PDFKitSPI.h */,
+ E5DEFA6726F8F42600AB68DB /* PhotosUISPI.h */,
2D279E1826955768004B3EEB /* PrototypeToolsSPI.h */,
46F38E8B2416E66D0059375A /* RunningBoardServicesSPI.h */,
079D1D9926960CD300883577 /* SystemStatusSPI.h */,
@@ -12481,6 +12484,7 @@
5C298DA01C3DF02100470AFE /* PendingDownload.h in Headers */,
832ED18C1E2FE157006BA64A /* PerActivityStateCPUUsageSampler.h in Headers */,
7AFBD36F21E546F8005DBACB /* PersistencyUtils.h in Headers */,
+ E5DEFA6826F8F42600AB68DB /* PhotosUISPI.h in Headers */,
5CE85B201C88E64B0070BFCE /* PingLoad.h in Headers */,
0F5E200418E77051003EC3E5 /* PlatformCAAnimationRemote.h in Headers */,
2DA049B4180CCCD300AAFA9E /* PlatformCALayerRemote.h in Headers */,
