On 12/17/2010 5:03 PM, Gregg Tavares (wrk) wrote:
On Fri, Dec 17, 2010 at 4:16 PM, Charles Pritchard <[email protected]
<mailto:[email protected]>> wrote:
We're actively developing such functionality.
The limit per directory is for the sake of the os file system. If
you want to create a data store, use indexedDB or use standard
file system practices (create a subdirectory tree).
I think you're missing the point. If I have a folder with 6000 files
on some server and I want to mirror that through the FileAPI to the
user's local machine I'm screwed. I can't **mirror** it if it's not a
mirror.
I'm not missing the point. I'm actively developing an app that downloads
images from photo sites.
A strict Mirror (lets use a capital M) is something you're not going to
pull off with the File System API at this time.
You can't set meta data, like permissions/flags and
modification/creation dates. Developing a Mirror is not feasible with
the current API.
You can't create a direct Mirror, one which would work seamlessly with
"rsync".
You can "mirror", the data on a remote server, and check that the data
already exists on your file system,
by using a subdirectory system, much like the two level directory
structure that many cache apps use (like Squid).
Disk space availability (quota) is an issue no matter what
happens. When downloading 1000 images, you'll still only be doing
so x at a time.
I don't see the point you're trying to make here. I don't know the
size of the images before hand. Many internet APIs make getting the
sizes prohibitively expensive. One REST call per file. So before I can
download a single file I'd have to issue 1000 REST XHRs to find the
sizes of the files (waiting the several minutes for that to complete)
before I can ask the user for the space needed. That's not a good
user experience. If on the other hand the user can give me permission
for unlimited space then I can just start downloading the files
without having to find out their ultimate size.
I suppose I can just request 1 terabyte up front. Or query how much
space is free and ask for all of it.
Yes, that's correct, you'd hope the space is available.
When you run out of space, you can use a limited amount of RAM while
waiting.
There are few resource management APIs available for memory/bandwidth
hungry applications.
My point was that these questions you're bringing up are common to all
cross-platform applications.
Regarding your issue of "a ray tracing program" : You'd also want to
either: create sub directories,
or simply create a large file with its own methods.
This issues are inherent in the design of any application of
scale. At this point, the file system API does work for the use
case you're describing.
It'd be nice to see Blob storage better integrated with Web
Storage Apis. Ian has already spoken to this, but no followers yet
(afaik).
-Charles
On Dec 17, 2010, at 3:34 PM, "Gregg Tavares (wrk)"
<[email protected] <mailto:[email protected]>> wrote:
> Sorry if this has been covered before.
>
> I've been wanting to write an app to download images from photo
sites and I'm wondering if this use case has been considered for
the FileAPI wrt Directories and System.
>
> If I understand the current spec it seems like their are some
possible issues.
>
> #1) The spec says there is a 5000 file limit or directory.
>
> #2) The spec requires an app to specify how much storage it needs
>
> I understand the desire for the limits. What about an app being
able to request unlimited storage and unlimited files? The UA can
decide how to present something to the user to grant permission if
they want.
>
> Arguments against leaving it as is:
>
> The 5000 file limit seems arbitrary. Any app that hits that
limit will probably require serious re-engineering to work around
it. It will not only have to some how describe a mapping between
files on a server that may not have that limit, it also has the
issue the user might have something organized that way and will
require the user to re-organize. I realize that 5000 is a large
number. I'm sure the author of csh though 1700 entires in a glob
was a reasonable limit as well. We all know how well that turned
out :-( It's easy to imagine a video editing app that edits and
composites still images. If there are a few layers and 1 image per
layer it could easily add up to more than 5000 files in a single
folder.
>
> The size limit also has issues. For many apps the size limit
will be no problem but for others.... Example: You make ray
tracing program, it traces each frame and saves the files to disc.
When it runs out of room, what is its options? (1) fail. (2)
request a file system of a larger size. The problem is (2) will
require user input. Imagine you start your render before you leave
work expecting it to finish by the time you get in only to find
that 2 hours after you left the UA popped up a confirmation "this
app requests more space"
>
> The same would be true for an image downloading app. You start
the app to download a 1000 images, not knowing their sizes before
hand. You start it and leave. Half way through the UA puts up a
conformation asking for more space.
>
> Have these use cases already come up and been rejected?
>
>
>
