Definitely no need to apologize -- I really haven't explained very well. We have some real-world data feeds that work on slightly different timescales -- some are daily reports, archived for a relatively long time remotely (but copied to campus daily), some are updated & polled every minute, and cache for a few days remotely, while another can be polled every minute or so, but only "current" data is ever available.
On our side, it is very useful to have deep histories available, so that folks can, say, get at a certain day or month of data from a day of interest in the past. Another application requires very-current data only, and doesn't care about anything more than a coup,e of minutes old. "Bag of Holding" was a d&d reference -- a magical bag whose inside is much bigger than its outside. (Like a TARDIS.) I should have written "rsync -av" in which files are never deleted from the destination. Anything new in the source gets copied to the destination, but he destination copy stays even if the source file disappears. This data usually comes in on Linux systems, but I did copy a big chunk over to a Mac to get to bDrive. -Greg Le lundi 5 octobre 2015, Richard DESHONG <rdesh...@berkeley.edu <javascript:_e(%7B%7D,'cvml','rdesh...@berkeley.edu');>> a écrit : > Sorry Greg, I guess I misunderstood your original post. It sounded like > you have a local library (was that the "Bag of Holding") that you wanted to > put "into the cloud". Once it is in the cloud (bDrive or Box, in this > case), are you saying that you don't want to have the local library? > > I was confused because you mentioned "additive rsync" which implies staff > are putting the files into a local library and then you want to have those > uploaded to the cloud. At least, that's the way I saw it. > > On Mon, Oct 5, 2015 at 5:02 PM, Greg Merritt <gmerr...@berkeley.edu> > wrote: > >> Too much data to stick in a local gDrive folder, really...want to have no >> local long-term archive on a random desktop, if you dig....maybe for a >> one-time initial backfill, but then would want to purge from >> local....hmmm..... >> >> Thanks for the tip on the sync index size! >> >> -Greg >> >> On Mon, Oct 5, 2015 at 4:54 PM, Richard DESHONG <rdesh...@berkeley.edu> >> wrote: >> >>> Hi Greg, >>> It seems like Google Drive is exactly what you want. Install it, move >>> all your files into the folder, and GD will churn away in the background >>> uploading all of the files. Moving forward, just drop new files in, and GD >>> will sync those. >>> >>> I have, in the past, seen forum issues with the sync index size - that >>> is, when the file count gets too large, the sync process fails. So you'd >>> need to research that first. >>> >>> On Mon, Oct 5, 2015 at 4:43 PM, Greg Merritt <gmerr...@berkeley.edu> >>> wrote: >>> >>>> Ooh, no! I'll check it out. This may indeed be helpful after we get the >>>> historical backlog up there. >>>> >>>> Thanks! >>>> >>>> -Greg >>>> >>>> On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <mch...@haas.berkeley.edu >>>> > wrote: >>>> >>>>> Hi Greg, >>>>> >>>>> Have you checked this out? >>>>> >>>>> http://www.duplicati.com/ >>>>> >>>>> Full disclaimer, I haven't used it myself, but it appears you can run >>>>> incremental transfers, which should at least give you "rsync"-like >>>>> functionality after an initial Full backup. >>>>> >>>>> Thanks, >>>>> >>>>> *Michael Chung* >>>>> Systems Administrator >>>>> Enterprise Computing & Service Management >>>>> Haas School of Business, UC Berkeley >>>>> Student Services Building, Room S300D >>>>> Berkeley, CA 94720-1900 >>>>> Tele: 510-643-3887 >>>>> >>>>> Typical Office Schedule >>>>> Offsite: M-F >>>>> At Haas: On-demand >>>>> >>>>> On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <gmerr...@berkeley.edu> >>>>> wrote: >>>>> >>>>>> Hello Micronet, >>>>>> >>>>>> We'd love to use our SPA bDrive as an archive location for a >>>>>> infrequently and selectively accessed Bag of Holding for data sets of >>>>>> millions of small, individual, already-compressed ~10K files. >>>>>> >>>>>> Folder drag & drop into the bDrive browser interface on Mac Chrome >>>>>> leads to unhappiness, as the browser crashes after a while. >>>>>> >>>>>> Rather than tar-ing them, or dragging them over in subgroups, is >>>>>> there a more direct way to send over the lot? We'd rather not do the >>>>>> desktop synching with these, unless there's a super-elegant way to use >>>>>> that >>>>>> mechanism. >>>>>> >>>>>> (Would *really* love to do an additive rsync here in the long run, >>>>>> but I'm probably just dreaming here.) >>>>>> >>>>>> Thanks for any tips! >>>>>> >>>>>> -Greg >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------- >>>>>> The following was automatically added to this message by the list >>>>>> server: >>>>>> >>>>>> To learn more about Micronet, including how to subscribe to or >>>>>> unsubscribe from its mailing list and how to find out about upcoming >>>>>> meetings, please visit the Micronet Web site: >>>>>> >>>>>> http://micronet.berkeley.edu >>>>>> >>>>>> Messages you send to this mailing list are public and world-viewable, >>>>>> and the list's archives can be browsed and searched on the Internet. >>>>>> This >>>>>> means these messages can be viewed by (among others) your bosses, >>>>>> prospective employers, and people who have known you in the past. >>>>>> >>>>>> ANNOUNCEMENTS: To send announcements to the Micronet list, please use >>>>>> the micronet-annou...@lists.berkeley.edu list. >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------- >>>> The following was automatically added to this message by the list >>>> server: >>>> >>>> To learn more about Micronet, including how to subscribe to or >>>> unsubscribe from its mailing list and how to find out about upcoming >>>> meetings, please visit the Micronet Web site: >>>> >>>> http://micronet.berkeley.edu >>>> >>>> Messages you send to this mailing list are public and world-viewable, >>>> and the list's archives can be browsed and searched on the Internet. This >>>> means these messages can be viewed by (among others) your bosses, >>>> prospective employers, and people who have known you in the past. >>>> >>>> ANNOUNCEMENTS: To send announcements to the Micronet list, please use >>>> the micronet-annou...@lists.berkeley.edu list. >>>> >>>> >>> >>> >>> -- >>> Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley >>> 164 Chavez Student Center, Berkeley, CA, 94720-4220 >>> 510-642-5123 asc.berkeley.edu >>> >>> >>> ------------------------------------------------------------------------- >>> The following was automatically added to this message by the list server: >>> >>> To learn more about Micronet, including how to subscribe to or >>> unsubscribe from its mailing list and how to find out about upcoming >>> meetings, please visit the Micronet Web site: >>> >>> http://micronet.berkeley.edu >>> >>> Messages you send to this mailing list are public and world-viewable, >>> and the list's archives can be browsed and searched on the Internet. This >>> means these messages can be viewed by (among others) your bosses, >>> prospective employers, and people who have known you in the past. >>> >>> ANNOUNCEMENTS: To send announcements to the Micronet list, please use >>> the micronet-annou...@lists.berkeley.edu list. >>> >>> >> > > > -- > Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley > 164 Chavez Student Center, Berkeley, CA, 94720-4220 > 510-642-5123 asc.berkeley.edu > -- Envoyé avec Gmail Mobile
------------------------------------------------------------------------- The following was automatically added to this message by the list server: To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site: http://micronet.berkeley.edu Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet. This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past. ANNOUNCEMENTS: To send announcements to the Micronet list, please use the micronet-annou...@lists.berkeley.edu list.
