I think it might change things for how much time you want the data to be available for desktop clients.
Using S3, IIUC, sounds nice because you don't worry about files size and you can specify policies on S3 to archive to glacier automatically. On Sunday, August 5, 2018, Daniel Russel <drus...@gmail.com> wrote: > Sorry, I wasn't very clear (in part because it is early stages and a bit > muddled in my head). We have batch jobs that are driven by desktop > software, which produces the inputs and then wants to collect the outputs > afterwards. The data can be largish (a few gigabytes of binary output). So > we need somewhere to put the data for pods doing the computation to read it > from, and then a place to put the output that the desktop client can > collect it from. > > We had thought it would be nice to have the storage be within the > kubernetes cluster, and so only have one set of things to clean up once the > data has been extracted. However, we are starting just by sending the data > through s3 and then cleaning it up separately. > > As you can probably tell, we are just getting started and trying to figure > out the space and mostly trying to look for standard patterns to adopt. > Thanks. > --Daniel > > On Fri, Aug 3, 2018 at 5:14 PM Rodrigo Campos <rodr...@sdfg.com.ar> wrote: > >> I'm not sure what would work for you. A configmap created for that job >> only? Or >> a secret, that can have binary data? >> >> Or just an URL for input and log to stdout the output? Or use an S3 >> bucket for >> input/output? >> >> Not sure I understand the no cleanup part. If you want the output, then >> you want >> the files in some place, right? So they can't be deleted until you >> processed or >> something, at least. Sorry, but not sure I follow :) >> >> On Fri, Aug 03, 2018 at 10:29:26AM -0700, Daniel Russel wrote: >> > Hi-- Any good solutions for job input and output? I'm running into the >> same >> > problem and was looking at spinning up a temp minio ( >> https://www.minio.io/) >> > instance (without persistent store) or something along those lines, but >> > simpler solutions would be nicer. Mostly the goal is to avoid having to >> > separately clean up data/persistent storage afterwards. >> > >> > On Monday, May 21, 2018 at 8:55:17 PM UTC-7, bimar...@google.com wrote: >> > > >> > > I am setting up a go application that will create jobs. For each job >> > > execution, I need to pass a blob of parameter data into the job >> container, >> > > and get a blob back out. I'd like to avoid a requirement for database >> > > access in the job. >> > > >> > > Volumes seem awfully complicated, and, anyway, I don't see a >> > > straightforward way to get a files on and off of a GCE PD volume. I >> suppose >> > > I'm missing something simple, or that there's some other volume type >> that >> > > would be more appropriate. >> > > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Kubernetes user discussion and Q&A" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to kubernetes-users+unsubscr...@googlegroups.com. >> > To post to this group, send email to kubernetes-users@googlegroups.com. >> > Visit this group at https://groups.google.com/group/kubernetes-users. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/kubernetes-users/DHZdfJ3iBLg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> kubernetes-users+unsubscr...@googlegroups.com. >> To post to this group, send email to kubernetes-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
