[google-appengine] Re: AppEngine Dev, QA and production environments, how to manage them and and set it up?

['Diogo Almeida' via Google App Engine]

As per the document [1], the different development environments should be 
managed by setting up a different project for each environment. That said, 
you would manage the different environments with separate repositories and 
could use Cloud Source Repository and Cloud Build to deploy the code to App 
Engine.

I would recommend you explore the documentation of these two products:

1 - Cloud Source Repositories [2][3]: `Google Cloud Source Repositories 
provides Git version control to support collaborative development of an 
application or service, such as those that run on App Engine and Compute 
Engine`. You can automatically deploy an application stored in Cloud Source 
Repositories to App Engine when there is a new commit on your repository. 

2 - Cloud Build [4]: `Cloud Build can import source code from a variety of 
repositories or cloud storage spaces, execute a build to your 
specifications, and produce artifacts such as Docker containers or Java 
archives`.

I also recommend you take a look at this Youtube video [5] where you will 
see a hands-on presentation on how to use both products.

[1] 
https://cloud.google.com/appengine/docs/standard/go/creating-separate-dev-environments
[2] 
https://cloud.google.com/source-repositories/docs/quickstart-triggering-builds-with-source-repositories
[3] https://cloud.google.com/source-repositories/
[4] https://cloud.google.com/cloud-build/
[5] https://www.youtube.com/watch?v=yP9a5gDRUqY


On Thursday, August 22, 2019 at 8:03:33 PM UTC-4, Eduardo Moreira wrote:
>
> Hello Everyone
>
> We have a project that we would like to manage in the most professional 
> way, so we would like to have a DEV environment for developers to upload 
> the project and test in the AppEnginne environment, then we would like to 
> have a QA Environment, where we after the Devs are ok with the work, we 
> upload the project for customers and Selenium automated test to their part, 
> finally, when we are all happy, then we need to move the project into a 
> PRODUCTION environment which is the one where customer actually access the 
> final product.
>
> All we have found about this is the following:
>
> https://cloud.google.com/appengine/docs/standard/go/creating-separate-dev-environments
>
> But we have many questions, like what is the best way to setup this 
> environment, how to copy code from one environment to the other, for 
> example how do we move the code from DEV to QA. 
>
> Is there a course or a step by step video anyone can share with us? we 
> need some help with this.
>
> Thank you
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/76f925fc-0a89-4ea7-ba46-f7a44e32b6f7%40googlegroups.com.

[google-appengine] Re: Does gae support boto3?

[Ray Wong]

Thanks for your reply! In fact I need to use Mturk api which is included in 
boto and boto3 in my project. However the Mturk api in boto is now 
deprecated so that means I have to use boto3 continue my work.

在 2019年8月23日星期五 UTC+8上午3:31:34，Nicolas (Google Cloud Platform Support)写道：
>
> Hi Ray,
>
> Thank you for opening this thread, I’m not a Boto3 expert however from 
> what I understand of the research that I made, I think you are right Boto3 
> seems to be experiencing some issues with App Engine. 
>
> Also I found some unofficial resource that seems to imply that you could 
> rollback and use Boto2.
>
>
> On Thursday, August 22, 2019 at 9:09:00 AM UTC-4, Ray Wong wrote:
>>
>> Recently I migrated from boto to boto3 in my gae project. But when I run 
>> my project(environment: python27), I get the error:
>> def __init__(self, profile_name, load_config, popen=subprocess.Popen):
>>
>> AttributeError: 'module' object has no attribute 'Popen'
>> I searched on Google and find some people say that boto3 doesn't support 
>> gae sandbox. Now that my project has to use boto3, is there any solution to 
>> run boto3 on gae? Thanks! 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/a6f6cbb9-b7cc-4e3b-84fb-22cc752c2440%40googlegroups.com.

[google-appengine] AppEngine Dev, QA and production environments, how to manage them and and set it up?

[Eduardo Moreira]

Hello Everyone

We have a project that we would like to manage in the most professional 
way, so we would like to have a DEV environment for developers to upload 
the project and test in the AppEnginne environment, then we would like to 
have a QA Environment, where we after the Devs are ok with the work, we 
upload the project for customers and Selenium automated test to their part, 
finally, when we are all happy, then we need to move the project into a 
PRODUCTION environment which is the one where customer actually access the 
final product.

All we have found about this is the following:
https://cloud.google.com/appengine/docs/standard/go/creating-separate-dev-environments

But we have many questions, like what is the best way to setup this 
environment, how to copy code from one environment to the other, for 
example how do we move the code from DEV to QA. 

Is there a course or a step by step video anyone can share with us? we need 
some help with this.

Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/dc3fb00d-306d-42de-afe7-31e2e7998cde%40googlegroups.com.

Re: [google-appengine] High latencies in google app engine in the same timeframe everyday

[Luciano César Natale]

Issues did not arise again!

On Wed, Aug 21, 2019 at 12:03 AM Luciano César Natale 
wrote:

> Nope! PHP!!
>
> On Tue, Aug 20, 2019, 23:51 Cristian Bullokles <
> cristian.bullok...@gmail.com> wrote:
>
>> Do you have a NodeJS Service running in Standard env connected thru
>> socket to a MySQL database?
>>
>> My environment is exaclty like that, any advice to improve performance?
>>
>> Cristian
>>
>>
>> On Tue, Aug 20, 2019 at 11:22 PM Luciano César Natale 
>> wrote:
>>
>>> Today, no strange latencies happened at all! Now I'm almost sure it's a
>>> platform problem...
>>>
>>> On Tue, Aug 20, 2019 at 11:00 PM Cristian Bullokles <
>>> cristian.bullok...@gmail.com> wrote:
>>>
 I've exactly the same problem,the app engine is really slow. I only
 have one "client" device, and the responses are around 300ms.. So I think
 it is not an scalability issue.
 Any thoughts? I think we are all in the same boat, I'm from argentina
 too.


 Cristian

 On Tue, Aug 20, 2019 at 6:43 PM Luciano César Natale <
 lucho...@gmail.com> wrote:

> 2 nd generation!
>
> On Tue, Aug 20, 2019, 16:00 Mauricio Lumbreras 
> wrote:
>
>> Hello
>> are you using 1st generation or 2nd generation cloud sql databases?
>> Regards
>> mauricio
>>
>>
 --
>> You received this message because you are subscribed to the Google
>> Groups "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to google-appengine+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/81dce6dd-1f41-4a35-95ea-d6d39f9cee3f%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/CAEzzm7HOP%3Dus%2Bf4MQuUZusRdjjw%3DKdFD-0YvtGQ2w6MK_RTJ1g%40mail.gmail.com
> 
> .
>
 --
 You received this message because you are subscribed to the Google
 Groups "Google App Engine" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to google-appengine+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/google-appengine/CAOJhfnejuO5TezEEHiG_Ox-%3DBKx%2B5AtKxpa6PPNP36V%3DveyUgQ%40mail.gmail.com
 
 .

>>>
>>>
>>> --
>>> Luciano César Natale
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/google-appengine/CAEzzm7HQE8RNW6J5XVt0urmJ8Z_dTJzy%2BS95xLPSsQEVSyJaVQ%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/CAOJhfnfYfLYhsK5zeqjQMAESZiXByFDPvu3JCbimmwx0pKQKsQ%40mail.gmail.com
>> 
>> .
>>
>

-- 
Luciano César Natale

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAEzzm7H7c-MaZKixteatF6xMRJvZ_pBLa6zhtjwOWpJbc5mgrA%40mail.gmail.com.

Re: [google-appengine] Re: Source Code download from GAE Flex

['Diogo Almeida' via Google App Engine]

Have you tried to upload the files directly to a Cloud Storage bucket? Read 
here  how to do it 
using gsutil.

On Thursday, August 22, 2019 at 1:58:09 PM UTC-4, Sai M wrote:
>
> Yes Andrew , i think  your suggestion makes sense
> I got similar instructions from Google Support team as follows...
>
> 1.- Enter to App Engine > Instances
> 2.-Click on ssh option of the instance you need to explore.
> 3.- In the popup window type the command "docker ps" to list the docker 
> container.
> 4.- Write "docker exec -it CONTAINER_NAME bash" for access to the 
> container (for example gaeapp).
> 5.- Do click on the cogwheel and choose the option "Download file", then 
> write the full path of the file you need to download.
>
> I have exactly followed the above instructions and i am able to see my 
> source code in side the Container. But when i try to download , i always 
> get the error "No Such File" .
>
> For your reference, i have attached screenshots, 
> *ScreenShot-1* Shows the details .pwd as (/app) and file name i am 
> trying to down is (/package.json)
> *Screen Shot-2* shows the full path of the file i am trying to download..
> *Screenshot-3* shows the error of "No Such file" i am repeatedly getting 
> even though i think i am entering the correct file path.
>   
> As far as i know, i think i am in correct directory of Container and using 
> correct full path to download as you can see in the screenshots .  Pls. 
> suggest where i am going wrong.. 
>
> On Tue, 20 Aug 2019 at 18:25, 'Andrew Gorcester' via Google App Engine <
> google-appengine@googlegroups.com> wrote:
>
>> As a workaround, may I suggest enabling debug mode and SSHing into a 
>> running instance to inspect the filesystem directly? You might be able to 
>> exfiltrate the code manually. 
>> https://cloud.google.com/appengine/docs/flexible/python/debugging-an-instance
>>
>> On Tue, Aug 20, 2019 at 10:16 AM 'Nicolas (Google Cloud Platform 
>> Support)' via Google App Engine  
>> wrote:
>>
>>> Hi Sai,
>>>
>>> It used to be possible to download your source code with the appcfg.py 
>>> tools but unfortunately this tool is now deprecated and it’s replacement, 
>>> the Cloud SDK   does not support 
>>> this feature.
>>> There is a feature request for this, and you can follow their progress 
>>> here , although there is no 
>>> ETA for when it will be implemented.
>>>
>>> On Monday, August 19, 2019 at 6:26:58 PM UTC-4, Sai M wrote:

 Hello,

 By accident i have lost the data on my personal laptop where i have 
 kept the Source code of the project i have deployed on Google App Flex.  
 Unfortunately i don't have the backup as it happened by accident when i 
 was 
 trying to upgrade my OS.

 The Project ID is..."smartcc".

 Can you pls. help on instructions of how to down load the latest 
 application source code deployed on Google App Flex env please.

 Thanks a ton in advance.

 Best Regards
 Sai

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/e52822af-78fd-45dc-97d1-4a72915fe6ef%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/CANtYgFKkbxfDY6kXoXB_-E8n8HB_C_o1LM217PbYRXmTEoVhDg%40mail.gmail.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0b125507-de7c-4ac3-bb9b-3a38d513a359%40googlegroups.com.

[google-appengine] Re: Does gae support boto3?

['Nicolas (Google Cloud Platform Support)' via Google App Engine]

Hi Ray,

Thank you for opening this thread, I’m not a Boto3 expert however from what 
I understand of the research that I made, I think you are right Boto3 seems 
to be experiencing some issues with App Engine. 

Also I found some unofficial resource that seems to imply that you could 
rollback and use Boto2.


On Thursday, August 22, 2019 at 9:09:00 AM UTC-4, Ray Wong wrote:
>
> Recently I migrated from boto to boto3 in my gae project. But when I run 
> my project(environment: python27), I get the error:
> def __init__(self, profile_name, load_config, popen=subprocess.Popen):
>
> AttributeError: 'module' object has no attribute 'Popen'
> I searched on Google and find some people say that boto3 doesn't support 
> gae sandbox. Now that my project has to use boto3, is there any solution to 
> run boto3 on gae? Thanks! 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c01ad856-6717-4c80-87a2-e9a80c949e62%40googlegroups.com.

[google-appengine] Re: Active PHP 7.2 sessions lost after gcloud app deploy

['Brent Shaffer' via Google App Engine]

You can also use Firestore for sessions, which I've serendipitously just added 
to the client library 
. I think this is 
the best way because 

   1. There is a free tier
   2. It's very simple to set up compared to a MySQL instance (no need to 
   set up a database/username/password).
   3. It's more "serverless" e.g. you don't configure instances.

I have a "getting started" tutorial for using sessions w/ App Engine and 
Firestore. It'll be at */php/getting-started/sessions*, so we can add it to 
the appengine navigation. Check out the code here 
.

- Brent

On Wednesday, August 21, 2019 at 10:47:38 AM UTC-7, Jason Collins wrote:
>
> In GAE Std, for php72, the default session store is configured. This 
> default is `session.save_handler=files` which uses /tmp to store session 
> files.
>
> In GAE Std, each instance has its own /tmp and the /tmp is empty on any 
> cold start. So this is not really a viable option for session storage 
> (e.g., different requests from the same user may land on different 
> instances, and thus different sessions).
>
> You can use php.ini to configure an alternative (e.g., MySQL on Cloud SQL, 
> or Cloud Memorystore), but you need to do some set up work (e.g., create a 
> Cloud SQL instance, configure access).
>
> We need to have better documentation about this; I will create an internal 
> bug to have some created.
>
>
>
> On Wednesday, 21 August 2019 02:10:53 UTC-7, Andy Whittle wrote:
>>
>> Hi Jason,
>>
>> We're using whichever method GAE Standard env provides to store sessions.
>>
>> On Tuesday, August 20, 2019 at 2:40:24 AM UTC+1, Jason Collins wrote:
>>>
>>> How are you storing your sessions? Is it the /tmp folder default, or 
>>> have you used something like MySQL (Cloud SQL) to store sessions?
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2d92a98b-f684-4401-b484-fdcdc0dd8ad4%40googlegroups.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Barry Hunter]

On Thu, Aug 22, 2019 at 6:16 PM ALT-EMAIL Virilo Tejedor wrote:

> thanks Barry,
>
> I'm not very sure if paralellizing could work, because it has a delay of
> seconds to get a single image from this bucket.
>

If an single image has a latecny of say 2 seconds, and 2 seconds to
download. so 4 seconds per image. Say 15/minute.

But if download say 10 in parallel, then thats 150/minute. Even with the
latency per image.
... overall the 'latency' gets spread around, So your script wouldn't be
waiting for all 10 at the same time. so will waiting for one to start, it
can be downloading a differnt one.

AWS does not store all 10 million images on the *same physical disk*. Its
possibly spread over *millions *of disks.



>
> I should open several threads, and probably I'm going to have problems
> with another limitations
>

In theory yes. But AWS will cape with high concurrency. It's designed that
way. Could easily download 1,000 images concurrently, if you had the
bandwidth.


But however you download the data (even if it just to upload elsewhere -
still dont understand why) - will have to deal with this latency to
download it all in a realistic timeframe.

Downloading them all one by might take 463 days ;)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAJCAUuK3eq6EkAgcn5qznr7jeAtG9DCCHcb6zTFO_UZVFSazcA%40mail.gmail.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Joshua Smith]

Google Cloud Storage is like S3 with cloudfront built-in.

You could absolutely migrate that database to GCS and then access those objects 
via https or via API.

> On Aug 22, 2019, at 1:17 PM, ALT-EMAIL Virilo Tejedor 
>  wrote:
> 
> thanks Joshua... has GAE something similar to cloudfront?
> 
> I could move the S3 bucket to GAE and use the GAE's cloudfront?
> 
> I've no credits for AWS
> 
> On Thursday, August 22, 2019 at 7:12:34 PM UTC+2, Joshua Smith wrote:
> GAE isn’t going to serve those images any faster than S3 does (unless they’re 
> in glacier storage). 
> 
> What you propose won’t be possible because of limits on the size of GAE apps 
> (10K files, max file sizes, etc.) 
> 
> If you are reading the images more than once, putting a cloudfront 
> distribution between your app and the S3 bucket might help. 
> 
> On Aug 22, 2019, at 8:53 AM, ALT-EMAIL Virilo Tejedor  <>> wrote: 
> > 
> > Hi all, 
> > 
> > I'd like to create a static web server to store almost 1 TB of images. 
> > 
> > It is an opensource dataset that I'd like to use to train a Deep Learning 
> > model. 
> > 
> > I have free usage of GPUs and Internet conexion in another plattform, but 
> > they don't provide me 1 TB storage. 
> > 
> > I've also 600$ credits in Google Cloud, I was wondering if there was an 
> > easy way to create something to feed with images the server in the other 
> > plattform. 
> > 
> > The datasource is available as an AWS bucket.  I tried to connect the GPU 
> > machine directly to the ASW bucket via awscli, but it is too much slow.  
> > Like if the bucket were thought for a complete sync but not for coninuous 
> > random access to the files. 
> > 
> > I've though two possible approaches: 
> > 
> > - Execute a python script in GAE to download the dataset and to 
> > create a GAE web server: 
> > https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
> >  
> > 
> >  
> > 
> > - Execute a python script in GAE to download the dataset and to 
> > create a Google Cloud CDN. 
> > 
> > Do you think any of this approaches are valid to feed the model during the 
> > training? 
> > 
> > I'm a newbie in GAE and any help, starting point or idea will be very 
> > wellcomed. 
> > 
> > Thanks in advance 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group. 
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to google-a...@googlegroups.com <>. 
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com
> >  
> > .
> >  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/6dc6f001-f27e-4206-96e0-dca3d1f3fbee%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/AEB097FC-DF59-4D68-BEFE-89578069F0FF%40gmail.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[ALT-EMAIL Virilo Tejedor]

thanks Joshua... has GAE something similar to cloudfront?

I could move the S3 bucket to GAE and use the GAE's cloudfront?

I've no credits for AWS

On Thursday, August 22, 2019 at 7:12:34 PM UTC+2, Joshua Smith wrote:
>
> GAE isn’t going to serve those images any faster than S3 does (unless 
> they’re in glacier storage). 
>
> What you propose won’t be possible because of limits on the size of GAE 
> apps (10K files, max file sizes, etc.) 
>
> If you are reading the images more than once, putting a cloudfront 
> distribution between your app and the S3 bucket might help. 
>
> On Aug 22, 2019, at 8:53 AM, ALT-EMAIL Virilo Tejedor  > wrote: 
> > 
> > Hi all, 
> > 
> > I'd like to create a static web server to store almost 1 TB of images. 
> > 
> > It is an opensource dataset that I'd like to use to train a Deep 
> Learning model. 
> > 
> > I have free usage of GPUs and Internet conexion in another plattform, 
> but they don't provide me 1 TB storage. 
> > 
> > I've also 600$ credits in Google Cloud, I was wondering if there was an 
> easy way to create something to feed with images the server in the other 
> plattform. 
> > 
> > The datasource is available as an AWS bucket.  I tried to connect the 
> GPU machine directly to the ASW bucket via awscli, but it is too much slow. 
>  Like if the bucket were thought for a complete sync but not for coninuous 
> random access to the files. 
> > 
> > I've though two possible approaches: 
> > 
> > - Execute a python script in GAE to download the dataset and to 
> create a GAE web server: 
> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
>  
> > 
> > - Execute a python script in GAE to download the dataset and to 
> create a Google Cloud CDN. 
> > 
> > Do you think any of this approaches are valid to feed the model during 
> the training? 
> > 
> > I'm a newbie in GAE and any help, starting point or idea will be very 
> wellcomed. 
> > 
> > Thanks in advance 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Google App Engine" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to google-a...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com.
>  
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6dc6f001-f27e-4206-96e0-dca3d1f3fbee%40googlegroups.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[ALT-EMAIL Virilo Tejedor]

thanks Barry,

I'm not very sure if paralellizing could work, because it has a delay of 
seconds to get a single image from this bucket.

I should open several threads, and probably I'm going to have problems with 
another limitations

On Thursday, August 22, 2019 at 6:56:50 PM UTC+2, barryhunter wrote:
>
> Well putting them into 'static' files in an app wont work! Its a bit 
> hidden, but there is a 10,000 file limit
>
> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-handled
>
> ... Plus you dont really 'upload' apps incrementally, so would need 
> 'somewhere' to first download entire dataset, package it, then upload. 
> Doubt that would be any easy process with one 1TB (Even if can workaround 
> the 10,000 file limit! )
>
>
> You *could *upload the data to https://cloud.google.com/storage/ - which 
> is roughly comparable with S3 Bucket. But again you will be downloading all 
> the data, uploading to Cloud Storage, then just downloading them *again* for 
> use in the process. (downloading  from Cloud Storage, is going to be 
> roughly comparable from S3, maybe bit quicker, but not massively) 
>
>
> ... seems wasteful. You going to have to download the data to anyway, so 
> just download from AWS, and use *directly*. It might be painful, but 
> should work. If you find AWS slow, then download iamges in parallel (while 
> individual images might be relatively slow, it can sustain high (even 
> massive) concurrency - ie downloading lots of images at once!) 
>
>
> This is an exercise in concurrent processing and throughput. Don't get 
> *distracted 
> *trying to build another storage platform, its unlikely you will do *better 
> *than S3. 
>
>
>
>
>
>
>
> On Thu, Aug 22, 2019 at 5:18 PM ALT-EMAIL Virilo Tejedor <
> virilo...@gmail.com > wrote:
>
>> Hi all,
>>
>> I'd like to create a static web server to store almost 1 TB of images.
>>
>> It is an opensource dataset that I'd like to use to train a Deep Learning 
>> model.
>>
>> I have free usage of GPUs and Internet conexion in another plattform, but 
>> they don't provide me 1 TB storage.
>>
>> I've also 600$ credits in Google Cloud, I was wondering if there was an 
>> easy way to create something to feed with images the server in the other 
>> plattform.
>>
>> The datasource is available as an AWS bucket.  I tried to connect the GPU 
>> machine directly to the ASW bucket via awscli, but it is too much slow.  
>> Like if the bucket were thought for a complete sync but not for coninuous 
>> random access to the files.
>>
>> I've though two possible approaches:
>>
>> - Execute a python script in GAE to download the dataset and to 
>> create a GAE web server: 
>> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
>>
>> - Execute a python script in GAE to download the dataset and to 
>> create a Google Cloud CDN.
>>
>> Do you think any of this approaches are valid to feed the model during 
>> the training?
>>
>> I'm a newbie in GAE and any help, starting point or idea will be very 
>> wellcomed.
>>
>> Thanks in advance
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-a...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/cbdc86dd-9ef5-4c64-a66d-1d10c91006bd%40googlegroups.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Joshua Smith]

GAE isn’t going to serve those images any faster than S3 does (unless they’re 
in glacier storage).

What you propose won’t be possible because of limits on the size of GAE apps 
(10K files, max file sizes, etc.)

If you are reading the images more than once, putting a cloudfront distribution 
between your app and the S3 bucket might help.

On Aug 22, 2019, at 8:53 AM, ALT-EMAIL Virilo Tejedor  
wrote:
> 
> Hi all,
> 
> I'd like to create a static web server to store almost 1 TB of images.
> 
> It is an opensource dataset that I'd like to use to train a Deep Learning 
> model.
> 
> I have free usage of GPUs and Internet conexion in another plattform, but 
> they don't provide me 1 TB storage.
> 
> I've also 600$ credits in Google Cloud, I was wondering if there was an easy 
> way to create something to feed with images the server in the other plattform.
> 
> The datasource is available as an AWS bucket.  I tried to connect the GPU 
> machine directly to the ASW bucket via awscli, but it is too much slow.  Like 
> if the bucket were thought for a complete sync but not for coninuous random 
> access to the files.
> 
> I've though two possible approaches:
> 
>   - Execute a python script in GAE to download the dataset and to create 
> a GAE web server: 
> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
> 
>   - Execute a python script in GAE to download the dataset and to create 
> a Google Cloud CDN.
> 
> Do you think any of this approaches are valid to feed the model during the 
> training?
> 
> I'm a newbie in GAE and any help, starting point or idea will be very 
> wellcomed.
> 
> Thanks in advance
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/FAE6108D-A359-414F-BF7E-0C2E3DE81EF8%40gmail.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Barry Hunter]

Well putting them into 'static' files in an app wont work! Its a bit
hidden, but there is a 10,000 file limit
https://cloud.google.com/appengine/docs/standard/python/how-requests-are-handled

... Plus you dont really 'upload' apps incrementally, so would need
'somewhere' to first download entire dataset, package it, then upload.
Doubt that would be any easy process with one 1TB (Even if can workaround
the 10,000 file limit! )


You *could *upload the data to https://cloud.google.com/storage/ - which is
roughly comparable with S3 Bucket. But again you will be downloading all
the data, uploading to Cloud Storage, then just downloading them *again* for
use in the process. (downloading  from Cloud Storage, is going to be
roughly comparable from S3, maybe bit quicker, but not massively)


... seems wasteful. You going to have to download the data to anyway, so
just download from AWS, and use *directly*. It might be painful, but should
work. If you find AWS slow, then download iamges in parallel (while
individual images might be relatively slow, it can sustain high (even
massive) concurrency - ie downloading lots of images at once!)


This is an exercise in concurrent processing and throughput. Don't get
*distracted
*trying to build another storage platform, its unlikely you will do *better
*than S3.







On Thu, Aug 22, 2019 at 5:18 PM ALT-EMAIL Virilo Tejedor <
viriloteje...@gmail.com> wrote:

> Hi all,
>
> I'd like to create a static web server to store almost 1 TB of images.
>
> It is an opensource dataset that I'd like to use to train a Deep Learning
> model.
>
> I have free usage of GPUs and Internet conexion in another plattform, but
> they don't provide me 1 TB storage.
>
> I've also 600$ credits in Google Cloud, I was wondering if there was an
> easy way to create something to feed with images the server in the other
> plattform.
>
> The datasource is available as an AWS bucket.  I tried to connect the GPU
> machine directly to the ASW bucket via awscli, but it is too much slow.
> Like if the bucket were thought for a complete sync but not for coninuous
> random access to the files.
>
> I've though two possible approaches:
>
> - Execute a python script in GAE to download the dataset and to
> create a GAE web server:
> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
>
> - Execute a python script in GAE to download the dataset and to
> create a Google Cloud CDN.
>
> Do you think any of this approaches are valid to feed the model during the
> training?
>
> I'm a newbie in GAE and any help, starting point or idea will be very
> wellcomed.
>
> Thanks in advance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAJCAUuJsz4yrgBM6RK1rUcuJz4jJ1rfQYgS7gdD2%2BnO1a22zoA%40mail.gmail.com.

[google-appengine] Using GAE to store images for other machine out of GAE

[ALT-EMAIL Virilo Tejedor]

Hi all,

I'd like to create a static web server to store almost 1 TB of images.

It is an opensource dataset that I'd like to use to train a Deep Learning model.

I have free usage of GPUs and Internet conexion in another plattform, but they 
don't provide me 1 TB storage.

I've also 600$ credits in Google Cloud, I was wondering if there was an easy 
way to create something to feed with images the server in the other plattform.

The datasource is available as an AWS bucket.  I tried to connect the GPU 
machine directly to the ASW bucket via awscli, but it is too much slow.  Like 
if the bucket were thought for a complete sync but not for coninuous random 
access to the files.

I've though two possible approaches:

- Execute a python script in GAE to download the dataset and to create 
a GAE web server: 
https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website

- Execute a python script in GAE to download the dataset and to create 
a Google Cloud CDN.

Do you think any of this approaches are valid to feed the model during the 
training?

I'm a newbie in GAE and any help, starting point or idea will be very wellcomed.

Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com.

[google-appengine] Does gae support boto3?

[Ray Wong]

Recently I migrated from boto to boto3 in my gae project. But when I run my 
project(environment: python27), I get the error:
def __init__(self, profile_name, load_config, popen=subprocess.Popen):

AttributeError: 'module' object has no attribute 'Popen'
I searched on Google and find some people say that boto3 doesn't support 
gae sandbox. Now that my project has to use boto3, is there any solution to 
run boto3 on gae? Thanks! 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9b6651df-907a-473e-8322-0f44c645b75c%40googlegroups.com.