Luckey, I couldn't recreate your issue, but I still haven't done a full build. I created a new GCE VM with using the ubuntu-1804-bionic-v20190212a image (n1-standard-4 machine type).
Ran the following: sudo apt-get update sudo apt-get install python-pip sudo apt-get install python-virtualenv git clone https://github.com/apache/beam.git cd beam ./gradlew :beam-sdks-python:testPy2Gcp [failed: no JAVA_HOME] sudo apt-get install openjdk-8-jdk ./gradlew :beam-sdks-python:testPy2Gcp Got: BUILD SUCCESSFUL in 7m 52s Then I tried: ./gradlew build And ran out of disk space. :) (beam/ is taking 4.5G and the VM boot disk is 10G total) On Tue, Mar 26, 2019 at 1:35 PM Robert Burke <rob...@frantil.com> wrote: > Michael, your concern is reasonable, especially with the experience with > python, though that does help me bootstrap this work. :) > > The go tools provide caching and avoid redoing work if the source files > haven't changed. This applies most particularly for `go build` and `go > test`. As long as the go code isn't changing at every invocation, this > should be fine. I'm not aware of the same being the case for the usual > python tools. > > The real trick is ensuring a valid and consistent environment for the go > code. > > The environment question becomes easier for everyone by moving to go > modules, which were designed to provide these kinds of consistent builds. > It also avoids needing a GOPATH set. Any directory is permitted, as long as > the go.mod is present. > > (The Go SDK doesn't yet us go modules, so go.mod and go.sum aren't yet in > the repo.) > > The main blocker is see is updating the Jenkins machines to have the > latest version of Go (1.12) instead of 1.10, which doesn't support modules. > This only blocks a final submission, rather than the work fortunately. > > On Tue, Mar 26, 2019, 1:08 PM Udi Meiri <eh...@google.com> wrote: > >> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one package >> with issues). >> My ~/.bashrc has >> export GOPATH=$HOME/go >> so maybe that's making the difference in my setup. >> >> On Tue, Mar 26, 2019 at 11:28 AM Thomas Weise <t...@apache.org> wrote: >> >>> Can this be addressed by having "clean" remove all state that gogradle >>> leaves behind? This staleness issue has bitten me a few times also and it >>> would be good to have a reliable way to deal with it, even if it involves >>> an extra clean. >>> >>> >>> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey <adude3...@gmail.com> >>> wrote: >>> >>>> @Udi >>>> Did you try to just delete the >>>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder? >>>> >>>> @Robert >>>> As said before, I am a bit scared about the implications. Shelling out >>>> is done by python, and from build perspective, this does not work very >>>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>>> >>>> But of course, we need to play with this a bit more. >>>> >>>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> >>>> wrote: >>>> >>>>> Reading the error from the gradle scan, it largely looks like some >>>>> part of the GCP dependencies for the build depends on a package, where the >>>>> commit version is no longer around. The main issue with gogradle is that >>>>> it's entirely distinct from the usual Go workflow, which means deps users >>>>> use are likely to be different to what's in the lock file. >>>>> >>>>> This work will be tracked in >>>>> https://issues.apache.org/jira/browse/BEAM-5379 >>>>> GoGradle hasn't moved to support the new-go way of handling deps, so >>>>> my inclination is to simplify to simple scripts for Gradle that shell out >>>>> the to Go tool for handling Go dep management, over trying to fix >>>>> GoGradle. >>>>> >>>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> Robert, from what I recall it's not flaky for me - it consistently >>>>>> fails. Let me know if there's a way to get more logging about this error. >>>>>> >>>>>> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: >>>>>> >>>>>>> It's concerning to me that 1) the Go dependency resolution via >>>>>>> gogradle is flaky, and 2) that it can block other languages. >>>>>>> >>>>>>> I suppose 2) makes sense since it's part of the container >>>>>>> bootstrapping code, but that makes 1) a serious problem, of which I >>>>>>> wasn't >>>>>>> aware. >>>>>>> I should have time to investigate this in the next two weeks. >>>>>>> >>>>>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Just for the record, >>>>>>>> >>>>>>>> using a vm here, because did not yet get all task running on my >>>>>>>> mac, and did not want to mess with my setup. >>>>>>>> >>>>>>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 >>>>>>>> cores and further >>>>>>>> >>>>>>>> sudo apt update >>>>>>>> >>>>>>>> sudo apt install gcc >>>>>>>> >>>>>>>> sudo apt install make >>>>>>>> >>>>>>>> sudo apt install perl >>>>>>>> >>>>>>>> sudo apt install curl >>>>>>>> >>>>>>>> sudo apt install openjdk-8-jdk >>>>>>>> >>>>>>>> sudo apt install python >>>>>>>> >>>>>>>> sudo apt install -y software-properties-common >>>>>>>> >>>>>>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>>>>>> >>>>>>>> sudo apt update >>>>>>>> >>>>>>>> sudo apt install python3.5 >>>>>>>> >>>>>>>> sudo apt-get install apt-transport-https ca-certificates curl >>>>>>>> gnupg-agent software-properties-common >>>>>>>> >>>>>>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo >>>>>>>> apt-key add - >>>>>>>> >>>>>>>> sudo apt-key fingerprint 0EBFCD88 >>>>>>>> >>>>>>>> sudo add-apt-repository "deb [arch=amd64] >>>>>>>> https://download.docker.com/linux/ubuntu \ >>>>>>>> >>>>>>>> $(lsb_release -cs) \ >>>>>>>> >>>>>>>> stable" >>>>>>>> >>>>>>>> sudo apt-get update >>>>>>>> >>>>>>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>>>>>> >>>>>>>> sudo groupadd docker >>>>>>>> >>>>>>>> sudo usermod -aG docker $USER >>>>>>>> >>>>>>>> git config --global user.email "d...@spam.me" >>>>>>>> >>>>>>>> git config --global user.name "Some Guy" >>>>>>>> >>>>>>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>>>>>> >>>>>>>> sudo python get-pip.py >>>>>>>> >>>>>>>> rm get-pip.py >>>>>>>> >>>>>>>> sudo pip install --upgrade virtualenv >>>>>>>> >>>>>>>> sudo pip install cython >>>>>>>> >>>>>>>> sudo apt-get install python-dev >>>>>>>> >>>>>>>> sudo apt-get install python3-distutils >>>>>>>> >>>>>>>> sudo apt-get install python3-dev # for python3.x installs >>>>>>>> >>>>>>>> >>>>>>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew >>>>>>>> build >>>>>>>> >>>>>>>> Nothing else changed/added. (hopefully, need to reassure myself >>>>>>>> here) >>>>>>>> >>>>>>>> Unfortunately, this is failing. Need to exclude those python tests >>>>>>>> (and of course website, which usually fails on lira links) >>>>>>>> >>>>>>>> So I might be missing some env settings for gap, dunno. Probably >>>>>>>> missed some docs. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks Udi for trying that! >>>>>>>>> >>>>>>>>> In fact, the go dependency resolution is flaky. Did not look into >>>>>>>>> that, but just rerunning usually works. Of course, less than optimal, >>>>>>>>> but, >>>>>>>>> well... >>>>>>>>> >>>>>>>>> Running build target is of course just an aggregation of task to >>>>>>>>> run. And unfortunately just running that >>>>>>>>> >>>>>>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>>>>>> >>>>>>>>> stalls on my (virtual) machine. >>>>>>>>> >>>>>>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>>>>>> >>>>>>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>>>>>> cloud.google.com/go: >>>>>>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating >>>>>>>>>> will be performed. >>>>>>>>>> >>>>>>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>>>>>> >>>>>>>>>> (Now I remember why I stopped using `build` :/) >>>>>>>>>> >>>>>>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> It shouldn't stall. That's a bug. >>>>>>>>>>> OTOH, I never use the `build` target. >>>>>>>>>>> I'll try running that myself. >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>>>>>> consistently stalls during execution of python gcp tests, e.g. on >>>>>>>>>>>> both of >>>>>>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>>>>>> >>>>>>>>>>>> Console output: >>>>>>>>>>>> #### snip #### >>>>>>>>>>>> test_big_query_standard_sql >>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>>>>>> specified >>>>>>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>>> ... SKIP: This test requires BQ Dataflow native source support for >>>>>>>>>>>> KMS, >>>>>>>>>>>> which is not available yet. >>>>>>>>>>>> test_multiple_destinations_transform >>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>> ... SKIP: >>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>>> ... SKIP: >>>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) >>>>>>>>>>>> ... >>>>>>>>>>>> >>>>>>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Afterwards no progress - even waiting for hours. Any idea, what >>>>>>>>>>>> might be causing this? Do I need to add some GCP properties for >>>>>>>>>>>> this task ? >>>>>>>>>>>> >>>>>>>>>>>> Any ideas, what I am doing wrong? >>>>>>>>>>>> >>>>>>>>>>>> best, >>>>>>>>>>>> >>>>>>>>>>>> michel >>>>>>>>>>>> >>>>>>>>>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
