+1 to separating integration tests from "build". It should be able to succeed without internet access (if deps are cached).
On Tue, Mar 26, 2019 at 3:18 PM Michael Luckey <adude3...@gmail.com> wrote: > Of course, we could implement something here. But I am worried about the > consequences. As gogradle writes into (user) global state this would have > unexpected side effects. > > Consider a developer running Project A - which happens to also use > gogradle - and during build of that project issues an innocent beam clean. > Would be no fun to track all that failures arising out of those now deleted > state. > > Unfortunately, I do not have a clear understanding yet, what's going on > here, why that cache is put there and how it gets inconsistent. Need to > look into that. (But might be obsoleted anyway by Roberts plans to drop > gogradle for something different.) > > On Tue, Mar 26, 2019 at 7:28 PM Thomas Weise <t...@apache.org> wrote: > >> Can this be addressed by having "clean" remove all state that gogradle >> leaves behind? This staleness issue has bitten me a few times also and it >> would be good to have a reliable way to deal with it, even if it involves >> an extra clean. >> >> >> On Tue, Mar 26, 2019 at 11:14 AM Michael Luckey <adude3...@gmail.com> >> wrote: >> >>> @Udi >>> Did you try to just delete the >>> '/usr/local/google/home/ehudm/.gradle/go/repo/cloud.google.com' folder? >>> >>> @Robert >>> As said before, I am a bit scared about the implications. Shelling out >>> is done by python, and from build perspective, this does not work very >>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>> >>> But of course, we need to play with this a bit more. >>> >>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke <rob...@frantil.com> wrote: >>> >>>> Reading the error from the gradle scan, it largely looks like some part >>>> of the GCP dependencies for the build depends on a package, where the >>>> commit version is no longer around. The main issue with gogradle is that >>>> it's entirely distinct from the usual Go workflow, which means deps users >>>> use are likely to be different to what's in the lock file. >>>> >>>> This work will be tracked in >>>> https://issues.apache.org/jira/browse/BEAM-5379 >>>> GoGradle hasn't moved to support the new-go way of handling deps, so my >>>> inclination is to simplify to simple scripts for Gradle that shell out the >>>> to Go tool for handling Go dep management, over trying to fix GoGradle. >>>> >>>> On Tue, 26 Mar 2019 at 09:43, Udi Meiri <eh...@google.com> wrote: >>>> >>>>> Robert, from what I recall it's not flaky for me - it consistently >>>>> fails. Let me know if there's a way to get more logging about this error. >>>>> >>>>> On Mon, Mar 25, 2019, 19:50 Robert Burke <rob...@frantil.com> wrote: >>>>> >>>>>> It's concerning to me that 1) the Go dependency resolution via >>>>>> gogradle is flaky, and 2) that it can block other languages. >>>>>> >>>>>> I suppose 2) makes sense since it's part of the container >>>>>> bootstrapping code, but that makes 1) a serious problem, of which I >>>>>> wasn't >>>>>> aware. >>>>>> I should have time to investigate this in the next two weeks. >>>>>> >>>>>> On Mon, 25 Mar 2019 at 18:08, Michael Luckey <adude3...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Just for the record, >>>>>>> >>>>>>> using a vm here, because did not yet get all task running on my mac, >>>>>>> and did not want to mess with my setup. >>>>>>> >>>>>>> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 >>>>>>> cores and further >>>>>>> >>>>>>> sudo apt update >>>>>>> >>>>>>> sudo apt install gcc >>>>>>> >>>>>>> sudo apt install make >>>>>>> >>>>>>> sudo apt install perl >>>>>>> >>>>>>> sudo apt install curl >>>>>>> >>>>>>> sudo apt install openjdk-8-jdk >>>>>>> >>>>>>> sudo apt install python >>>>>>> >>>>>>> sudo apt install -y software-properties-common >>>>>>> >>>>>>> sudo add-apt-repository ppa:deadsnakes/ppa >>>>>>> >>>>>>> sudo apt update >>>>>>> >>>>>>> sudo apt install python3.5 >>>>>>> >>>>>>> sudo apt-get install apt-transport-https ca-certificates curl >>>>>>> gnupg-agent software-properties-common >>>>>>> >>>>>>> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo >>>>>>> apt-key add - >>>>>>> >>>>>>> sudo apt-key fingerprint 0EBFCD88 >>>>>>> >>>>>>> sudo add-apt-repository "deb [arch=amd64] >>>>>>> https://download.docker.com/linux/ubuntu \ >>>>>>> >>>>>>> $(lsb_release -cs) \ >>>>>>> >>>>>>> stable" >>>>>>> >>>>>>> sudo apt-get update >>>>>>> >>>>>>> sudo apt-get install docker-ce docker-ce-cli containerd.io >>>>>>> >>>>>>> sudo groupadd docker >>>>>>> >>>>>>> sudo usermod -aG docker $USER >>>>>>> >>>>>>> git config --global user.email "d...@spam.me" >>>>>>> >>>>>>> git config --global user.name "Some Guy" >>>>>>> >>>>>>> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >>>>>>> >>>>>>> sudo python get-pip.py >>>>>>> >>>>>>> rm get-pip.py >>>>>>> >>>>>>> sudo pip install --upgrade virtualenv >>>>>>> >>>>>>> sudo pip install cython >>>>>>> >>>>>>> sudo apt-get install python-dev >>>>>>> >>>>>>> sudo apt-get install python3-distutils >>>>>>> >>>>>>> sudo apt-get install python3-dev # for python3.x installs >>>>>>> >>>>>>> >>>>>>> git clone https://github.com/apache/beam.git cd beam/ ./gradlew >>>>>>> build >>>>>>> >>>>>>> Nothing else changed/added. (hopefully, need to reassure myself here) >>>>>>> >>>>>>> Unfortunately, this is failing. Need to exclude those python tests >>>>>>> (and of course website, which usually fails on lira links) >>>>>>> >>>>>>> So I might be missing some env settings for gap, dunno. Probably >>>>>>> missed some docs. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey <adude3...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Udi for trying that! >>>>>>>> >>>>>>>> In fact, the go dependency resolution is flaky. Did not look into >>>>>>>> that, but just rerunning usually works. Of course, less than optimal, >>>>>>>> but, >>>>>>>> well... >>>>>>>> >>>>>>>> Running build target is of course just an aggregation of task to >>>>>>>> run. And unfortunately just running that >>>>>>>> >>>>>>>> ./gradlew :beam-sdks-python:testPy2Gcp >>>>>>>> >>>>>>>> stalls on my (virtual) machine. >>>>>>>> >>>>>>>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri <eh...@google.com> wrote: >>>>>>>> >>>>>>>>> Okay, `./gradlew build` failed pretty quickly for me: >>>>>>>>> >>>>>>>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED >>>>>>>>> cloud.google.com/go: >>>>>>>>> commit='4f6c921ec566a33844f4e7879b31cd8575a6982d', urls=[ >>>>>>>>> https://code.googlesource.com/gocloud] does not exist in >>>>>>>>> /usr/local/google/home/ehudm/.gradle/go/repo/ >>>>>>>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating >>>>>>>>> will be performed. >>>>>>>>> >>>>>>>>> https://gradle.com/s/x5zqbc5zwd3bg >>>>>>>>> >>>>>>>>> (Now I remember why I stopped using `build` :/) >>>>>>>>> >>>>>>>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri <eh...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It shouldn't stall. That's a bug. >>>>>>>>>> OTOH, I never use the `build` target. >>>>>>>>>> I'll try running that myself. >>>>>>>>>> >>>>>>>>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey <adude3...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> trying to run './gradlew build' on vanilla setup, my build >>>>>>>>>>> consistently stalls during execution of python gcp tests, e.g. on >>>>>>>>>>> both of >>>>>>>>>>> - > :beam-sdks-python:testPy2Gcp >>>>>>>>>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp >>>>>>>>>>> >>>>>>>>>>> Console output: >>>>>>>>>>> #### snip #### >>>>>>>>>>> test_big_query_standard_sql >>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>> ... SKIP: IT is skipped because --test-pipeline-options is not >>>>>>>>>>> specified >>>>>>>>>>> test_big_query_standard_sql_kms_key >>>>>>>>>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) >>>>>>>>>>> ... SKIP: This test requires BQ Dataflow native source support for >>>>>>>>>>> KMS, >>>>>>>>>>> which is not available yet. >>>>>>>>>>> test_multiple_destinations_transform >>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>> ... SKIP: >>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>> test_one_job_fails_all_jobs_fail >>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) >>>>>>>>>>> ... SKIP: >>>>>>>>>>> IT is skipped because --test-pipeline-options is not specified >>>>>>>>>>> test_records_traverse_transform_with_mocks >>>>>>>>>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> output ends here, would expect a failed or ok here. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Afterwards no progress - even waiting for hours. Any idea, what >>>>>>>>>>> might be causing this? Do I need to add some GCP properties for >>>>>>>>>>> this task ? >>>>>>>>>>> >>>>>>>>>>> Any ideas, what I am doing wrong? >>>>>>>>>>> >>>>>>>>>>> best, >>>>>>>>>>> >>>>>>>>>>> michel >>>>>>>>>>> >>>>>>>>>>>
