This is what I got from a straightforward build of the source distro here ... really, ideally, it builds as-is from source. You're saying someone would have to first build a k8s distro from source too? It's not a 'must' that this be automatic but nothing else fails out of the box. I feel like I might be misunderstanding the setup here. On Mon, Oct 22, 2018 at 7:25 PM Stavros Kontopoulos <stavros.kontopou...@lightbend.com> wrote: > > >> >> tar (child): Error is not recoverable: exiting now >> tar: Child returned status 2 >> tar: Error is not recoverable: exiting now >> scripts/setup-integration-test-env.sh: line 85: >> /home/srowen/spark-2.4.0/resource-managers/kubernetes/integration-tests/target/spark-dist-unpacked/bin/docker-image-tool.sh: > > > It seems you are missing the distro file... here is how I run it locally: > > DOCKER_USERNAME=... > SPARK_K8S_IMAGE_TAG=... > > ./dev/make-distribution.sh --name test --tgz -Phadoop-2.7 -Pkubernetes -Phive > tar -zxvf spark-2.4.0-SNAPSHOT-bin-test.tgz > cd spark-2.4.0-SNAPSHOT-bin-test > ./bin/docker-image-tool.sh -r $DOCKER_USERNAME -t $SPARK_K8S_IMAGE_TAG build > cd .. > TGZ_PATH=$(pwd)/spark-2.4.0-SNAPSHOT-bin-test.tgz > cd resource-managers/kubernetes/integration-tests > ./dev/dev-run-integration-tests.sh --image-tag $SPARK_K8S_IMAGE_TAG > --spark-tgz $TGZ_PATH --image-repo $DOCKER_USERNAME > > Stavros > > On Tue, Oct 23, 2018 at 1:54 AM, Sean Owen <sro...@gmail.com> wrote: >> >> Provisionally looking good to me, but I had a few questions. >> >> We have these open for 2.4, but I presume they aren't actually going >> to be in 2.4 and should be untargeted: >> >> SPARK-25507 Update documents for the new features in 2.4 release >> SPARK-25179 Document the features that require Pyarrow 0.10 >> SPARK-25783 Spark shell fails because of jline incompatibility >> SPARK-25347 Document image data source in doc site >> SPARK-25584 Document libsvm data source in doc site >> SPARK-25346 Document Spark builtin data sources >> SPARK-24464 Unit tests for MLlib's Instrumentation >> SPARK-23197 Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle" >> SPARK-22809 pyspark is sensitive to imports with dots >> SPARK-21030 extend hint syntax to support any expression for Python and R >> >> Comments in several of the doc issues suggest they are needed for 2.4 >> though. How essential? >> >> (Brief digression: SPARK-21030 is an example of a pattern I see >> sometimes. Parent Epic A is targeted for version X. Children B and C >> are not. Epic A's description is basically "do X and Y". Is the parent >> helping? And now that Y is done, is there a point in tracking X with >> two JIRAs? can I just close the Epic?) >> >> I am not sure I've tried running K8S in my test runs before, but I get >> this on my Linux machine: >> >> [INFO] --- exec-maven-plugin:1.4.0:exec (setup-integration-test-env) @ >> spark-kubernetes-integration-tests_2.12 --- >> fatal: not a git repository (or any of the parent directories): .git >> tar (child): --strip-components=1: Cannot open: No such file or directory >> tar (child): Error is not recoverable: exiting now >> tar: Child returned status 2 >> tar: Error is not recoverable: exiting now >> scripts/setup-integration-test-env.sh: line 85: >> /home/srowen/spark-2.4.0/resource-managers/kubernetes/integration-tests/target/spark-dist-unpacked/bin/docker-image-tool.sh: >> No such file or directory >> /home/srowen/spark-2.4.0/resource-managers/kubernetes/integration-tests >> [INFO] >> [INFO] --- scalatest-maven-plugin:1.0:test (integration-test) @ >> spark-kubernetes-integration-tests_2.12 --- >> Discovery starting. >> Discovery completed in 289 milliseconds. >> Run starting. Expected test count is: 14 >> KubernetesSuite: >> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED *** >> java.lang.NullPointerException: >> at >> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:92) >> at >> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) >> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) >> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) >> at >> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.org$scalatest$BeforeAndAfter$$super$run(KubernetesSuite.scala:39) >> at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:258) >> at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:256) >> at >> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.run(KubernetesSuite.scala:39) >> at org.scalatest.Suite.callExecuteOnSuite$1(Suite.scala:1210) >> at org.scalatest.Suite.$anonfun$runNestedSuites$1(Suite.scala:1257) >> ... >> >> Clearly it's expecting something about the env that isn't true, but I >> don't know if it's a problem with those expectations versus what is in >> the source release, or, just something to do with my env. This is with >> Scala 2.12. >> >> >> >> On Mon, Oct 22, 2018 at 12:42 PM Wenchen Fan <cloud0...@gmail.com> wrote: >> > >> > Please vote on releasing the following candidate as Apache Spark version >> > 2.4.0. >> > >> > The vote is open until October 26 PST and passes if a majority +1 PMC >> > votes are cast, with >> > a minimum of 3 +1 votes. >> > >> > [ ] +1 Release this package as Apache Spark 2.4.0 >> > [ ] -1 Do not release this package because ... >> > >> > To learn more about Apache Spark, please see http://spark.apache.org/ >> > >> > The tag to be voted on is v2.4.0-rc4 (commit >> > e69e2bfa486d8d3b9d203b96ca9c0f37c2b6cabe): >> > https://github.com/apache/spark/tree/v2.4.0-rc4 >> > >> > The release files, including signatures, digests, etc. can be found at: >> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc4-bin/ >> > >> > Signatures used for Spark RCs can be found in this file: >> > https://dist.apache.org/repos/dist/dev/spark/KEYS >> > >> > The staging repository for this release can be found at: >> > https://repository.apache.org/content/repositories/orgapachespark-1290 >> > >> > The documentation corresponding to this release can be found at: >> > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc4-docs/ >> > >> > The list of bug fixes going into 2.4.0 can be found at the following URL: >> > https://issues.apache.org/jira/projects/SPARK/versions/12342385 >> > >> > FAQ >> > >> > ========================= >> > How can I help test this release? >> > ========================= >> > >> > If you are a Spark user, you can help us test this release by taking >> > an existing Spark workload and running on this release candidate, then >> > reporting any regressions. >> > >> > If you're working in PySpark you can set up a virtual env and install >> > the current RC and see if anything important breaks, in the Java/Scala >> > you can add the staging repository to your projects resolvers and test >> > with the RC (make sure to clean up the artifact cache before/after so >> > you don't end up building with a out of date RC going forward). >> > >> > =========================================== >> > What should happen to JIRA tickets still targeting 2.4.0? >> > =========================================== >> > >> > The current list of open tickets targeted at 2.4.0 can be found at: >> > https://issues.apache.org/jira/projects/SPARK and search for "Target >> > Version/s" = 2.4.0 >> > >> > Committers should look at those and triage. Extremely important bug >> > fixes, documentation, and API tweaks that impact compatibility should >> > be worked on immediately. Everything else please retarget to an >> > appropriate release. >> > >> > ================== >> > But my bug isn't fixed? >> > ================== >> > >> > In order to make timely releases, we will typically not hold the >> > release unless the bug in question is a regression from the previous >> > release. That being said, if there is something which is a regression >> > that has not been correctly targeted please ping me or a committer to >> > help target the issue. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > > >
--------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
