Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Actually +1 from me... This is a recommendAll feature we are testing which is really compute intensive... For ranking metric calculation I was trying to run through the Netflix matrix and generate a ranked list of recommendation for all 17K products and perhaps it needs more compute than what is needed. I was running 6 nodes, 120 cores, 240 GB...It needed to shuffle around 100 GB over 6 nodes... A version with topK runs fine where K = (some multipler on number of movies each user saw and we cross validate on that) Running the following JIRA on Netflix dataset (the dataset is distributed with Jellyfish code http://i.stanford.edu/hazy/victor/Hogwild/), will reproduce the failure... https://issues.apache.org/jira/browse/SPARK-4231 The failed job I will debug more and figure out the real cause. If needed I will open up new JIRAs. On Sun, Nov 23, 2014 at 9:50 AM, Debasish Das debasish.da...@gmail.com wrote: -1 from me...same FetchFailed issue as what Hector saw... I am running Netflix dataset and dumping out recommendation for all users. It shuffles around 100 GB data on disk to run a reduceByKey per user on utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset... I gave Spark 10 nodes, 8 cores, 160 GB of memory. Fails with the following FetchFailed errors. 14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID 2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1, tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28, reduceId=28) It's a consistent behavior on master as well. I tested it both on YARN and Standalone. I compiled spark-1.1 branch (assuming it has all the fixes from RC2 tag. I am now compiling spark-1.0 branch and see if this issue shows up there as well. If it is related to hash/sort based shuffle most likely it won't show up on 1.0. Thanks. Deb On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee hector@gmail.com wrote: Whoops I must have used the 1.2 preview and mixed them up. spark-shell -version shows version 1.2.0 Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to 1.2 On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem? Matei On Nov 20, 2014, at 11:50 AM, Hector Yee hector@gmail.com wrote: This is whatever was in http://people.apache.org/~andrewor14/spark-1 .1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1 Release this package as Apache Spark 1.1.1 On 20 Nov 2014 04:22, Andrew Or and...@databricks.com wrote: I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or and...@databricks.com: Please vote on releasing the following candidate as Apache Spark version 1 .1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
-1 from me...same FetchFailed issue as what Hector saw... I am running Netflix dataset and dumping out recommendation for all users. It shuffles around 100 GB data on disk to run a reduceByKey per user on utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset... I gave Spark 10 nodes, 8 cores, 160 GB of memory. Fails with the following FetchFailed errors. 14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID 2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1, tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28, reduceId=28) It's a consistent behavior on master as well. I tested it both on YARN and Standalone. I compiled spark-1.1 branch (assuming it has all the fixes from RC2 tag. I am now compiling spark-1.0 branch and see if this issue shows up there as well. If it is related to hash/sort based shuffle most likely it won't show up on 1.0. Thanks. Deb On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee hector@gmail.com wrote: Whoops I must have used the 1.2 preview and mixed them up. spark-shell -version shows version 1.2.0 Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to 1.2 On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem? Matei On Nov 20, 2014, at 11:50 AM, Hector Yee hector@gmail.com wrote: This is whatever was in http://people.apache.org/~andrewor14/spark-1 .1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1 (binding). Don't see any evidence of regressions at this point. The issue reported by Hector was not related to this rlease. On Sun, Nov 23, 2014 at 9:50 AM, Debasish Das debasish.da...@gmail.com wrote: -1 from me...same FetchFailed issue as what Hector saw... I am running Netflix dataset and dumping out recommendation for all users. It shuffles around 100 GB data on disk to run a reduceByKey per user on utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset... I gave Spark 10 nodes, 8 cores, 160 GB of memory. Fails with the following FetchFailed errors. 14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID 2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1, tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28, reduceId=28) It's a consistent behavior on master as well. I tested it both on YARN and Standalone. I compiled spark-1.1 branch (assuming it has all the fixes from RC2 tag. I am now compiling spark-1.0 branch and see if this issue shows up there as well. If it is related to hash/sort based shuffle most likely it won't show up on 1.0. Thanks. Deb On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee hector@gmail.com wrote: Whoops I must have used the 1.2 preview and mixed them up. spark-shell -version shows version 1.2.0 Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to 1.2 On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem? Matei On Nov 20, 2014, at 11:50 AM, Hector Yee hector@gmail.com wrote: This is whatever was in http://people.apache.org/~andrewor14/spark-1 .1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Hi, I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but therc releases not being tagged with -rcX means the pre-built artifacts are basically useless to me. (Pedantically, to test a release, I have to upload it into our internal repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact ends up not being final, then I'm screwed, because I would have to clear the local cache of any of our machines, dev/Jenkins/etc., that ever downloaded the formerly known as 1.1.1 but not really rc artifacts.) What's frustrating is that I know other Apache projects do rc releases, and even get them into Maven central, e.g.: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22 So, I apologize for the distraction from getting real work done, but perhaps you guys could find a creative way to work around the well-intentioned mandate on artifact voting? (E.g. perhaps have multiple votes, one for each successive rc (with -rcX suffix), then, once blessed, another one on the actually-final/no-rcX artifact (built from the last rc's tag); or publish no-rcX artifacts for official voting, as today, but then, at the same time, add -rcX artifacts to Maven central for non-binding/3rd party testing, etc.) Thanks, Stephen
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Interesting, perhaps we could publish each one with two IDs, of which the rc one is unofficial. The problem is indeed that you have to vote on a hash for a potentially final artifact. Matei On Nov 23, 2014, at 7:54 PM, Stephen Haberman stephen.haber...@gmail.com wrote: Hi, I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but therc releases not being tagged with -rcX means the pre-built artifacts are basically useless to me. (Pedantically, to test a release, I have to upload it into our internal repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact ends up not being final, then I'm screwed, because I would have to clear the local cache of any of our machines, dev/Jenkins/etc., that ever downloaded the formerly known as 1.1.1 but not really rc artifacts.) What's frustrating is that I know other Apache projects do rc releases, and even get them into Maven central, e.g.: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22 So, I apologize for the distraction from getting real work done, but perhaps you guys could find a creative way to work around the well-intentioned mandate on artifact voting? (E.g. perhaps have multiple votes, one for each successive rc (with -rcX suffix), then, once blessed, another one on the actually-final/no-rcX artifact (built from the last rc's tag); or publish no-rcX artifacts for official voting, as today, but then, at the same time, add -rcX artifacts to Maven central for non-binding/3rd party testing, etc.) Thanks, Stephen - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Hey Stephen, Thanks for bringing this up. Technically when we call a release vote it needs to be on the exact commit that will be the final release. However, one thing I've thought of doing for a while would be to publish the maven artifacts using a version tag with $VERSION-rcX even if the underlying commit has $VERSION in the pom files. Some recent changes I've made to the way we do publishing in branch 1.2 should make this pretty easy - it wasn't very easy before because we used maven's publishing plugin which makes modifying the published version tricky. Our current approach is, indeed, problematic because maven artifacts are supposed to be immutable once they have a specific version identifier. I created SPARK-4568 to track this: https://issues.apache.org/jira/browse/SPARK-4568 - Patrick On Sun, Nov 23, 2014 at 8:11 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Interesting, perhaps we could publish each one with two IDs, of which the rc one is unofficial. The problem is indeed that you have to vote on a hash for a potentially final artifact. Matei On Nov 23, 2014, at 7:54 PM, Stephen Haberman stephen.haber...@gmail.com wrote: Hi, I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but therc releases not being tagged with -rcX means the pre-built artifacts are basically useless to me. (Pedantically, to test a release, I have to upload it into our internal repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact ends up not being final, then I'm screwed, because I would have to clear the local cache of any of our machines, dev/Jenkins/etc., that ever downloaded the formerly known as 1.1.1 but not really rc artifacts.) What's frustrating is that I know other Apache projects do rc releases, and even get them into Maven central, e.g.: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22 So, I apologize for the distraction from getting real work done, but perhaps you guys could find a creative way to work around the well-intentioned mandate on artifact voting? (E.g. perhaps have multiple votes, one for each successive rc (with -rcX suffix), then, once blessed, another one on the actually-final/no-rcX artifact (built from the last rc's tag); or publish no-rcX artifacts for official voting, as today, but then, at the same time, add -rcX artifacts to Maven central for non-binding/3rd party testing, etc.) Thanks, Stephen - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Awesome, sounds great, guys; thanks for understanding. Depending on how badly I need 1.1.1-rc2 (I'll check my jobs tomorrow) I'll just build a local version for now. Should be easy, it's just been awhile. :-) Thanks, Stephen On Sun Nov 23 2014 at 11:01:09 PM Patrick Wendell pwend...@gmail.com wrote: Hey Stephen, Thanks for bringing this up. Technically when we call a release vote it needs to be on the exact commit that will be the final release. However, one thing I've thought of doing for a while would be to publish the maven artifacts using a version tag with $VERSION-rcX even if the underlying commit has $VERSION in the pom files. Some recent changes I've made to the way we do publishing in branch 1.2 should make this pretty easy - it wasn't very easy before because we used maven's publishing plugin which makes modifying the published version tricky. Our current approach is, indeed, problematic because maven artifacts are supposed to be immutable once they have a specific version identifier. I created SPARK-4568 to track this: https://issues.apache.org/jira/browse/SPARK-4568 - Patrick On Sun, Nov 23, 2014 at 8:11 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Interesting, perhaps we could publish each one with two IDs, of which the rc one is unofficial. The problem is indeed that you have to vote on a hash for a potentially final artifact. Matei On Nov 23, 2014, at 7:54 PM, Stephen Haberman stephen.haber...@gmail.com wrote: Hi, I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but therc releases not being tagged with -rcX means the pre-built artifacts are basically useless to me. (Pedantically, to test a release, I have to upload it into our internal repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact ends up not being final, then I'm screwed, because I would have to clear the local cache of any of our machines, dev/Jenkins/etc., that ever downloaded the formerly known as 1.1.1 but not really rc artifacts.) What's frustrating is that I know other Apache projects do rc releases, and even get them into Maven central, e.g.: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org. apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22 So, I apologize for the distraction from getting real work done, but perhaps you guys could find a creative way to work around the well-intentioned mandate on artifact voting? (E.g. perhaps have multiple votes, one for each successive rc (with -rcX suffix), then, once blessed, another one on the actually-final/no-rcX artifact (built from the last rc's tag); or publish no-rcX artifacts for official voting, as today, but then, at the same time, add -rcX artifacts to Maven central for non-binding/3rd party testing, etc.) Thanks, Stephen - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
http://maven.apache.org/plugins/maven-install-plugin/ examples/specific-local-repo.html Hm, I didn't know about that plugin--assuming it does all of the jar/pom/sources/etc., then, yes, that could work... At first glance, I'm not sure it'll bring over the pom with all of the transitive dependencies (because at least this invocation is only pointing at a raw jar file to import), but I'll try it out tomorrow. That is a good point though; if either you or other people have already been solving this import a renamed-to-rc artifact in terribly simple/obvious ways that I've just missed, I'm happy to be enlightened. :-) Thanks, Stephen
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1 (non binding) Signatures and license looks good. I built the plain-vanilla distribution and ran tests. While I still see the Java 8 + Hive test failure, I think we've established this is ignorable. On Wed, Nov 19, 2014 at 11:51 PM, Andrew Or and...@databricks.com wrote: I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or and...@databricks.com: Please vote on releasing the following candidate as Apache Spark version 1 .1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1 Built successfully and ran the python examples. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-1-RC2-tp9439p9452.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
This is whatever was in http://people.apache.org/~andrewor14/spark-1 .1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem? Matei On Nov 20, 2014, at 11:50 AM, Hector Yee hector@gmail.com wrote: This is whatever was in http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com mailto:matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com mailto:hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com mailto:hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com mailto:van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com mailto:and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org mailto:dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org mailto:dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org mailto:dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org mailto:dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee http://google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee http://google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee google.com/+HectorYee http://google.com/+HectorYee
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Whoops I must have used the 1.2 preview and mixed them up. spark-shell -version shows version 1.2.0 Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to 1.2 On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem? Matei On Nov 20, 2014, at 11:50 AM, Hector Yee hector@gmail.com wrote: This is whatever was in http://people.apache.org/~andrewor14/spark-1 .1.1-rc2/ On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hector, is this a comment on 1.1.1 or on the 1.2 preview? Matei On Nov 20, 2014, at 11:39 AM, Hector Yee hector@gmail.com wrote: I think it is a race condition caused by netty deactivating a channel while it is active. Switched to nio and it works fine --conf spark.shuffle.blockTransferService=nio On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee hector@gmail.com wrote: I'm still seeing the fetch failed error and updated https://issues.apache.org/jira/browse/SPARK-3633 On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin van...@cloudera.com wrote: +1 (non-binding) . ran simple things on spark-shell . ran jobs in yarn client cluster modes, and standalone cluster mode On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee* -- Yee Yang Li Hector http://google.com/+HectorYee *google.com/+HectorYee http://google.com/+HectorYee*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or and...@databricks.com: Please vote on releasing the following candidate as Apache Spark version 1 .1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1. Checked version numbers and doc. Tested a few ML examples with Java 6 and verified some recently merged bug fixes. -Xiangrui On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or and...@databricks.com: Please vote on releasing the following candidate as Apache Spark version 1 .1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
+1 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min 2. Tested pyspark, mlib 2.1. statistics OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK 2.5. rdd operations OK 2.6. recommendation OK 2.7. Good work ! In 1.1.0, there was an error and my program used to hang (over memory allocation) consistently running validation using itertools, compute optimum rank, lambda,numofiterations/rmse; data - movielens medium dataset (1 million records) . It works well in 1.1.1 ! Cheers k/ On Wed, Nov 19, 2014 at 6:00 PM, Xiangrui Meng men...@gmail.com wrote: +1. Checked version numbers and doc. Tested a few ML examples with Java 6 and verified some recently merged bug fixes. -Xiangrui On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or and...@databricks.com wrote: I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or and...@databricks.com: Please vote on releasing the following candidate as Apache Spark version 1 .1.1. This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones are - [SPARK-3426] Sort-based shuffle compression settings are incompatible - [SPARK-3948] Stream corruption issues in sort-based shuffle - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached. Additionally, this candidate fixes two blockers from the previous RC: - [SPARK-4434] Cluster mode jar URLs are broken - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d): http://s.apache.org/p8 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/andrewor14.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1043/ The documentation corresponding to this release can be found at: http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.1.1! The vote is open until Saturday, November 22, at 23:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.1.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ Cheers, Andrew - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
