Re:[VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread witgo
-1 The following bug should be fixed: https://issues.apache.org/jira/browse/SPARK-2677‍ -- Original -- From: Tathagata Das;tathagata.das1...@gmail.com; Date: Sat, Jul 26, 2014 07:08 AM To: dev@spark.apache.orgdev@spark.apache.org; Subject: [VOTE] Release

Re:[VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Andrew Ash
Is that a regression since 1.0.0? On Jul 27, 2014 10:43 AM, witgo wi...@qq.com wrote: -1 The following bug should be fixed: https://issues.apache.org/jira/browse/SPARK-2677‍ -- Original -- From: Tathagata Das;tathagata.das1...@gmail.com; Date: Sat, Jul

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Patrick Wendell
SPARK-2677 is a long standing issue and not a regression. Also, as far as I can see there is no patch for it or clear understanding of the cause. This type of bug does not warrant holding a release. If we fix SPARK-2677 we can just make another release with the fix. On Sun, Jul 27, 2014 at 10:47

branch-1.1 will be cut on Friday

2014-07-27 Thread Patrick Wendell
Hey All, Just a heads up, we'll cut branch-1.1 on this Friday, August 1st. Once the release branch is cut we'll start community QA and go into the normal triage process for merging patches into that branch. For Spark core, we'll be conservative in merging things past the freeze date (e.g. high

Re: branch-1.1 will be cut on Friday

2014-07-27 Thread Nan Zhu
Good news, we will see the official version containing JDBC in very soon! Also, I have several pending PRs, can anyone continue the review process in this week? Avoid overwriting already-set SPARK_HOME in spark-submit: https://github.com/apache/spark/pull/1331 fix locality inversion bug in

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Patrick Wendell
Hey Ted, We always intend Spark to work with the newer Hadoop versions and encourage Spark users to use the newest Hadoop versions for best performance. We do try to be liberal in terms of supporting older versions as well. This is because many people run older HDFS versions and we want Spark to

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Sean Owen
Good idea, although it gets difficult in the context of multiple distributions. Say change X is not present in version A, but present in version B. If you depend on X, what version can you look for to detect it? The distribution will return A or A+X or somesuch, but testing for A will give an

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Matei Zaharia
+1 Tested this on Mac OS X. Matei On Jul 25, 2014, at 4:08 PM, Tathagata Das tathagata.das1...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.2. This release fixes a number of bugs in Spark 1.0.1. Some of the notable ones are - SPARK-2452:

Re: Utilize newer hadoop releases WAS: [VOTE] Release Apache Spark 1.0.2 (RC1)

2014-07-27 Thread Matei Zaharia
We could also do this, though it would be great if the Hadoop project provided this version number as at least a baseline. It's up to distributors to decide which version they report but I imagine they won't remove stuff that's in the reported version number. Matei On Jul 27, 2014, at 1:57

new JDBC server test cases seems failed ?

2014-07-27 Thread Nan Zhu
Hi, all It seems that the JDBC test cases are failed unexpectedly in Jenkins? [info] - test query execution against a Hive Thrift server *** FAILED *** [info] java.sql.SQLException: Could not open connection to jdbc:hive2://localhost:45518/: java.net.ConnectException: Connection refused

Re: new JDBC server test cases seems failed ?

2014-07-27 Thread Michael Armbrust
How recent is this? We've already reverted this patch once due to failing tests. It would be helpful to include a link to the failed build. If its failing again we'll have to revert again. On Sun, Jul 27, 2014 at 5:26 PM, Nan Zhu zhunanmcg...@gmail.com wrote: Hi, all It seems that the JDBC

Re: new JDBC server test cases seems failed ?

2014-07-27 Thread Patrick Wendell
I'm going to revert it again - Cheng can you try to look into this? Thanks. On Sun, Jul 27, 2014 at 6:06 PM, Nan Zhu zhunanmcg...@gmail.com wrote: it's 20 minutes ago https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17259/consoleFull -- Nan Zhu On Sunday, July 27, 2014

No such file or directory errors running tests

2014-07-27 Thread Stephen Boesch
I have pulled latest from github this afternoon. There are many many errors: source_home/assembly/target/scala-2.10: No such file or directory This causes many tests to fail. Here is the command line I am running mvn -Pyarn -Phadoop-2.3 -Phive package test

Re: No such file or directory errors running tests

2014-07-27 Thread Reynold Xin
To run through all the tests you'd need to create the assembly jar first. I've seen this asked a few times. Maybe we should make it more obvious. http://spark.apache.org/docs/latest/building-with-maven.html Spark Tests in Maven Tests are run by default via the ScalaTest Maven plugin

Re: No such file or directory errors running tests

2014-07-27 Thread Stephen Boesch
i Reynold, thanks for responding here. Yes I had looked at the building with maven page in the past. I have not noticed that the package step must happen *before *the test. I had assumed it were a corequisite -as seen in my command line. So the following sequence appears to work fine (so far

Can I translate the documentations of Spark in Japanese?

2014-07-27 Thread Yu Ishikawa
Hi all, I'm Yu Ishikawa, a Japanese. I would like to translate the documentations of Spark 1.0.x officially. If I will translate them and send a pull request, then can you merge it ? And where is the best directory to create the Japanese documentations ? Best, Yu -- View this message in

Re: No such file or directory errors running tests

2014-07-27 Thread Reynold Xin
Would you like to submit a pull request? All doc source code are in the docs folder. Cheers. On Sun, Jul 27, 2014 at 7:35 PM, Stephen Boesch java...@gmail.com wrote: i Reynold, thanks for responding here. Yes I had looked at the building with maven page in the past. I have not noticed

Re: No such file or directory errors running tests

2014-07-27 Thread Stephen Boesch
OK i'll do it after confirming all the tests run 2014-07-27 19:36 GMT-07:00 Reynold Xin r...@databricks.com: Would you like to submit a pull request? All doc source code are in the docs folder. Cheers. On Sun, Jul 27, 2014 at 7:35 PM, Stephen Boesch java...@gmail.com wrote: i Reynold,

Re: No such file or directory errors running tests

2014-07-27 Thread Steve Nunez
Whilst we¹re on this topic, I¹d be interested to see if you get hive failures. I¹m trying to build on a Mac using HDP and seem to be getting failures related to Parquet. I¹ll know for sure once I get in tomorrow and confirm with engineering, but this is likely because the version of Hive is

Re: No such file or directory errors running tests

2014-07-27 Thread Stephen Boesch
Hi Steve, I am running on the cdh5.0.0 VM (which is CentOS 6.5) Given the difference in O/S and Hadoop distro between us my results are not likely to be of direct help to you. But in any case i will let you know (likely offline). 2014-07-27 20:02 GMT-07:00 Steve Nunez snu...@hortonworks.com:

Re: Can I translate the documentations of Spark in Japanese?

2014-07-27 Thread Patrick Wendell
Hey Yu, I think we could definitely put a pointer to documentation in other languages that is hosted somewhere welse, but since we are not in a position to maintain this, I'm not sure we could merge it into the mainline Spark codebase. I'd be interested to know what other projects do about this