date:20160518

Re: SparkR dataframe error

2016-05-18 Thread Sun Rui

I saw it, but I can’t see the complete error message on it.
I mean the part after “error in invokingJava(…)”

> On May 19, 2016, at 08:37, Gayathri Murali  
> wrote:
> 
> There was a screenshot attached to my original email. If you did not get it, 
> attaching here again.
> 
> On Wed, May 18, 2016 at 5:27 PM, Sun Rui  > wrote:
> It’s wrong behaviour that head(df) outputs no row
> Could you send a screenshot displaying whole error message?
>> On May 19, 2016, at 08:12, Gayathri Murali > > wrote:
>> 
>> I am trying to run a basic example on Interactive R shell and run into the 
>> following error. Also note that head(df) does not display any rows. Can 
>> someone please help if I am missing something?
>> 
>> 
>> 
>>  Thanks
>> Gayathri
>> 
>>  邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
>> 共有 1 个附件
>> Screen Shot 2016-05-18 at 5.09.29 PM.png(155K)
>> 极速下载 
>> 
>>  在线预览 
>> 
> 
>  邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
> 共有 1 个附件
> Screen Shot 2016-05-18 at 5.09.29 PM.png(155K)
> 极速下载 
> 
>  在线预览 
>   Shot 2016-05-18 at 5.09.29 PM.png>

Re: SparkR dataframe error

2016-05-18 Thread Sun Rui

It’s wrong behaviour that head(df) outputs no row
Could you send a screenshot displaying whole error message?
> On May 19, 2016, at 08:12, Gayathri Murali  
> wrote:
> 
> I am trying to run a basic example on Interactive R shell and run into the 
> following error. Also note that head(df) does not display any rows. Can 
> someone please help if I am missing something?
> 
> 
> 
>  Thanks
> Gayathri
> 
>  邮件带有附件预览链接，若您转发或回复此邮件时不希望对方预览附件，建议您手动删除链接。
> 共有 1 个附件
> Screen Shot 2016-05-18 at 5.09.29 PM.png(155K)
> 极速下载 
> 
>  在线预览 
>

Re: Query parsing error for the join query between different database

2016-05-18 Thread JaeSung Jun

Thanks Herman,
I didn't recognise the "user" is reserved word. it works now.

On 19 May 2016 at 08:02, Herman van Hövell tot Westerflier <
hvanhov...@questtec.nl> wrote:

> 'User' is a SQL2003 keyword. This is normally not a problem, except when
> you use it as a table alias (which you are doing). Change the alias or
> place it between backticks and you should be fine.
>
>
> 2016-05-18 23:51 GMT+02:00 JaeSung Jun :
>
>> It's spark 1.6.1 and hive 1.2.1 (spark-sql saying "SET
>> spark.sql.hive.version=1.2.1").
>>
>> Thanks
>>
>> On 18 May 2016 at 23:31, Ted Yu  wrote:
>>
>>> Which release of Spark / Hive are you using ?
>>>
>>> Cheers
>>>
>>> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
>>>
>>> Hi,
>>>
>>> I'm working on custom data source provider, and i'm using fully
>>> qualified table name in FROM clause like following :
>>>
>>> SELECT user. uid, dept.name
>>> FROM userdb.user user, deptdb.dept
>>> WHERE user.dept_id = dept.id
>>>
>>> and i've got the following error :
>>>
>>> MismatchedTokenException(279!=26)
>>> at
>>> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
>>> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
>>> at
>>> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
>>> at
>>> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
>>> at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
>>> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
>>> at
>>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
>>> at
>>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
>>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>> at
>>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>> at
>>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>>> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>>> at
>>> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>>> at
>>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
>>> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
>>>
>>> Any idea?
>>>
>>> Thanks
>>> Jason
>>>
>>>
>>
>

Re: Query parsing error for the join query between different database

2016-05-18 Thread Herman van Hövell tot Westerflier

'User' is a SQL2003 keyword. This is normally not a problem, except when
you use it as a table alias (which you are doing). Change the alias or
place it between backticks and you should be fine.


2016-05-18 23:51 GMT+02:00 JaeSung Jun :

> It's spark 1.6.1 and hive 1.2.1 (spark-sql saying "SET
> spark.sql.hive.version=1.2.1").
>
> Thanks
>
> On 18 May 2016 at 23:31, Ted Yu  wrote:
>
>> Which release of Spark / Hive are you using ?
>>
>> Cheers
>>
>> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
>>
>> Hi,
>>
>> I'm working on custom data source provider, and i'm using fully qualified
>> table name in FROM clause like following :
>>
>> SELECT user. uid, dept.name
>> FROM userdb.user user, deptdb.dept
>> WHERE user.dept_id = dept.id
>>
>> and i've got the following error :
>>
>> MismatchedTokenException(279!=26)
>> at
>> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
>> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
>> at
>> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
>> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
>> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
>> at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
>> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
>> at
>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
>> at
>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at
>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at
>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>> at
>> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>> at
>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
>> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
>>
>> Any idea?
>>
>> Thanks
>> Jason
>>
>>
>

Re: Query parsing error for the join query between different database

2016-05-18 Thread JaeSung Jun

It's spark 1.6.1 and hive 1.2.1 (spark-sql saying "SET
spark.sql.hive.version=1.2.1").

Thanks

On 18 May 2016 at 23:31, Ted Yu  wrote:

> Which release of Spark / Hive are you using ?
>
> Cheers
>
> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
>
> Hi,
>
> I'm working on custom data source provider, and i'm using fully qualified
> table name in FROM clause like following :
>
> SELECT user. uid, dept.name
> FROM userdb.user user, deptdb.dept
> WHERE user.dept_id = dept.id
>
> and i've got the following error :
>
> MismatchedTokenException(279!=26)
> at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
> at
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
> at
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
>
> Any idea?
>
> Thanks
> Jason
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Reynold Xin

Hi Ovidiu-Cristian ,

The best source of truth is change the filter with target version to 2.1.0.
Not a lot of tickets have been targeted yet, but I'd imagine as we get
closer to 2.0 release, more will be retargeted at 2.1.0.



On Wed, May 18, 2016 at 10:43 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.ma...@inria.fr> wrote:

> Yes, I can filter..
> Did that and for example:
>
> https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
> 
>
> To rephrase: for 2.0 do you have specific issues that are not a priority
> and will released maybe with 2.1 for example?
>
> Keep up the good work!
>
> On 18 May 2016, at 18:19, Reynold Xin  wrote:
>
> You can find that by changing the filter to target version = 2.0.0. Cheers.
>
> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
> ovidiu-cristian.ma...@inria.fr> wrote:
>
>> +1 Great, I see the list of resolved issues, do you have a list of known
>> issue you plan to stay with this release?
>>
>> with
>> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
>> -Phive-thriftserver -DskipTests clean package
>>
>> mvn -version
>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> 2015-11-10T17:41:47+01:00)
>> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
>> Java version: 1.7.0_80, vendor: Oracle Corporation
>> Java home:
>> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>>
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Spark Project Parent POM ... SUCCESS [
>> 2.635 s]
>> [INFO] Spark Project Tags . SUCCESS [
>> 1.896 s]
>> [INFO] Spark Project Sketch ... SUCCESS [
>> 2.560 s]
>> [INFO] Spark Project Networking ... SUCCESS [
>> 6.533 s]
>> [INFO] Spark Project Shuffle Streaming Service  SUCCESS [
>> 4.176 s]
>> [INFO] Spark Project Unsafe ... SUCCESS [
>> 4.809 s]
>> [INFO] Spark Project Launcher . SUCCESS [
>> 6.242 s]
>> [INFO] Spark Project Core . SUCCESS
>> [01:20 min]
>> [INFO] Spark Project GraphX ... SUCCESS [
>> 9.148 s]
>> [INFO] Spark Project Streaming  SUCCESS [
>> 22.760 s]
>> [INFO] Spark Project Catalyst . SUCCESS [
>> 50.783 s]
>> [INFO] Spark Project SQL .. SUCCESS
>> [01:05 min]
>> [INFO] Spark Project ML Local Library . SUCCESS [
>> 4.281 s]
>> [INFO] Spark Project ML Library ... SUCCESS [
>> 54.537 s]
>> [INFO] Spark Project Tools  SUCCESS [
>> 0.747 s]
>> [INFO] Spark Project Hive . SUCCESS [
>> 33.032 s]
>> [INFO] Spark Project HiveContext Compatibility  SUCCESS [
>> 3.198 s]
>> [INFO] Spark Project REPL . SUCCESS [
>> 3.573 s]
>> [INFO] Spark Project YARN Shuffle Service . SUCCESS [
>> 4.617 s]
>> [INFO] Spark Project YARN . SUCCESS [
>> 7.321 s]
>> [INFO] Spark Project Hive Thrift Server ... SUCCESS [
>> 16.496 s]
>> [INFO] Spark Project Assembly . SUCCESS [
>> 2.300 s]
>> [INFO] Spark Project External Flume Sink .. SUCCESS [
>> 4.219 s]
>> [INFO] Spark Project External Flume ... SUCCESS [
>> 6.987 s]
>> [INFO] Spark Project External Flume Assembly .. SUCCESS [
>> 1.465 s]
>> [INFO] Spark Integration for Kafka 0.8  SUCCESS [
>> 6.891 s]
>> [INFO] Spark Project Examples . SUCCESS [
>> 13.465 s]
>> [INFO] Spark Project External Kafka Assembly .. SUCCESS [
>> 2.815 s]
>> [INFO]
>> 
>> [INFO] BUILD SUCCESS
>> [INFO]
>> 
>> [INFO] Total time: 07:04 min
>> [INFO] Finished at: 2016-05-18T17:55:33+02:00
>> [INFO] Final Memory: 90M/824M
>> [INFO]
>> 
>>
>> On 18 May 2016, at 16:28, Sean Owen  wrote:
>>
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>>
>> +1 to the release. License, sigs, e

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Ovidiu-Cristian MARCU

Yes, I can filter..
Did that and for example:
https://issues.apache.org/jira/browse/SPARK-15370?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%202.0.0
 


To rephrase: for 2.0 do you have specific issues that are not a priority and 
will released maybe with 2.1 for example?

Keep up the good work!

> On 18 May 2016, at 18:19, Reynold Xin  wrote:
> 
> You can find that by changing the filter to target version = 2.0.0. Cheers.
> 
> On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU 
> mailto:ovidiu-cristian.ma...@inria.fr>> 
> wrote:
> +1 Great, I see the list of resolved issues, do you have a list of known 
> issue you plan to stay with this release?
> 
> with
> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive 
> -Phive-thriftserver -DskipTests clean package
> 
> mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-10T17:41:47+01:00)
> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
> Java version: 1.7.0_80, vendor: Oracle Corporation
> Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Spark Project Parent POM ... SUCCESS [  2.635 
> s]
> [INFO] Spark Project Tags . SUCCESS [  1.896 
> s]
> [INFO] Spark Project Sketch ... SUCCESS [  2.560 
> s]
> [INFO] Spark Project Networking ... SUCCESS [  6.533 
> s]
> [INFO] Spark Project Shuffle Streaming Service  SUCCESS [  4.176 
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [  4.809 
> s]
> [INFO] Spark Project Launcher . SUCCESS [  6.242 
> s]
> [INFO] Spark Project Core . SUCCESS [01:20 
> min]
> [INFO] Spark Project GraphX ... SUCCESS [  9.148 
> s]
> [INFO] Spark Project Streaming  SUCCESS [ 22.760 
> s]
> [INFO] Spark Project Catalyst . SUCCESS [ 50.783 
> s]
> [INFO] Spark Project SQL .. SUCCESS [01:05 
> min]
> [INFO] Spark Project ML Local Library . SUCCESS [  4.281 
> s]
> [INFO] Spark Project ML Library ... SUCCESS [ 54.537 
> s]
> [INFO] Spark Project Tools  SUCCESS [  0.747 
> s]
> [INFO] Spark Project Hive . SUCCESS [ 33.032 
> s]
> [INFO] Spark Project HiveContext Compatibility  SUCCESS [  3.198 
> s]
> [INFO] Spark Project REPL . SUCCESS [  3.573 
> s]
> [INFO] Spark Project YARN Shuffle Service . SUCCESS [  4.617 
> s]
> [INFO] Spark Project YARN . SUCCESS [  7.321 
> s]
> [INFO] Spark Project Hive Thrift Server ... SUCCESS [ 16.496 
> s]
> [INFO] Spark Project Assembly . SUCCESS [  2.300 
> s]
> [INFO] Spark Project External Flume Sink .. SUCCESS [  4.219 
> s]
> [INFO] Spark Project External Flume ... SUCCESS [  6.987 
> s]
> [INFO] Spark Project External Flume Assembly .. SUCCESS [  1.465 
> s]
> [INFO] Spark Integration for Kafka 0.8  SUCCESS [  6.891 
> s]
> [INFO] Spark Project Examples . SUCCESS [ 13.465 
> s]
> [INFO] Spark Project External Kafka Assembly .. SUCCESS [  2.815 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 07:04 min
> [INFO] Finished at: 2016-05-18T17:55:33+02:00
> [INFO] Final Memory: 90M/824M
> [INFO] 
> 
> 
>> On 18 May 2016, at 16:28, Sean Owen > > wrote:
>> 
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>> 
>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>> 8, compilation and tests succeed for "-Pyarn -Phive
>> -Phive-thriftserver -Phadoop-2.6".
>> 
>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin > > wrote:
>>> Hi,
>>> 
>>> In the past the Apache Spark community have created preview packages (not
>>> official releases) an

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Michael Armbrust

+1, excited for 2.0!

On Wed, May 18, 2016 at 10:06 AM, Krishna Sankar 
wrote:

> +1. Looks Good.
> The mllib results are in line with 1.6.1. Deprecation messages. I will
> convert to ml and test later in the day.
> Also will try GraphX exercises for our Strata London Tutorial
>
> Quick Notes:
>
>1. pyspark env variables need to be changed
>- IPYTHON and IPYTHON_OPTS are removed
>   - This works
>  - PYSPARK_DRIVER_PYTHON=ipython
>  PYSPARK_DRIVER_PYTHON_OPTS="notebook"
>  ~/Downloads/spark-2.0.0-preview/bin/pyspark --packages
>  com.databricks:spark-csv_2.10:1.4.0
>   2.  maven 3.3.9 is required. (I was running 3.3.3)
>3.  Tons of interesting warnings and deprecations.
>   - The messages look descriptive and very helpful (Thanks. This will
>   help migration to 2.0, mllib -> ml et al). Will dig deeper.
>   4. Compiled OSX 10.10 (Yosemite) OK Total time: 31:28 min
> mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
>- Spark version is 2.0.0-preview
>   - Tested pyspark, mllib (iPython 4.2.0)
>
> Cheers & Good work folks
> 
>
> On Wed, May 18, 2016 at 7:28 AM, Sean Owen  wrote:
>
>> I think it's a good idea. Although releases have been preceded before
>> by release candidates for developers, it would be good to get a formal
>> preview/beta release ratified for public consumption ahead of a new
>> major release. Better to have a little more testing in the wild to
>> identify problems before 2.0.0 is finalized.
>>
>> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
>> 8, compilation and tests succeed for "-Pyarn -Phive
>> -Phive-thriftserver -Phadoop-2.6".
>>
>> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin  wrote:
>> > Hi,
>> >
>> > In the past the Apache Spark community have created preview packages
>> (not
>> > official releases) and used those as opportunities to ask community
>> members
>> > to test the upcoming versions of Apache Spark. Several people in the
>> Apache
>> > community have suggested we conduct votes for these preview packages and
>> > turn them into formal releases by the Apache foundation's standard.
>> Preview
>> > releases are not meant to be functional, i.e. they can and highly likely
>> > will contain critical bugs or documentation errors, but we will be able
>> to
>> > post them to the project's website to get wider feedback. They should
>> > satisfy the legal requirements of Apache's release policy
>> > (http://www.apache.org/dev/release.html) such as having proper
>> licenses.
>> >
>> >
>> > Please vote on releasing the following candidate as Apache Spark version
>> > 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
>> PDT
>> > and passes if a majority of at least 3 +1 PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is 2.0.0-preview
>> > (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> >
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/pwendell.asc
>> >
>> > The documentation corresponding to this release can be found at:
>> >
>> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>> >
>> > The list of resolved issues are:
>> >
>> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>> >
>> >
>> > If you are a Spark user, you can help us test this release by taking an
>> > existing Apache Spark workload and running on this candidate, then
>> reporting
>> > any regressions.
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Krishna Sankar

+1. Looks Good.
The mllib results are in line with 1.6.1. Deprecation messages. I will
convert to ml and test later in the day.
Also will try GraphX exercises for our Strata London Tutorial

Quick Notes:

   1. pyspark env variables need to be changed
   - IPYTHON and IPYTHON_OPTS are removed
  - This works
 - PYSPARK_DRIVER_PYTHON=ipython
 PYSPARK_DRIVER_PYTHON_OPTS="notebook"
 ~/Downloads/spark-2.0.0-preview/bin/pyspark --packages
 com.databricks:spark-csv_2.10:1.4.0
  2.  maven 3.3.9 is required. (I was running 3.3.3)
   3.  Tons of interesting warnings and deprecations.
  - The messages look descriptive and very helpful (Thanks. This will
  help migration to 2.0, mllib -> ml et al). Will dig deeper.
  4. Compiled OSX 10.10 (Yosemite) OK Total time: 31:28 min
mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
   - Spark version is 2.0.0-preview
  - Tested pyspark, mllib (iPython 4.2.0)

Cheers & Good work folks


On Wed, May 18, 2016 at 7:28 AM, Sean Owen  wrote:

> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
>
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
>
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin  wrote:
> > Hi,
> >
> > In the past the Apache Spark community have created preview packages (not
> > official releases) and used those as opportunities to ask community
> members
> > to test the upcoming versions of Apache Spark. Several people in the
> Apache
> > community have suggested we conduct votes for these preview packages and
> > turn them into formal releases by the Apache foundation's standard.
> Preview
> > releases are not meant to be functional, i.e. they can and highly likely
> > will contain critical bugs or documentation errors, but we will be able
> to
> > post them to the project's website to get wider feedback. They should
> > satisfy the legal requirements of Apache's release policy
> > (http://www.apache.org/dev/release.html) such as having proper licenses.
> >
> >
> > Please vote on releasing the following candidate as Apache Spark version
> > 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM
> PDT
> > and passes if a majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 2.0.0-preview
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is 2.0.0-preview
> > (8f5a04b6299e3a47aca13cbb40e72344c0114860)
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The documentation corresponding to this release can be found at:
> >
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
> >
> > The list of resolved issues are:
> >
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
> >
> >
> > If you are a Spark user, you can help us test this release by taking an
> > existing Apache Spark workload and running on this candidate, then
> reporting
> > any regressions.
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Reynold Xin

Michael,

You can comment on the JIRA ticket and tag some of the more active
contributors to Mesos/SparkR. That said, committers are focusing on bug
fixes and stability fixes at the moment for 2.0, and it's unlikely at this
point for new features to get in. It can of course target the next release.




On Wed, May 18, 2016 at 9:32 AM, Michael F Ringenburg  wrote:

> Hi,
>
> Are there any plans to address SPARK-11524
>  (Support SparkR with
> Mesos cluster) for the 2.0 release?
>
> This is a fairly big issue for people using Spark with Mesos.
>
> Thanks!
> Mike
>
> From:  on behalf of Reynold Xin 
> Date: Wednesday, May 18, 2016 at 6:40 AM
> To: "dev@spark.apache.org" 
> Subject: [vote] Apache Spark 2.0.0-preview release (rc1)
>
> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy (
> http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then
> reporting any regressions.
>
>

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Reynold Xin

You can find that by changing the filter to target version = 2.0.0. Cheers.

On Wed, May 18, 2016 at 9:00 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.ma...@inria.fr> wrote:

> +1 Great, I see the list of resolved issues, do you have a list of known
> issue you plan to stay with this release?
>
> with
> build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive
> -Phive-thriftserver -DskipTests clean package
>
> mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T17:41:47+01:00)
> Maven home: /Users/omarcu/tools/apache-maven-3.3.9
> Java version: 1.7.0_80, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [
> 2.635 s]
> [INFO] Spark Project Tags . SUCCESS [
> 1.896 s]
> [INFO] Spark Project Sketch ... SUCCESS [
> 2.560 s]
> [INFO] Spark Project Networking ... SUCCESS [
> 6.533 s]
> [INFO] Spark Project Shuffle Streaming Service  SUCCESS [
> 4.176 s]
> [INFO] Spark Project Unsafe ... SUCCESS [
> 4.809 s]
> [INFO] Spark Project Launcher . SUCCESS [
> 6.242 s]
> [INFO] Spark Project Core . SUCCESS [01:20
> min]
> [INFO] Spark Project GraphX ... SUCCESS [
> 9.148 s]
> [INFO] Spark Project Streaming  SUCCESS [
> 22.760 s]
> [INFO] Spark Project Catalyst . SUCCESS [
> 50.783 s]
> [INFO] Spark Project SQL .. SUCCESS [01:05
> min]
> [INFO] Spark Project ML Local Library . SUCCESS [
> 4.281 s]
> [INFO] Spark Project ML Library ... SUCCESS [
> 54.537 s]
> [INFO] Spark Project Tools  SUCCESS [
> 0.747 s]
> [INFO] Spark Project Hive . SUCCESS [
> 33.032 s]
> [INFO] Spark Project HiveContext Compatibility  SUCCESS [
> 3.198 s]
> [INFO] Spark Project REPL . SUCCESS [
> 3.573 s]
> [INFO] Spark Project YARN Shuffle Service . SUCCESS [
> 4.617 s]
> [INFO] Spark Project YARN . SUCCESS [
> 7.321 s]
> [INFO] Spark Project Hive Thrift Server ... SUCCESS [
> 16.496 s]
> [INFO] Spark Project Assembly . SUCCESS [
> 2.300 s]
> [INFO] Spark Project External Flume Sink .. SUCCESS [
> 4.219 s]
> [INFO] Spark Project External Flume ... SUCCESS [
> 6.987 s]
> [INFO] Spark Project External Flume Assembly .. SUCCESS [
> 1.465 s]
> [INFO] Spark Integration for Kafka 0.8  SUCCESS [
> 6.891 s]
> [INFO] Spark Project Examples . SUCCESS [
> 13.465 s]
> [INFO] Spark Project External Kafka Assembly .. SUCCESS [
> 2.815 s]
> [INFO]
> 
> [INFO] BUILD SUCCESS
> [INFO]
> 
> [INFO] Total time: 07:04 min
> [INFO] Finished at: 2016-05-18T17:55:33+02:00
> [INFO] Final Memory: 90M/824M
> [INFO]
> 
>
> On 18 May 2016, at 16:28, Sean Owen  wrote:
>
> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
>
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
>
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin  wrote:
>
> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy
> (http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Ovidiu-Cristian MARCU

+1 Great, I see the list of resolved issues, do you have a list of known issue 
you plan to stay with this release?

with
build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Phive -Phive-thriftserver 
-DskipTests clean package

mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T17:41:47+01:00)
Maven home: /Users/omarcu/tools/apache-maven-3.3.9
Java version: 1.7.0_80, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.5", arch: "x86_64", family: “mac"

[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ... SUCCESS [  2.635 s]
[INFO] Spark Project Tags . SUCCESS [  1.896 s]
[INFO] Spark Project Sketch ... SUCCESS [  2.560 s]
[INFO] Spark Project Networking ... SUCCESS [  6.533 s]
[INFO] Spark Project Shuffle Streaming Service  SUCCESS [  4.176 s]
[INFO] Spark Project Unsafe ... SUCCESS [  4.809 s]
[INFO] Spark Project Launcher . SUCCESS [  6.242 s]
[INFO] Spark Project Core . SUCCESS [01:20 min]
[INFO] Spark Project GraphX ... SUCCESS [  9.148 s]
[INFO] Spark Project Streaming  SUCCESS [ 22.760 s]
[INFO] Spark Project Catalyst . SUCCESS [ 50.783 s]
[INFO] Spark Project SQL .. SUCCESS [01:05 min]
[INFO] Spark Project ML Local Library . SUCCESS [  4.281 s]
[INFO] Spark Project ML Library ... SUCCESS [ 54.537 s]
[INFO] Spark Project Tools  SUCCESS [  0.747 s]
[INFO] Spark Project Hive . SUCCESS [ 33.032 s]
[INFO] Spark Project HiveContext Compatibility  SUCCESS [  3.198 s]
[INFO] Spark Project REPL . SUCCESS [  3.573 s]
[INFO] Spark Project YARN Shuffle Service . SUCCESS [  4.617 s]
[INFO] Spark Project YARN . SUCCESS [  7.321 s]
[INFO] Spark Project Hive Thrift Server ... SUCCESS [ 16.496 s]
[INFO] Spark Project Assembly . SUCCESS [  2.300 s]
[INFO] Spark Project External Flume Sink .. SUCCESS [  4.219 s]
[INFO] Spark Project External Flume ... SUCCESS [  6.987 s]
[INFO] Spark Project External Flume Assembly .. SUCCESS [  1.465 s]
[INFO] Spark Integration for Kafka 0.8  SUCCESS [  6.891 s]
[INFO] Spark Project Examples . SUCCESS [ 13.465 s]
[INFO] Spark Project External Kafka Assembly .. SUCCESS [  2.815 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 07:04 min
[INFO] Finished at: 2016-05-18T17:55:33+02:00
[INFO] Final Memory: 90M/824M
[INFO] 

> On 18 May 2016, at 16:28, Sean Owen  wrote:
> 
> I think it's a good idea. Although releases have been preceded before
> by release candidates for developers, it would be good to get a formal
> preview/beta release ratified for public consumption ahead of a new
> major release. Better to have a little more testing in the wild to
> identify problems before 2.0.0 is finalized.
> 
> +1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
> 8, compilation and tests succeed for "-Pyarn -Phive
> -Phive-thriftserver -Phadoop-2.6".
> 
> On Wed, May 18, 2016 at 6:40 AM, Reynold Xin  wrote:
>> Hi,
>> 
>> In the past the Apache Spark community have created preview packages (not
>> official releases) and used those as opportunities to ask community members
>> to test the upcoming versions of Apache Spark. Several people in the Apache
>> community have suggested we conduct votes for these preview packages and
>> turn them into formal releases by the Apache foundation's standard. Preview
>> releases are not meant to be functional, i.e. they can and highly likely
>> will contain critical bugs or documentation errors, but we will be able to
>> post them to the project's website to get wider feedback. They should
>> satisfy the legal requirements of Apache's release policy
>> (http://www.apache.org/dev/release.html) such as having proper licenses.
>> 
>> 
>> Please vote on releasing the following candidate as Apache Spark version
>> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
>> and passes if a majority of at least 3 +1 PMC votes are cast.
>> 
>> [ ] +1 Release this package as Apache Spark 2.0.0-preview
>> [ ] -1 Do not release this package because ...
>> 
>> To learn

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

2016-05-18 Thread Sean Owen

I think it's a good idea. Although releases have been preceded before
by release candidates for developers, it would be good to get a formal
preview/beta release ratified for public consumption ahead of a new
major release. Better to have a little more testing in the wild to
identify problems before 2.0.0 is finalized.

+1 to the release. License, sigs, etc check out. On Ubuntu 16 + Java
8, compilation and tests succeed for "-Pyarn -Phive
-Phive-thriftserver -Phadoop-2.6".

On Wed, May 18, 2016 at 6:40 AM, Reynold Xin  wrote:
> Hi,
>
> In the past the Apache Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the Apache
> community have suggested we conduct votes for these preview packages and
> turn them into formal releases by the Apache foundation's standard. Preview
> releases are not meant to be functional, i.e. they can and highly likely
> will contain critical bugs or documentation errors, but we will be able to
> post them to the project's website to get wider feedback. They should
> satisfy the legal requirements of Apache's release policy
> (http://www.apache.org/dev/release.html) such as having proper licenses.
>
>
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.0-preview. The vote is open until Friday, May 20, 2015 at 11:00 PM PDT
> and passes if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is 2.0.0-preview
> (8f5a04b6299e3a47aca13cbb40e72344c0114860)
>
> The release files, including signatures, digests, etc. can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The documentation corresponding to this release can be found at:
> http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/
>
> The list of resolved issues are:
> https://issues.apache.org/jira/browse/SPARK-15351?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.0.0
>
>
> If you are a Spark user, you can help us test this release by taking an
> existing Apache Spark workload and running on this candidate, then reporting
> any regressions.
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: CompileException for spark-sql generated code in 2.0.0-SNAPSHOT

2016-05-18 Thread Koert Kuipers

https://issues.apache.org/jira/browse/SPARK-15384

On Tue, May 17, 2016 at 9:06 PM, Michael Armbrust 
wrote:

> Yeah, can you open a JIRA with that reproduction please?  You can ping me
> on it.
>
> On Tue, May 17, 2016 at 4:55 PM, Reynold Xin  wrote:
>
>> It seems like the problem here is that we are not using unique names
>> for mapelements_isNull?
>>
>>
>>
>> On Tue, May 17, 2016 at 3:29 PM, Koert Kuipers  wrote:
>>
>>> hello all, we are slowly expanding our test coverage for spark
>>> 2.0.0-SNAPSHOT to more in-house projects. today i ran into this issue...
>>>
>>> this runs fine:
>>> val df = sc.parallelize(List(("1", "2"), ("3", "4"))).toDF("a", "b")
>>> df
>>>   .map(row => row)(RowEncoder(df.schema))
>>>   .select("a", "b")
>>>   .show
>>>
>>> however this fails:
>>> val df = sc.parallelize(List(("1", "2"), ("3", "4"))).toDF("a", "b")
>>> df
>>>   .map(row => row)(RowEncoder(df.schema))
>>>   .select("b", "a")
>>>   .show
>>>
>>> the error is:
>>> java.lang.Exception: failed to compile:
>>> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
>>> 94, Column 57: Expression "mapelements_isNull" is not an rvalue
>>> /* 001 */ public Object generate(Object[] references) {
>>> /* 002 */   return new GeneratedIterator(references);
>>> /* 003 */ }
>>> /* 004 */
>>> /* 005 */ /** Codegened pipeline for:
>>> /* 006 */ * Project [b#11,a#10]
>>> /* 007 */ +- SerializeFromObject [if (input[0,
>>> org.apache.spark.sql.Row].isNullAt) null else staticinvoke(class org.ap...
>>> /* 008 */   */
>>> /* 009 */ final class GeneratedIterator extends
>>> org.apache.spark.sql.execution.BufferedRowIterator {
>>> /* 010 */   private Object[] references;
>>> /* 011 */   private scala.collection.Iterator inputadapter_input;
>>> /* 012 */   private UnsafeRow project_result;
>>> /* 013 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
>>> project_holder;
>>> /* 014 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
>>> project_rowWriter;
>>> /* 015 */   private Object[] deserializetoobject_values;
>>> /* 016 */   private org.apache.spark.sql.types.StructType
>>> deserializetoobject_schema;
>>> /* 017 */   private UnsafeRow deserializetoobject_result;
>>> /* 018 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
>>> deserializetoobject_holder;
>>> /* 019 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
>>> deserializetoobject_rowWriter;
>>> /* 020 */   private UnsafeRow mapelements_result;
>>> /* 021 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
>>> mapelements_holder;
>>> /* 022 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
>>> mapelements_rowWriter;
>>> /* 023 */   private UnsafeRow serializefromobject_result;
>>> /* 024 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
>>> serializefromobject_holder;
>>> /* 025 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
>>> serializefromobject_rowWriter;
>>> /* 026 */   private UnsafeRow project_result1;
>>> /* 027 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
>>> project_holder1;
>>> /* 028 */   private
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
>>> project_rowWriter1;
>>> /* 029 */
>>> /* 030 */   public GeneratedIterator(Object[] references) {
>>> /* 031 */ this.references = references;
>>> /* 032 */   }
>>> /* 033 */
>>> /* 034 */   public void init(int index, scala.collection.Iterator
>>> inputs[]) {
>>> /* 035 */ partitionIndex = index;
>>> /* 036 */ inputadapter_input = inputs[0];
>>> /* 037 */ project_result = new UnsafeRow(2);
>>> /* 038 */ this.project_holder = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result,
>>> 64);
>>> /* 039 */ this.project_rowWriter = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
>>> 2);
>>> /* 040 */
>>> /* 041 */ this.deserializetoobject_schema =
>>> (org.apache.spark.sql.types.StructType) references[0];
>>> /* 042 */ deserializetoobject_result = new UnsafeRow(1);
>>> /* 043 */ this.deserializetoobject_holder = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(deserializetoobject_result,
>>> 32);
>>> /* 044 */ this.deserializetoobject_rowWriter = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(deserializetoobject_holder,
>>> 1);
>>> /* 045 */ mapelements_result = new UnsafeRow(1);
>>> /* 046 */ this.mapelements_holder = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(mapelements_result,
>>> 32);
>>> /* 047 */ this.mapelements_rowWriter = new
>>> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(mapelements_holder,
>>> 1);
>>> /* 048 */ serializefromobject_result = new UnsafeRow(2);
>>> /* 049 */

PR for In-App Scheduling

2016-05-18 Thread Nick White

Hi  I raised a PR here: https://github.com/apache/spark/pull/12951 add a
mechanism that prevents starvation when scheduling work within a single
application. Could a committer take a look? Thanks -

Nick




smime.p7s
Description: S/MIME cryptographic signature

Re: Query parsing error for the join query between different database

2016-05-18 Thread Ted Yu

Which release of Spark / Hive are you using ?

Cheers

> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
> 
> Hi,
> 
> I'm working on custom data source provider, and i'm using fully qualified 
> table name in FROM clause like following :
> 
> SELECT user. uid, dept.name
> FROM userdb.user user, deptdb.dept
> WHERE user.dept_id = dept.id
> 
> and i've got the following error :
> 
> MismatchedTokenException(279!=26)
>   at 
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
>   at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
>   at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
>   at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
>   at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
>   at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
>   at 
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
>   at 
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
>   at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>   at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>   at 
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>   at 
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>   at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>   at 
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>   at 
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>   at 
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>   at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>   at 
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>   at 
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>   at 
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
>   at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
> 
> Any idea?
> 
> Thanks
> Jason

Query parsing error for the join query between different database

2016-05-18 Thread JaeSung Jun

Hi,

I'm working on custom data source provider, and i'm using fully qualified
table name in FROM clause like following :

SELECT user. uid, dept.name
FROM userdb.user user, deptdb.dept
WHERE user.dept_id = dept.id

and i've got the following error :

MismatchedTokenException(279!=26)
at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
at
org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
at
org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
at
org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
at
scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
at
scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
at
scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
at
org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)

Any idea?

Thanks
Jason

Re: SparkR dataframe error

Re: SparkR dataframe error

Re: Query parsing error for the join query between different database

Re: Query parsing error for the join query between different database

Re: Query parsing error for the join query between different database

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Re: CompileException for spark-sql generated code in 2.0.0-SNAPSHOT

PR for In-App Scheduling

Re: Query parsing error for the join query between different database

Query parsing error for the join query between different database

17 matches

Site Navigation

Mail list logo

Footer information