Re: Nested "struct" fonction call creates a compilation error in Spark SQL
You might also try with a newer version. Several instance of code generation failures have been fixed since 2.0. On Thu, Jun 15, 2017 at 1:15 PM, Olivier Girardot < o.girar...@lateral-thoughts.com> wrote: > Hi Michael, > Spark 2.0.2 - but I have a very interesting test case actually > The optimiser seems to be at fault in a way, I've joined to this email the > explain when I limit myself to 2 levels of struct mutation and when it goes > to 5. > As you can see the optimiser seems to be doing a lot more in the later > case. > After further investigation, the code is not "failing" per se - spark is > trying the whole stage codegen, the compilation is failing due to the > compilation error and I think it's falling back to the "non codegen" way. > > I'll try to create a simpler test case to reproduce this if I can, what do > you think ? > > Regards, > > Olivier. > > > 2017-06-15 21:08 GMT+02:00 Michael Armbrust <mich...@databricks.com>: > >> Which version of Spark? If its recent I'd open a JIRA. >> >> On Thu, Jun 15, 2017 at 6:04 AM, Olivier Girardot < >> o.girar...@lateral-thoughts.com> wrote: >> >>> Hi everyone, >>> when we create recursive calls to "struct" (up to 5 levels) for >>> extending a complex datastructure we end up with the following compilation >>> error : >>> >>> org.codehaus.janino.JaninoRuntimeException: Code of method >>> "(I[Lscala/collection/Iterator;)V" of class >>> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" >>> grows beyond 64 KB >>> >>> The CreateStruct code itself is properly using the ctx.splitExpression >>> command but the "end result" of the df.select( struct(struct(struct() >>> ))) ends up being too much. >>> >>> Should I open a JIRA or is there a workaround ? >>> >>> Regards, >>> >>> -- >>> *Olivier Girardot* | Associé >>> o.girar...@lateral-thoughts.com >>> >> >> > > > -- > *Olivier Girardot* | Associé > o.girar...@lateral-thoughts.com > +33 6 24 09 17 94 >
Re: Nested "struct" fonction call creates a compilation error in Spark SQL
Which version of Spark? If its recent I'd open a JIRA. On Thu, Jun 15, 2017 at 6:04 AM, Olivier Girardot < o.girar...@lateral-thoughts.com> wrote: > Hi everyone, > when we create recursive calls to "struct" (up to 5 levels) for extending > a complex datastructure we end up with the following compilation error : > > org.codehaus.janino.JaninoRuntimeException: Code of method > "(I[Lscala/collection/Iterator;)V" of class "org.apache.spark.sql. > catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KB > > The CreateStruct code itself is properly using the ctx.splitExpression > command but the "end result" of the df.select( struct(struct(struct() > ))) ends up being too much. > > Should I open a JIRA or is there a workaround ? > > Regards, > > -- > *Olivier Girardot* | Associé > o.girar...@lateral-thoughts.com >
Nested "struct" fonction call creates a compilation error in Spark SQL
Hi everyone, when we create recursive calls to "struct" (up to 5 levels) for extending a complex datastructure we end up with the following compilation error : org.codehaus.janino.JaninoRuntimeException: Code of method "(I[Lscala/collection/Iterator;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KB The CreateStruct code itself is properly using the ctx.splitExpression command but the "end result" of the df.select( struct(struct(struct() ))) ends up being too much. Should I open a JIRA or is there a workaround ? Regards, -- *Olivier Girardot* | Associé o.girar...@lateral-thoughts.com
Re: TestSQLContext compilation error when run SparkPi in Intellij ?
Thanks Andrew. On Sun, Aug 16, 2015 at 1:53 PM, Andrew Or and...@databricks.com wrote: Hi Canan, TestSQLContext is no longer a singleton but now a class. It is never meant to be a fully public API, but if you wish to use it you can just instantiate a new one: val sqlContext = new TestSQLContext or just create a new SQLContext from a SparkContext. -Andrew 2015-08-15 20:33 GMT-07:00 canan chen ccn...@gmail.com: I am not sure other people's spark debugging environment ( I mean for the master branch) , Anyone can share his experience ? On Sun, Aug 16, 2015 at 10:40 AM, canan chen ccn...@gmail.com wrote: I import the spark source code to intellij, and want to run SparkPi in intellij, but meet the folliwing weird compilation error? I googled it and sbt clean doesn't work for me. I am not sure whether anyone else has meet this issue also, any help is appreciated Error:scalac: while compiling: /Users/root/github/spark/sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: -nobootcp -javabootclasspath : -deprecation -feature -classpath
TestSQLContext compilation error when run SparkPi in Intellij ?
I import the spark source code to intellij, and want to run SparkPi in intellij, but meet the folliwing weird compilation error? I googled it and sbt clean doesn't work for me. I am not sure whether anyone else has meet this issue also, any help is appreciated Error:scalac: while compiling: /Users/root/github/spark/sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: -nobootcp -javabootclasspath : -deprecation -feature -classpath
Re: TestSQLContext compilation error when run SparkPi in Intellij ?
Hi Canan, TestSQLContext is no longer a singleton but now a class. It is never meant to be a fully public API, but if you wish to use it you can just instantiate a new one: val sqlContext = new TestSQLContext or just create a new SQLContext from a SparkContext. -Andrew 2015-08-15 20:33 GMT-07:00 canan chen ccn...@gmail.com: I am not sure other people's spark debugging environment ( I mean for the master branch) , Anyone can share his experience ? On Sun, Aug 16, 2015 at 10:40 AM, canan chen ccn...@gmail.com wrote: I import the spark source code to intellij, and want to run SparkPi in intellij, but meet the folliwing weird compilation error? I googled it and sbt clean doesn't work for me. I am not sure whether anyone else has meet this issue also, any help is appreciated Error:scalac: while compiling: /Users/root/github/spark/sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: -nobootcp -javabootclasspath : -deprecation -feature -classpath
Re: TestSQLContext compilation error when run SparkPi in Intellij ?
I am not sure other people's spark debugging environment ( I mean for the master branch) , Anyone can share his experience ? On Sun, Aug 16, 2015 at 10:40 AM, canan chen ccn...@gmail.com wrote: I import the spark source code to intellij, and want to run SparkPi in intellij, but meet the folliwing weird compilation error? I googled it and sbt clean doesn't work for me. I am not sure whether anyone else has meet this issue also, any help is appreciated Error:scalac: while compiling: /Users/root/github/spark/sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: -nobootcp -javabootclasspath : -deprecation -feature -classpath
Compilation error
I am trying out streaming example as documented and I am using spark 1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
*private* *void* map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
*new* *FlatMapFunctionString, String()* {
@Override *public* IterableString call(String x) {
*return* Arrays.*asList*(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.*map*(
*new* *PairFunctionString, String, Integer()* {
@Override *public* *Tuple2*String, Integer call(String s) *throws*
Exception {
*return* *new* *Tuple2*String, Integer(s, 1);
}
});
}
Re: Compilation error
A couple points:
You've got mismatched versions here -- 1.2.0 vs 1.2.1. You should fix
that but it's not your problem.
These are also supposed to be 'provided' scope dependencies in Maven.
You should get the Scala deps transitively and can import scala.*
classes. However, it would be a little bit more correct to depend
directly on the scala library classes, but in practice, easiest not to
in simple use cases.
If you're still having trouble look at the output of mvn dependency:tree
On Tue, Mar 10, 2015 at 6:32 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
I am using maven and my dependency looks like this, but this doesn't seem to
be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have to
download all the recursive dependencies. May be you should create a Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Compilation error
, 2015 at 11:40 AM, Sean Owen so...@cloudera.com wrote:
A couple points:
You've got mismatched versions here -- 1.2.0 vs 1.2.1. You should fix
that but it's not your problem.
These are also supposed to be 'provided' scope dependencies in Maven.
You should get the Scala deps transitively and can import scala.*
classes. However, it would be a little bit more correct to depend
directly on the scala library classes, but in practice, easiest not to
in simple use cases.
If you're still having trouble look at the output of mvn
dependency:tree
On Tue, Mar 10, 2015 at 6:32 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am using maven and my dependency looks like this, but this doesn't
seem to
be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com
wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all
the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head
what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you
have to
download all the recursive dependencies. May be you should create a
Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia
mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com
wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia
mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using
spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2
class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error
If you are using tools like SBT/Maven/Gradle/etc, they figure out all the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have to
download all the recursive dependencies. May be you should create a Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not able
to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error
You have to include Scala libraries in the Eclipse dependencies.
TD
On Tue, Mar 10, 2015 at 10:54 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark 1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
*private* *void* map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
*new* *FlatMapFunctionString, String()* {
@Override *public* IterableString call(String x) {
*return* Arrays.*asList*(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.*map*(
*new* *PairFunctionString, String, Integer()* {
@Override *public* *Tuple2*String, Integer call(String s) *throws*
Exception {
*return* *new* *Tuple2*String, Integer(s, 1);
}
});
}
Re: Compilation error
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not able
to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws Exception
{
return new Tuple2String, Integer(s, 1);
}
});
}
Compilation error on JavaPairDStream
I am getting following error. When I look at the sources it seems to be a
scala source, but not sure why it's complaining about it.
The method map(FunctionString,R) in the type JavaDStreamString is not
applicable for the arguments (new
PairFunctionString,String,Integer(){})
And my code has been taken from the spark examples site:
JavaPairDStreamString, Integer pairs = words.*map*(
*new* *PairFunctionString, String, Integer()* {
@Override *public* Tuple2String, Integer call(String s) *throws*
Exception {
*return* *new* Tuple2String, Integer(s, 1);
}
});
Re: Compilation error
should fix
that but it's not your problem.
These are also supposed to be 'provided' scope dependencies in Maven.
You should get the Scala deps transitively and can import scala.*
classes. However, it would be a little bit more correct to depend
directly on the scala library classes, but in practice, easiest not to
in simple use cases.
If you're still having trouble look at the output of mvn dependency:tree
On Tue, Mar 10, 2015 at 6:32 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am using maven and my dependency looks like this, but this doesn't
seem to
be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com
wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all
the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you
have to
download all the recursive dependencies. May be you should create a
Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia
mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com
wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia
mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using
spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error on JavaPairDStream
Ah, that's a typo in the example: use words.mapToPair
I can make a little PR to fix that.
On Tue, Mar 10, 2015 at 8:32 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
I am getting following error. When I look at the sources it seems to be a
scala source, but not sure why it's complaining about it.
The method map(FunctionString,R) in the type JavaDStreamString is not
applicable for the arguments (new
PairFunctionString,String,Integer(){})
And my code has been taken from the spark examples site:
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws Exception {
return new Tuple2String, Integer(s, 1);
}
});
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
RE: Compilation error
Or another option is to use Scala-IDE, which is built on top of Eclipse,
instead of pure Eclipse, so Scala comes with it.
Yong
From: so...@cloudera.com
Date: Tue, 10 Mar 2015 18:40:44 +
Subject: Re: Compilation error
To: mohitanch...@gmail.com
CC: t...@databricks.com; user@spark.apache.org
A couple points:
You've got mismatched versions here -- 1.2.0 vs 1.2.1. You should fix
that but it's not your problem.
These are also supposed to be 'provided' scope dependencies in Maven.
You should get the Scala deps transitively and can import scala.*
classes. However, it would be a little bit more correct to depend
directly on the scala library classes, but in practice, easiest not to
in simple use cases.
If you're still having trouble look at the output of mvn dependency:tree
On Tue, Mar 10, 2015 at 6:32 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
I am using maven and my dependency looks like this, but this doesn't seem to
be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have to
download all the recursive dependencies. May be you should create a Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Compilation error
I am using maven and my dependency looks like this, but this doesn't seem
to be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have to
download all the recursive dependencies. May be you should create a Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error
See if you can import scala libraries in your project.
On Tue, Mar 10, 2015 at 11:32 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am using maven and my dependency looks like this, but this doesn't seem
to be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com
wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have to
download all the recursive dependencies. May be you should create a Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error
to
be working
dependencies
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-streaming_2.10/artifactId
version1.2.0/version
/dependency
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-core_2.10/artifactId
version1.2.1/version
/dependency
/dependencies
On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das t...@databricks.com
wrote:
If you are using tools like SBT/Maven/Gradle/etc, they figure out all
the
recursive dependencies and includes them in the class path. I haven't
touched Eclipse in years so I am not sure off the top of my head what's
going on instead. Just in case you only downloaded the
spark-streaming_2.10.jar then that is indeed insufficient and you have
to
download all the recursive dependencies. May be you should create a
Maven
project inside Eclipse?
TD
On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia mohitanch...@gmail.com
wrote:
How do I do that? I haven't used Scala before.
Also, linking page doesn't mention that:
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking
On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen so...@cloudera.com
wrote:
It means you do not have Scala library classes in your project
classpath.
On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia
mohitanch...@gmail.com
wrote:
I am trying out streaming example as documented and I am using spark
1.2.1
streaming from maven for Java.
When I add this code I get compilation error on and eclipse is not
able to
recognize Tuple2. I also don't see any import scala.Tuple2 class.
http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example
private void map(JavaReceiverInputDStreamString lines) {
JavaDStreamString words = lines.flatMap(
new FlatMapFunctionString, String() {
@Override public IterableString call(String x) {
return Arrays.asList(x.split( ));
}
});
// Count each word in each batch
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws
Exception {
return new Tuple2String, Integer(s, 1);
}
});
}
Re: Compilation error on JavaPairDStream
works now. I should have checked :)
On Tue, Mar 10, 2015 at 1:44 PM, Sean Owen so...@cloudera.com wrote:
Ah, that's a typo in the example: use words.mapToPair
I can make a little PR to fix that.
On Tue, Mar 10, 2015 at 8:32 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am getting following error. When I look at the sources it seems to be a
scala source, but not sure why it's complaining about it.
The method map(FunctionString,R) in the type JavaDStreamString is not
applicable for the arguments (new
PairFunctionString,String,Integer(){})
And my code has been taken from the spark examples site:
JavaPairDStreamString, Integer pairs = words.map(
new PairFunctionString, String, Integer() {
@Override public Tuple2String, Integer call(String s) throws Exception
{
return new Tuple2String, Integer(s, 1);
}
});
Re: scala RDD sortby compilation error
I'm using the same code https://github.com/apache/spark/blob/83b7a1c6503adce1826fc537b4db47e534da5cae/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala#L687, though still receive not enough arguments for method sortBy: (f: String = K, ascending: Boolean, numPartitions: Int)(implicit ord: Ordering[K], implicit ctag: scala.reflect.ClassTag[K])org.apache.spark.rdd.RDD[String]. Unspecified value parameter f. On Tue, Nov 4, 2014 at 11:28 AM, Josh J joshjd...@gmail.com wrote: Hi, Does anyone have any good examples of using sortby for RDDs and scala? I'm receiving not enough arguments for method sortBy: (f: String = K, ascending: Boolean, numPartitions: Int)(implicit ord: Ordering[K], implicit ctag: scala.reflect.ClassTag[K])org.apache.spark.rdd.RDD[String]. Unspecified value parameter f. I tried to follow the example in the test case https://github.com/apache/spark/blob/83b7a1c6503adce1826fc537b4db47e534da5cae/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala by using the same approach even same method names and parameters though no luck. Thanks, Josh
Re: Spark Streaming compilation error: algebird not a member of package com.twitter
There is not artifact call spark-streaming-algebird . To use the algebird,
you will have add the following dependency (in maven format)
dependency
groupIdcom.twitter/groupId
artifactIdalgebird-core_${scala.binary.version}/artifactId
version0.1.11/version
/dependency
This is what is used in spark/examples/pom.xml
TD
On Sat, Sep 20, 2014 at 6:22 AM, SK skrishna...@gmail.com wrote:
Hi,
I am using the latest release Spark 1.1.0. I am trying to build the
streaming examples (under examples/streaming) as a standalone project with
the following streaming.sbt file. When I run sbt assembly, I get an error
stating that object algebird is not a member of package com.twitter. I
tried adding the dependency spark-streaming-algebird, but that was not
recognized. What dependency should I be including for algebird?
import AssemblyKeys._
assemblySettings
name := spark_stream_examples
version := 1.0
scalaVersion := 2.10.4
libraryDependencies += org.apache.spark %% spark-sql % 1.1.0 %
provided
libraryDependencies += org.apache.spark %% spark-streaming % 1.1.0 %
provided
libraryDependencies += org.apache.spark %% spark-streaming-twitter %
1.1.0
libraryDependencies += org.apache.spark %% spark-streaming-flume %
1.1.0
libraryDependencies += org.apache.spark %% spark-streaming-zeromq %
1.1.0
libraryDependencies += org.apache.spark %% spark-streaming-mqtt %
1.1.0
libraryDependencies += org.apache.spark %% spark-streaming-kafka %
1.1.0
//libraryDependencies += org.apache.spark %% spark-streaming-algebird %
1.1.0
resolvers += Akka Repository at http://repo.akka.io/releases/;
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-compilation-error-algebird-not-a-member-of-package-com-twitter-tp14709.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Spark Streaming compilation error: algebird not a member of package com.twitter
Hi, I am using the latest release Spark 1.1.0. I am trying to build the streaming examples (under examples/streaming) as a standalone project with the following streaming.sbt file. When I run sbt assembly, I get an error stating that object algebird is not a member of package com.twitter. I tried adding the dependency spark-streaming-algebird, but that was not recognized. What dependency should I be including for algebird? import AssemblyKeys._ assemblySettings name := spark_stream_examples version := 1.0 scalaVersion := 2.10.4 libraryDependencies += org.apache.spark %% spark-sql % 1.1.0 % provided libraryDependencies += org.apache.spark %% spark-streaming % 1.1.0 % provided libraryDependencies += org.apache.spark %% spark-streaming-twitter % 1.1.0 libraryDependencies += org.apache.spark %% spark-streaming-flume % 1.1.0 libraryDependencies += org.apache.spark %% spark-streaming-zeromq % 1.1.0 libraryDependencies += org.apache.spark %% spark-streaming-mqtt % 1.1.0 libraryDependencies += org.apache.spark %% spark-streaming-kafka % 1.1.0 //libraryDependencies += org.apache.spark %% spark-streaming-algebird % 1.1.0 resolvers += Akka Repository at http://repo.akka.io/releases/; -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-compilation-error-algebird-not-a-member-of-package-com-twitter-tp14709.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
wrote:
Hi Ted,Thank you so much!!As I am new to Spark, can you please advise the steps about how to apply this patch to my spark-1.0.2 source folder?
RegardsArthurOn 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
SeeSPARK-1297
The pull request is here:https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote:
(correction: "Compilation Error: Spark 1.0.2 with HBase 0.98” , please ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = "0.98.2"
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES "hbase;0.98.2"
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or should I set HBASE_VERSION back to “0.94.6"?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency: org.apache.hbase#hbase;0.98.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1170)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1168)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1191)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1189)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1193)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1188)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1196)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1161)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1139)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.r
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
I see 0.98.5 in dep.txt
You should be good to go.
On Thu, Aug 28, 2014 at 3:16 AM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi,
tried
mvn -Phbase-hadoop2,hadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests
dependency:tree dep.txt
Attached the dep. txt for your information.
Regards
Arthur
On 28 Aug, 2014, at 12:22 pm, Ted Yu yuzhih...@gmail.com wrote:
I forgot to include '-Dhadoop.version=2.4.1' in the command below.
The modified command passed.
You can verify the dependence on hbase 0.98 through this command:
mvn -Phbase-hadoop2,hadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests
dependency:tree dep.txt
Cheers
On Wed, Aug 27, 2014 at 8:58 PM, Ted Yu yuzhih...@gmail.com wrote:
Looks like the patch given by that URL only had the last commit.
I have attached pom.xml for spark-1.0.2 to SPARK-1297
You can download it and replace examples/pom.xml with the downloaded pom
I am running this command locally:
mvn -Phbase-hadoop2,hadoop-2.4,yarn -DskipTests clean package
Cheers
On Wed, Aug 27, 2014 at 7:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thanks.
Tried [patch -p1 -i 1893.patch](Hunk #1 FAILED at 45.)
Is this normal?
Regards
Arthur
patch -p1 -i 1893.patch
patching file examples/pom.xml
Hunk #1 FAILED at 45.
Hunk #2 succeeded at 94 (offset -16 lines).
1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file examples/pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 succeeded at 122 (offset -49 lines).
2 out of 3 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file docs/building-with-maven.md
patching file examples/pom.xml
Hunk #1 succeeded at 122 (offset -40 lines).
Hunk #2 succeeded at 195 (offset -40 lines).
On 28 Aug, 2014, at 10:53 am, Ted Yu yuzhih...@gmail.com wrote:
Can you use this command ?
patch -p1 -i 1893.patch
Cheers
On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk
FAILED, can you please advise how to get thru this error? or is my
spark-1.0.2 source not the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the
pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to
apply this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” ,
please ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2
with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Hi, I tried to start Spark but failed: $ ./sbin/start-all.sh starting org.apache.spark.deploy.master.Master, logging to /mnt/hadoop/spark-1.0.2/sbin/../logs/spark-edhuser-org.apache.spark.deploy.master.Master-1-m133.out failed to launch org.apache.spark.deploy.master.Master: Failed to find Spark assembly in /mnt/hadoop/spark-1.0.2/assembly/target/scala-2.10/ $ ll assembly/ total 20 -rw-rw-r--. 1 hduser hadoop 11795 Jul 26 05:50 pom.xml -rw-rw-r--. 1 hduser hadoop 507 Jul 26 05:50 README drwxrwxr-x. 4 hduser hadoop 4096 Jul 26 05:50 src Regards Arthur On 28 Aug, 2014, at 6:19 pm, Ted Yu yuzhih...@gmail.com wrote: I see 0.98.5 in dep.txt You should be good to go. On Thu, Aug 28, 2014 at 3:16 AM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: Hi, tried mvn -Phbase-hadoop2,hadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests dependency:tree dep.txt Attached the dep. txt for your information. Regards Arthur On 28 Aug, 2014, at 12:22 pm, Ted Yu yuzhih...@gmail.com wrote: I forgot to include '-Dhadoop.version=2.4.1' in the command below. The modified command passed. You can verify the dependence on hbase 0.98 through this command: mvn -Phbase-hadoop2,hadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests dependency:tree dep.txt Cheers On Wed, Aug 27, 2014 at 8:58 PM, Ted Yu yuzhih...@gmail.com wrote: Looks like the patch given by that URL only had the last commit. I have attached pom.xml for spark-1.0.2 to SPARK-1297 You can download it and replace examples/pom.xml with the downloaded pom I am running this command locally: mvn -Phbase-hadoop2,hadoop-2.4,yarn -DskipTests clean package Cheers On Wed, Aug 27, 2014 at 7:57 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: Hi Ted, Thanks. Tried [patch -p1 -i 1893.patch](Hunk #1 FAILED at 45.) Is this normal? Regards Arthur patch -p1 -i 1893.patch patching file examples/pom.xml Hunk #1 FAILED at 45. Hunk #2 succeeded at 94 (offset -16 lines). 1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej patching file examples/pom.xml Hunk #1 FAILED at 54. Hunk #2 FAILED at 72. Hunk #3 succeeded at 122 (offset -49 lines). 2 out of 3 hunks FAILED -- saving rejects to file examples/pom.xml.rej patching file docs/building-with-maven.md patching file examples/pom.xml Hunk #1 succeeded at 122 (offset -40 lines). Hunk #2 succeeded at 195 (offset -40 lines). On 28 Aug, 2014, at 10:53 am, Ted Yu yuzhih...@gmail.com wrote: Can you use this command ? patch -p1 -i 1893.patch Cheers On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: Hi Ted, I tried the following steps to apply the patch 1893 but got Hunk FAILED, can you please advise how to get thru this error? or is my spark-1.0.2 source not the correct one? Regards Arthur wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz tar -vxf spark-1.0.2.tgz cd spark-1.0.2 wget https://github.com/apache/spark/pull/1893.patch patch 1893.patch patching file pom.xml Hunk #1 FAILED at 45. Hunk #2 FAILED at 110. 2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej patching file pom.xml Hunk #1 FAILED at 54. Hunk #2 FAILED at 72. Hunk #3 FAILED at 171. 3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej can't find file to patch at input line 267 Perhaps you should have used the -p or --strip option? The text leading up to this was: -- | |From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001 |From: tedyu yuzhih...@gmail.com |Date: Mon, 11 Aug 2014 15:57:46 -0700 |Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add | description to building-with-maven.md | |--- | docs/building-with-maven.md | 3 +++ | 1 file changed, 3 insertions(+) | |diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md |index 672d0ef..f8bcd2b 100644 |--- a/docs/building-with-maven.md |+++ b/docs/building-with-maven.md -- File to patch: On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote: You can get the patch from this URL: https://github.com/apache/spark/pull/1893.patch BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the pom.xml Cheers On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: Hi Ted, Thank you so much!! As I am new to Spark, can you please advise the steps about how to apply this patch to my spark-1.0.2 source folder? Regards Arthur On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote: See SPARK-1297 The pull request is here: https://github.com/apache/spark/pull/1893 On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com arthur.hk.c...@gmail.com wrote: (correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please ignore if duplicated
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/
building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the
pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to
apply this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” ,
please ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2
with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency:
org.apache.hbase#hbase;0.98.2: not found
at
sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at
sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at
sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at
sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at
sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at
xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1170)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1168)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1191)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1189)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at
sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1193)
at
sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1188)
at
sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1196)
at
sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1161)
at
sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1139)
at
scala.Function1$$anonfun$compose$1
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with
HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency:
org.apache.hbase#hbase;0.98.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at
xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1170)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1168)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1191)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1189)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1193)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1188)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1196)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1161)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1139)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at
sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at
sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply this
patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with
HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or should I
set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency: org.apache.hbase#hbase;0.98.2:
not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1170)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1168)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1191)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1189)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1193)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1188)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1196)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1161)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1139)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at
sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply
this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with
HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency:
org.apache.hbase#hbase;0.98.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at
xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1170)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1168)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1191)
at
sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1189)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1193)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1188)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1196)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1161)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1139)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at
sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at
sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at
sbt.ConcurrentRestrictions$$anon$4$$anonfun$1
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk FAILED, can
you please advise how to get thru this error? or is my spark-1.0.2 source not
the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply this
patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with
HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or should
I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency: org.apache.hbase#hbase;0.98.2:
not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt$Module.withModule(Ivy.scala:116)
at sbt.IvyActions$.update(IvyActions.scala:125)
at
sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Can you use this command ?
patch -p1 -i 1893.patch
Cheers
On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk FAILED,
can you please advise how to get thru this error? or is my spark-1.0.2
source not the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the
pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply
this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2
with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency:
org.apache.hbase#hbase;0.98.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at
xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
at sbt.IvySbt.withIvy(Ivy.scala:101)
at sbt.IvySbt.withIvy(Ivy.scala:97)
at sbt.IvySbt
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Hi Ted,
Thanks.
Tried [patch -p1 -i 1893.patch](Hunk #1 FAILED at 45.)
Is this normal?
Regards
Arthur
patch -p1 -i 1893.patch
patching file examples/pom.xml
Hunk #1 FAILED at 45.
Hunk #2 succeeded at 94 (offset -16 lines).
1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file examples/pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 succeeded at 122 (offset -49 lines).
2 out of 3 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file docs/building-with-maven.md
patching file examples/pom.xml
Hunk #1 succeeded at 122 (offset -40 lines).
Hunk #2 succeeded at 195 (offset -40 lines).
On 28 Aug, 2014, at 10:53 am, Ted Yu yuzhih...@gmail.com wrote:
Can you use this command ?
patch -p1 -i 1893.patch
Cheers
On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk FAILED, can
you please advise how to get thru this error? or is my spark-1.0.2 source not
the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply
this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2 with
HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or should
I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency: org.apache.hbase#hbase;0.98.2:
not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
at sbt.IvySbt$$anon$3.call(Ivy.scala:60
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
Looks like the patch given by that URL only had the last commit.
I have attached pom.xml for spark-1.0.2 to SPARK-1297
You can download it and replace examples/pom.xml with the downloaded pom
I am running this command locally:
mvn -Phbase-hadoop2,hadoop-2.4,yarn -DskipTests clean package
Cheers
On Wed, Aug 27, 2014 at 7:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thanks.
Tried [patch -p1 -i 1893.patch](Hunk #1 FAILED at 45.)
Is this normal?
Regards
Arthur
patch -p1 -i 1893.patch
patching file examples/pom.xml
Hunk #1 FAILED at 45.
Hunk #2 succeeded at 94 (offset -16 lines).
1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file examples/pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 succeeded at 122 (offset -49 lines).
2 out of 3 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file docs/building-with-maven.md
patching file examples/pom.xml
Hunk #1 succeeded at 122 (offset -40 lines).
Hunk #2 succeeded at 195 (offset -40 lines).
On 28 Aug, 2014, at 10:53 am, Ted Yu yuzhih...@gmail.com wrote:
Can you use this command ?
patch -p1 -i 1893.patch
Cheers
On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk FAILED,
can you please advise how to get thru this error? or is my spark-1.0.2
source not the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the
pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to apply
this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” , please
ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2
with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase;0.98.2: not found
[warn] ::
sbt.ResolveException: unresolved dependency:
org.apache.hbase#hbase;0.98.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:217)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun
Re: Compilation Error: Spark 1.0.2 with HBase 0.98
I forgot to include '-Dhadoop.version=2.4.1' in the command below.
The modified command passed.
You can verify the dependence on hbase 0.98 through this command:
mvn -Phbase-hadoop2,hadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests
dependency:tree dep.txt
Cheers
On Wed, Aug 27, 2014 at 8:58 PM, Ted Yu yuzhih...@gmail.com wrote:
Looks like the patch given by that URL only had the last commit.
I have attached pom.xml for spark-1.0.2 to SPARK-1297
You can download it and replace examples/pom.xml with the downloaded pom
I am running this command locally:
mvn -Phbase-hadoop2,hadoop-2.4,yarn -DskipTests clean package
Cheers
On Wed, Aug 27, 2014 at 7:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thanks.
Tried [patch -p1 -i 1893.patch](Hunk #1 FAILED at 45.)
Is this normal?
Regards
Arthur
patch -p1 -i 1893.patch
patching file examples/pom.xml
Hunk #1 FAILED at 45.
Hunk #2 succeeded at 94 (offset -16 lines).
1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file examples/pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 succeeded at 122 (offset -49 lines).
2 out of 3 hunks FAILED -- saving rejects to file examples/pom.xml.rej
patching file docs/building-with-maven.md
patching file examples/pom.xml
Hunk #1 succeeded at 122 (offset -40 lines).
Hunk #2 succeeded at 195 (offset -40 lines).
On 28 Aug, 2014, at 10:53 am, Ted Yu yuzhih...@gmail.com wrote:
Can you use this command ?
patch -p1 -i 1893.patch
Cheers
On Wed, Aug 27, 2014 at 7:41 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
I tried the following steps to apply the patch 1893 but got Hunk FAILED,
can you please advise how to get thru this error? or is my spark-1.0.2
source not the correct one?
Regards
Arthur
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
wget https://github.com/apache/spark/pull/1893.patch
patch 1893.patch
patching file pom.xml
Hunk #1 FAILED at 45.
Hunk #2 FAILED at 110.
2 out of 2 hunks FAILED -- saving rejects to file pom.xml.rej
patching file pom.xml
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 72.
Hunk #3 FAILED at 171.
3 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
can't find file to patch at input line 267
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--
|
|From cd58437897bf02b644c2171404ccffae5d12a2be Mon Sep 17 00:00:00 2001
|From: tedyu yuzhih...@gmail.com
|Date: Mon, 11 Aug 2014 15:57:46 -0700
|Subject: [PATCH 3/4] SPARK-1297 Upgrade HBase dependency to 0.98 - add
| description to building-with-maven.md
|
|---
| docs/building-with-maven.md | 3 +++
| 1 file changed, 3 insertions(+)
|
|diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- a/docs/building-with-maven.md
|+++ b/docs/building-with-maven.md
--
File to patch:
On 28 Aug, 2014, at 10:24 am, Ted Yu yuzhih...@gmail.com wrote:
You can get the patch from this URL:
https://github.com/apache/spark/pull/1893.patch
BTW 0.98.5 has been released - you can specify 0.98.5-hadoop2 in the
pom.xml
Cheers
On Wed, Aug 27, 2014 at 7:18 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
Hi Ted,
Thank you so much!!
As I am new to Spark, can you please advise the steps about how to
apply this patch to my spark-1.0.2 source folder?
Regards
Arthur
On 28 Aug, 2014, at 10:13 am, Ted Yu yuzhih...@gmail.com wrote:
See SPARK-1297
The pull request is here:
https://github.com/apache/spark/pull/1893
On Wed, Aug 27, 2014 at 6:57 PM, arthur.hk.c...@gmail.com
arthur.hk.c...@gmail.com wrote:
(correction: Compilation Error: Spark 1.0.2 with HBase 0.98” ,
please ignore if duplicated)
Hi,
I need to use Spark with HBase 0.98 and tried to compile Spark 1.0.2
with HBase 0.98,
My steps:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz
tar -vxf spark-1.0.2.tgz
cd spark-1.0.2
edit project/SparkBuild.scala, set HBASE_VERSION
// HBase version; set as appropriate.
val HBASE_VERSION = 0.98.2
edit pom.xml with following values
hadoop.version2.4.1/hadoop.version
protobuf.version2.5.0/protobuf.version
yarn.version${hadoop.version}/yarn.version
hbase.version0.98.5/hbase.version
zookeeper.version3.4.6/zookeeper.version
hive.version0.13.1/hive.version
SPARK_HADOOP_VERSION=2.4.1 SPARK_YARN=true sbt/sbt clean assembly
but it fails because of UNRESOLVED DEPENDENCIES hbase;0.98.2
Can you please advise how to compile Spark 1.0.2 with HBase 0.98? or
should I set HBASE_VERSION back to “0.94.6?
Regards
Arthur
[warn] ::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::
[warn] :: org.apache.hbase#hbase
Compilation error in Spark 1.0.0
Hi everyone,
I am new to Spark and I'm having problems to make my code compile. I have
the feeling I might be misunderstanding the functions so I would be very
glad to get some insight in what could be wrong.
The problematic code is the following:
JavaRDDBody bodies = lines.map(l - {Body b = new Body(); b.parse(l);} );
JavaPairRDDPartition, IterableBody partitions =
bodies.mapToPair(b -
b.computePartitions(maxDistance)).groupByKey();
Partition and Body are defined inside the driver class. Body contains the
following definition:
protected IterableTuple2Partition, Body computePartitions (int
maxDistance)
The idea is to reproduce the following schema:
The first map results in: *body1, body2, ... *
The mapToPair should output several of these:* (partition_i, body1),
(partition_i, body2)...*
Which are gathered by key as follows: *(partition_i, (body1, body_n),
(partition_i', (body2, body_n') ...*
Thanks in advance.
Regards,
Silvina
Re: Compilation error in Spark 1.0.0
Right, the compile error is a casting issue telling me I cannot assign
a JavaPairRDDPartition,
Body to a JavaPairRDDObject, Object. It happens in the mapToPair()
method.
On 9 July 2014 19:52, Sean Owen so...@cloudera.com wrote:
You forgot the compile error!
On Wed, Jul 9, 2014 at 6:14 PM, Silvina Caíno Lores silvi.ca...@gmail.com
wrote:
Hi everyone,
I am new to Spark and I'm having problems to make my code compile. I
have the feeling I might be misunderstanding the functions so I would be
very glad to get some insight in what could be wrong.
The problematic code is the following:
JavaRDDBody bodies = lines.map(l - {Body b = new Body(); b.parse(l);}
);
JavaPairRDDPartition, IterableBody partitions =
bodies.mapToPair(b -
b.computePartitions(maxDistance)).groupByKey();
Partition and Body are defined inside the driver class. Body contains
the following definition:
protected IterableTuple2Partition, Body computePartitions (int
maxDistance)
The idea is to reproduce the following schema:
The first map results in: *body1, body2, ... *
The mapToPair should output several of these:* (partition_i, body1),
(partition_i, body2)...*
Which are gathered by key as follows: *(partition_i, (body1,
body_n), (partition_i', (body2, body_n') ...*
Thanks in advance.
Regards,
Silvina
