subject:"Issue with Spark shell and scalatest"

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

hm still struggling with those two above

scala> import org.apache.spark.internal.Logging
:57: error: object internal is not a member of package
org.apache.spark
 import org.apache.spark.internal.Logging
 ^
scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
:60: error: object KafkaCluster in package kafka cannot be
accessed in package org.apache.spark.streaming.kafka
 import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset


My jars are

spark-shell --master spark://50.140.197.217:7077 --jars
/home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar,/home/hduser/jars/spark-core_2.10-1.5.1-tests.jar,/home/hduser/jars/spark-core_2.10-1.5.1.jar,/home/hduser/jars/spark-streaming-kafka_2.10-1.5.1.jar



*jar tvf spark-core_2.10-1.5.1.jar|grep Logging*  1722 Wed Sep 23 23:34:40
BST 2015 org/apache/spark/scheduler/EventLoggingListener$$anonfun$8.class
  1584 Wed Sep 23 23:34:40 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$start$2.class
  1327 Wed Sep 23 23:34:40 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$6.class
  1528 Wed Sep 23 23:34:40 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$8$$anonfun$apply$1.class
  1664 Wed Sep 23 23:34:40 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$1.class
  1437 Wed Sep 23 23:34:40 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$4.class
  1114 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$5.class
  1687 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$logEvent$3.class
  1349 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$2.class
  1339 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$logEvent$2.class
  1347 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$3.class
  1651 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$start$1.class
   981 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$7.class
  1798 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$logEvent$1.class
 23625 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener.class
  1650 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$stop$2.class
 10011 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$.class
  1441 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$openEventLog$1.class
  1273 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$stop$1.class
  1137 Wed Sep 23 23:34:42 BST 2015
org/apache/spark/scheduler/EventLoggingListener$$anonfun$openEventLog$2.class
  2308 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/ErrorMonitor$$anonfun$receiveWithLogging$2.class
  2621 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$4.class
  2727 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$3.class
  2614 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$2.class
  1670 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$6.class
  5519 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1.class
  2156 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$7.class
  2611 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$1.class
  1284 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/ErrorMonitor$$anonfun$receiveWithLogging$2$$anonfun$applyOrElse$8.class
  2203 Wed Sep 23 23:34:44 BST 2015
org/apache/spark/rpc/akka/AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$receiveWithLogging$1$$anonfun$applyOrElse$5.class
  2213 Wed Sep 23 23:34:44 BST 2015 org/apache/spark/Logging.class
  1703 Wed Sep 23 23:34:46 BST 2015 org/apache/spark/Logging$.class
  6523 Wed Sep 23 23:34:46 BST 2015 org/apache/spark/Logging$class.class


And

jar tvf spark-streaming-kafka_2.10-1.5.1.jar|grep LeaderOffset
  2226 Wed Sep 23 23:47:50 BST 2015

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

For you, it should be spark-core_2.10-1.5.1.jar

Please replace version of Spark in my example with the version you use.

On Thu, Apr 21, 2016 at 1:23 PM, Mich Talebzadeh 
wrote:

> Hi Ted
>
> I cannot see spark-core_2.11-2.0.0-SNAPSHOT.jar  under
>
> https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/
>
> Sorry where are these artefacts please?
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 20:24, Ted Yu  wrote:
>
>> Plug in 1.5.1 for your jars:
>>
>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging
>> ...
>>   1781 Thu Apr 21 08:19:34 PDT 2016
>> org/apache/spark/internal/Logging$.class
>>
>> jar tvf
>> external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep
>> LeaderOffset
>> ...
>>   3310 Thu Apr 21 08:38:40 PDT 2016
>> org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class
>>
>> On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> These two are giving me griefs:
>>>
>>> scala> import org.apache.spark.internal.Logging
>>> :26: error: object internal is not a member of package
>>> org.apache.spark
>>>  import org.apache.spark.internal.Logging
>>>
>>> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>>> :29: error: object KafkaCluster in package kafka cannot be
>>> accessed in package org.apache.spark.streaming.kafka
>>>  import
>>> org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 18:29, Mich Talebzadeh 
>>> wrote:
>>>

 Thanks

 jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
   1787 Wed Sep 23 23:34:26 BST 2015
 org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
   1780 Wed Sep 23 23:34:26 BST 2015
 org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
   3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class


 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 18:21, Ted Yu  wrote:

> Please replace version number for the release you are using :
>
> spark-core_2.10-1.5.1-tests.jar
>
> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> I don't seem to be able to locate
>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 17:45, Ted Yu  wrote:
>>
>>> It is in core-XX-tests jar:
>>>
>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar |
>>> grep SparkFunSuite
>>>   1830 Thu Apr 21 08:19:14 PDT 2016
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>>   1823 Thu Apr 21 08:19:14 PDT 2016
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>>   6232 Thu Apr 21 08:19:14 PDT 2016
>>> org/apache/spark/SparkFunSuite.class
>>>
>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 like war of attrition :)

 now I get with sbt

 object SparkFunSuite is not a member of package org.apache.spark


 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 17:22, Ted Yu  wrote:

> Have you tried the following ?
>
> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>
> On Thu, Apr 21, 2016 at

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

Hi Ted

I cannot see spark-core_2.11-2.0.0-SNAPSHOT.jar  under

https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/

Sorry where are these artefacts please?



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 20:24, Ted Yu  wrote:

> Plug in 1.5.1 for your jars:
>
> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging
> ...
>   1781 Thu Apr 21 08:19:34 PDT 2016
> org/apache/spark/internal/Logging$.class
>
> jar tvf
> external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep
> LeaderOffset
> ...
>   3310 Thu Apr 21 08:38:40 PDT 2016
> org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class
>
> On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> These two are giving me griefs:
>>
>> scala> import org.apache.spark.internal.Logging
>> :26: error: object internal is not a member of package
>> org.apache.spark
>>  import org.apache.spark.internal.Logging
>>
>> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>> :29: error: object KafkaCluster in package kafka cannot be
>> accessed in package org.apache.spark.streaming.kafka
>>  import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 18:29, Mich Talebzadeh 
>> wrote:
>>
>>>
>>> Thanks
>>>
>>> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
>>>   1787 Wed Sep 23 23:34:26 BST 2015
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>>   1780 Wed Sep 23 23:34:26 BST 2015
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>>   3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 18:21, Ted Yu  wrote:
>>>
 Please replace version number for the release you are using :

 spark-core_2.10-1.5.1-tests.jar

 On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> I don't seem to be able to locate
> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 17:45, Ted Yu  wrote:
>
>> It is in core-XX-tests jar:
>>
>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar |
>> grep SparkFunSuite
>>   1830 Thu Apr 21 08:19:14 PDT 2016
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>   1823 Thu Apr 21 08:19:14 PDT 2016
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>   6232 Thu Apr 21 08:19:14 PDT 2016
>> org/apache/spark/SparkFunSuite.class
>>
>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> like war of attrition :)
>>>
>>> now I get with sbt
>>>
>>> object SparkFunSuite is not a member of package org.apache.spark
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 17:22, Ted Yu  wrote:
>>>
 Have you tried the following ?

 libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"

 On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Unfortunately this sbt dependency is not working
>
> libraryDependencies += "org.apache.spark" %% "spark-core" %
> "1.5.1" % "provided"
> libraryDependencies += "org.apache.spark" %% "spark-sql" %
> "1.5.1"  % "provided"
> libraryDependencies +=

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

Plug in 1.5.1 for your jars:

$ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging
...
  1781 Thu Apr 21 08:19:34 PDT 2016 org/apache/spark/internal/Logging$.class

jar tvf external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar
| grep LeaderOffset
...
  3310 Thu Apr 21 08:38:40 PDT 2016
org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class

On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh  wrote:

> These two are giving me griefs:
>
> scala> import org.apache.spark.internal.Logging
> :26: error: object internal is not a member of package
> org.apache.spark
>  import org.apache.spark.internal.Logging
>
> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
> :29: error: object KafkaCluster in package kafka cannot be
> accessed in package org.apache.spark.streaming.kafka
>  import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 18:29, Mich Talebzadeh 
> wrote:
>
>>
>> Thanks
>>
>> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
>>   1787 Wed Sep 23 23:34:26 BST 2015
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>   1780 Wed Sep 23 23:34:26 BST 2015
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>   3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 18:21, Ted Yu  wrote:
>>
>>> Please replace version number for the release you are using :
>>>
>>> spark-core_2.10-1.5.1-tests.jar
>>>
>>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 I don't seem to be able to locate
 spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(

 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 17:45, Ted Yu  wrote:

> It is in core-XX-tests jar:
>
> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar |
> grep SparkFunSuite
>   1830 Thu Apr 21 08:19:14 PDT 2016
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>   1823 Thu Apr 21 08:19:14 PDT 2016
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>   6232 Thu Apr 21 08:19:14 PDT 2016
> org/apache/spark/SparkFunSuite.class
>
> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> like war of attrition :)
>>
>> now I get with sbt
>>
>> object SparkFunSuite is not a member of package org.apache.spark
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 17:22, Ted Yu  wrote:
>>
>>> Have you tried the following ?
>>>
>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>>>
>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Unfortunately this sbt dependency is not working

 libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
 % "provided"
 libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
 % "provided"
 libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
 % "provided"
 libraryDependencies += "junit" % "junit" % "4.12"
 libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
 libraryDependencies += "org.apache.spark" %% "spark-streaming" %
 "1.6.1" % "provided"
 libraryDependencies += "org.apache.spark" %%
 "spark-streaming-kafka" % "1.6.1"
 libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
 *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
 "test"*

 Getting error

 [info] Compiling 1 Scala source to

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

These two are giving me griefs:

scala> import org.apache.spark.internal.Logging
:26: error: object internal is not a member of package
org.apache.spark
 import org.apache.spark.internal.Logging

scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset
:29: error: object KafkaCluster in package kafka cannot be
accessed in package org.apache.spark.streaming.kafka
 import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 18:29, Mich Talebzadeh 
wrote:

>
> Thanks
>
> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
>   1787 Wed Sep 23 23:34:26 BST 2015
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>   1780 Wed Sep 23 23:34:26 BST 2015
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>   3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 18:21, Ted Yu  wrote:
>
>> Please replace version number for the release you are using :
>>
>> spark-core_2.10-1.5.1-tests.jar
>>
>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> I don't seem to be able to locate
>>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 17:45, Ted Yu  wrote:
>>>
 It is in core-XX-tests jar:

 $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | grep
 SparkFunSuite
   1830 Thu Apr 21 08:19:14 PDT 2016
 org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
   1823 Thu Apr 21 08:19:14 PDT 2016
 org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
   6232 Thu Apr 21 08:19:14 PDT 2016 org/apache/spark/SparkFunSuite.class

 On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> like war of attrition :)
>
> now I get with sbt
>
> object SparkFunSuite is not a member of package org.apache.spark
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 17:22, Ted Yu  wrote:
>
>> Have you tried the following ?
>>
>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>>
>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Unfortunately this sbt dependency is not working
>>>
>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
>>> % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
>>> % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
>>> % "provided"
>>> libraryDependencies += "junit" % "junit" % "4.12"
>>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>>> "1.6.1" % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka"
>>> % "1.6.1"
>>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
>>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
>>> "test"*
>>>
>>> Getting error
>>>
>>> [info] Compiling 1 Scala source to
>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
>>> [error]
>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
>>> object scalatest is not a member of package org
>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> [error]^
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

Thanks

jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite
  1787 Wed Sep 23 23:34:26 BST 2015
org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
  1780 Wed Sep 23 23:34:26 BST 2015
org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
  3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 18:21, Ted Yu  wrote:

> Please replace version number for the release you are using :
>
> spark-core_2.10-1.5.1-tests.jar
>
> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> I don't seem to be able to locate
>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :(
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 17:45, Ted Yu  wrote:
>>
>>> It is in core-XX-tests jar:
>>>
>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | grep
>>> SparkFunSuite
>>>   1830 Thu Apr 21 08:19:14 PDT 2016
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>>   1823 Thu Apr 21 08:19:14 PDT 2016
>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>>   6232 Thu Apr 21 08:19:14 PDT 2016 org/apache/spark/SparkFunSuite.class
>>>
>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 like war of attrition :)

 now I get with sbt

 object SparkFunSuite is not a member of package org.apache.spark


 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 17:22, Ted Yu  wrote:

> Have you tried the following ?
>
> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>
> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Unfortunately this sbt dependency is not working
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
>> "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
>> "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
>> "provided"
>> libraryDependencies += "junit" % "junit" % "4.12"
>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>> "1.6.1" % "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka"
>> % "1.6.1"
>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
>> "test"*
>>
>> Getting error
>>
>> [info] Compiling 1 Scala source to
>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
>> [error]
>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
>> object scalatest is not a member of package org
>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> [error]^
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 16:49, Mich Talebzadeh > > wrote:
>>
>>> Thanks Ted. It was a typo in my alias and it is sorted now
>>>
>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077
>>> --jars
>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>>>
>>>
>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

Please replace version number for the release you are using :

spark-core_2.10-1.5.1-tests.jar

On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh  wrote:

> I don't seem to be able to locate spark-core_2.11-2.0.0-SNAPSHOT-tests.jar
> file :(
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 17:45, Ted Yu  wrote:
>
>> It is in core-XX-tests jar:
>>
>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | grep
>> SparkFunSuite
>>   1830 Thu Apr 21 08:19:14 PDT 2016
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>>   1823 Thu Apr 21 08:19:14 PDT 2016
>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>>   6232 Thu Apr 21 08:19:14 PDT 2016 org/apache/spark/SparkFunSuite.class
>>
>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> like war of attrition :)
>>>
>>> now I get with sbt
>>>
>>> object SparkFunSuite is not a member of package org.apache.spark
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 17:22, Ted Yu  wrote:
>>>
 Have you tried the following ?

 libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"

 On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Unfortunately this sbt dependency is not working
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
> "provided"
> libraryDependencies += "junit" % "junit" % "4.12"
> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
> "1.6.1" % "provided"
> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
> "1.6.1"
> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
> "test"*
>
> Getting error
>
> [info] Compiling 1 Scala source to
> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
> [error]
> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
> object scalatest is not a member of package org
> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> [error]^
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 16:49, Mich Talebzadeh 
> wrote:
>
>> Thanks Ted. It was a typo in my alias and it is sorted now
>>
>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077
>> --jars
>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>>
>>
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 16:44, Ted Yu  wrote:
>>
>>> I tried on refreshed copy of master branch:
>>>
>>> $ bin/spark-shell --jars
>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>> ...
>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>
>>> BTW I noticed an extra leading comma after '--jars' in your email.
>>> Not sure if that matters.
>>>
>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>>>
 Mich:

 $ jar tvf

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

I don't seem to be able to locate spark-core_2.11-2.0.0-SNAPSHOT-tests.jar
file :(

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 17:45, Ted Yu  wrote:

> It is in core-XX-tests jar:
>
> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | grep
> SparkFunSuite
>   1830 Thu Apr 21 08:19:14 PDT 2016
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
>   1823 Thu Apr 21 08:19:14 PDT 2016
> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
>   6232 Thu Apr 21 08:19:14 PDT 2016 org/apache/spark/SparkFunSuite.class
>
> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> like war of attrition :)
>>
>> now I get with sbt
>>
>> object SparkFunSuite is not a member of package org.apache.spark
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 17:22, Ted Yu  wrote:
>>
>>> Have you tried the following ?
>>>
>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>>>
>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Unfortunately this sbt dependency is not working

 libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
 "provided"
 libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
 "provided"
 libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
 "provided"
 libraryDependencies += "junit" % "junit" % "4.12"
 libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
 libraryDependencies += "org.apache.spark" %% "spark-streaming" %
 "1.6.1" % "provided"
 libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
 "1.6.1"
 libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
 *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
 "test"*

 Getting error

 [info] Compiling 1 Scala source to
 /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
 [error]
 /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
 object scalatest is not a member of package org
 [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
 [error]^



 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 16:49, Mich Talebzadeh 
 wrote:

> Thanks Ted. It was a typo in my alias and it is sorted now
>
> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>
>
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 16:44, Ted Yu  wrote:
>
>> I tried on refreshed copy of master branch:
>>
>> $ bin/spark-shell --jars
>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>> ...
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>> BTW I noticed an extra leading comma after '--jars' in your email.
>> Not sure if that matters.
>>
>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>>
>>> Mich:
>>>
>>> $ jar tvf
>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>> | grep BeforeAndAfter
>>>   4257 Sat Dec 26 14:35:48 PST 2015
>>> org/scalatest/BeforeAndAfter$class.class
>>>   2602 Sat Dec 26 14:35:48 PST 2015
>>> org/scalatest/BeforeAndAfter.class
>>>   1998 Sat Dec 26 14:35:48 PST 2015
>>>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

It is in core-XX-tests jar:

$ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | grep
SparkFunSuite
  1830 Thu Apr 21 08:19:14 PDT 2016
org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class
  1823 Thu Apr 21 08:19:14 PDT 2016
org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class
  6232 Thu Apr 21 08:19:14 PDT 2016 org/apache/spark/SparkFunSuite.class

On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh 
wrote:

> like war of attrition :)
>
> now I get with sbt
>
> object SparkFunSuite is not a member of package org.apache.spark
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 17:22, Ted Yu  wrote:
>
>> Have you tried the following ?
>>
>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>>
>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Unfortunately this sbt dependency is not working
>>>
>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
>>> "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
>>> "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
>>> "provided"
>>> libraryDependencies += "junit" % "junit" % "4.12"
>>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1"
>>> % "provided"
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
>>> "1.6.1"
>>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
>>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" %
>>> "test"*
>>>
>>> Getting error
>>>
>>> [info] Compiling 1 Scala source to
>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
>>> [error]
>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
>>> object scalatest is not a member of package org
>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> [error]^
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 16:49, Mich Talebzadeh 
>>> wrote:
>>>
 Thanks Ted. It was a typo in my alias and it is sorted now

 slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
 /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'


 scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
 import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}



 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



 On 21 April 2016 at 16:44, Ted Yu  wrote:

> I tried on refreshed copy of master branch:
>
> $ bin/spark-shell --jars
> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
> ...
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
> BTW I noticed an extra leading comma after '--jars' in your email.
> Not sure if that matters.
>
> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>
>> Mich:
>>
>> $ jar tvf
>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>> | grep BeforeAndAfter
>>   4257 Sat Dec 26 14:35:48 PST 2015
>> org/scalatest/BeforeAndAfter$class.class
>>   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
>>   1998 Sat Dec 26 14:35:48 PST 2015
>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>>
>> FYI
>>
>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> to Mario, Alonso, Luciano, user
>>> Hi,
>>>
>>> Following example in
>>>
>>>
>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>>>
>>> Does anyone know which jar file this belongs to?
>>>
>>> I use *scalatest_2.11-2.2.6.jar *in my

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

like war of attrition :)

now I get with sbt

object SparkFunSuite is not a member of package org.apache.spark


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 17:22, Ted Yu  wrote:

> Have you tried the following ?
>
> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"
>
> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Unfortunately this sbt dependency is not working
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
>> "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
>> "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
>> "provided"
>> libraryDependencies += "junit" % "junit" % "4.12"
>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1"
>> % "provided"
>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
>> "1.6.1"
>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"*
>>
>> Getting error
>>
>> [info] Compiling 1 Scala source to
>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
>> [error]
>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
>> object scalatest is not a member of package org
>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> [error]^
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 16:49, Mich Talebzadeh 
>> wrote:
>>
>>> Thanks Ted. It was a typo in my alias and it is sorted now
>>>
>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>>>
>>>
>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 April 2016 at 16:44, Ted Yu  wrote:
>>>
 I tried on refreshed copy of master branch:

 $ bin/spark-shell --jars
 /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
 ...
 scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
 import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}

 BTW I noticed an extra leading comma after '--jars' in your email.
 Not sure if that matters.

 On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:

> Mich:
>
> $ jar tvf
> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
> | grep BeforeAndAfter
>   4257 Sat Dec 26 14:35:48 PST 2015
> org/scalatest/BeforeAndAfter$class.class
>   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
>   1998 Sat Dec 26 14:35:48 PST 2015
> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>
> FYI
>
> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> to Mario, Alonso, Luciano, user
>> Hi,
>>
>> Following example in
>>
>>
>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>>
>> Does anyone know which jar file this belongs to?
>>
>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell
>>
>>  spark-shell --master spark://50.140.197.217:7077 --jars
>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>>
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> :28: error: object scalatest is not a member of package org
>>  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

Have you tried the following ?

libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6"

On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh 
wrote:

> Unfortunately this sbt dependency is not working
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
> "provided"
> libraryDependencies += "junit" % "junit" % "4.12"
> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
> "1.6.1"
> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"*
>
> Getting error
>
> [info] Compiling 1 Scala source to
> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
> [error]
> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
> object scalatest is not a member of package org
> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> [error]^
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 16:49, Mich Talebzadeh 
> wrote:
>
>> Thanks Ted. It was a typo in my alias and it is sorted now
>>
>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>>
>>
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 April 2016 at 16:44, Ted Yu  wrote:
>>
>>> I tried on refreshed copy of master branch:
>>>
>>> $ bin/spark-shell --jars
>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>> ...
>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>
>>> BTW I noticed an extra leading comma after '--jars' in your email.
>>> Not sure if that matters.
>>>
>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>>>
 Mich:

 $ jar tvf
 /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
 | grep BeforeAndAfter
   4257 Sat Dec 26 14:35:48 PST 2015
 org/scalatest/BeforeAndAfter$class.class
   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
   1998 Sat Dec 26 14:35:48 PST 2015
 org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class

 FYI

 On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> to Mario, Alonso, Luciano, user
> Hi,
>
> Following example in
>
>
> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>
> Does anyone know which jar file this belongs to?
>
> I use *scalatest_2.11-2.2.6.jar *in my spark-shell
>
>  spark-shell --master spark://50.140.197.217:7077 --jars
> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> :28: error: object scalatest is not a member of package org
>  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>


>>>
>>
>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

Unfortunately this sbt dependency is not working

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"  %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
"provided"
libraryDependencies += "junit" % "junit" % "4.12"
libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
"1.6.1"
libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
*libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"*

Getting error

[info] Compiling 1 Scala source to
/data6/hduser/scala/CEP_assembly/target/scala-2.10/classes...
[error]
/data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32:
object scalatest is not a member of package org
[error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
[error]^



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 16:49, Mich Talebzadeh 
wrote:

> Thanks Ted. It was a typo in my alias and it is sorted now
>
> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'
>
>
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 April 2016 at 16:44, Ted Yu  wrote:
>
>> I tried on refreshed copy of master branch:
>>
>> $ bin/spark-shell --jars
>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>> ...
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>> BTW I noticed an extra leading comma after '--jars' in your email.
>> Not sure if that matters.
>>
>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>>
>>> Mich:
>>>
>>> $ jar tvf
>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>>> | grep BeforeAndAfter
>>>   4257 Sat Dec 26 14:35:48 PST 2015
>>> org/scalatest/BeforeAndAfter$class.class
>>>   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
>>>   1998 Sat Dec 26 14:35:48 PST 2015
>>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>>>
>>> FYI
>>>
>>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 to Mario, Alonso, Luciano, user
 Hi,

 Following example in


 https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532

 Does anyone know which jar file this belongs to?

 I use *scalatest_2.11-2.2.6.jar *in my spark-shell

  spark-shell --master spark://50.140.197.217:7077 --jars
 ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
 /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*

 scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
 :28: error: object scalatest is not a member of package org
  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}

 Thanks

 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com



>>>
>>>
>>
>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

Thanks Ted. It was a typo in my alias and it is sorted now

slong='rlwrap spark-shell --master spark://50.140.197.217:7077 --jars
/home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar'


scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 April 2016 at 16:44, Ted Yu  wrote:

> I tried on refreshed copy of master branch:
>
> $ bin/spark-shell --jars
> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
> ...
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
> BTW I noticed an extra leading comma after '--jars' in your email.
> Not sure if that matters.
>
> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:
>
>> Mich:
>>
>> $ jar tvf
>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
>> | grep BeforeAndAfter
>>   4257 Sat Dec 26 14:35:48 PST 2015
>> org/scalatest/BeforeAndAfter$class.class
>>   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
>>   1998 Sat Dec 26 14:35:48 PST 2015
>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>>
>> FYI
>>
>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> to Mario, Alonso, Luciano, user
>>> Hi,
>>>
>>> Following example in
>>>
>>>
>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>>>
>>> Does anyone know which jar file this belongs to?
>>>
>>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell
>>>
>>>  spark-shell --master spark://50.140.197.217:7077 --jars
>>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
>>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>>>
>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>> :28: error: object scalatest is not a member of package org
>>>  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>>
>>> Thanks
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>
>>
>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

I tried on refreshed copy of master branch:

$ bin/spark-shell --jars
/home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
...
scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}

BTW I noticed an extra leading comma after '--jars' in your email.
Not sure if that matters.

On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu  wrote:

> Mich:
>
> $ jar tvf
> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
> | grep BeforeAndAfter
>   4257 Sat Dec 26 14:35:48 PST 2015
> org/scalatest/BeforeAndAfter$class.class
>   2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
>   1998 Sat Dec 26 14:35:48 PST 2015
> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class
>
> FYI
>
> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> to Mario, Alonso, Luciano, user
>> Hi,
>>
>> Following example in
>>
>>
>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>>
>> Does anyone know which jar file this belongs to?
>>
>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell
>>
>>  spark-shell --master spark://50.140.197.217:7077 --jars
>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>>
>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>> :28: error: object scalatest is not a member of package org
>>  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>

Re: Issue with Spark shell and scalatest

2016-04-21 Thread Ted Yu

Mich:

$ jar tvf
/home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar
| grep BeforeAndAfter
  4257 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter$class.class
  2602 Sat Dec 26 14:35:48 PST 2015 org/scalatest/BeforeAndAfter.class
  1998 Sat Dec 26 14:35:48 PST 2015
org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class

FYI

On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh 
wrote:

> to Mario, Alonso, Luciano, user
> Hi,
>
> Following example in
>
>
> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532
>
> Does anyone know which jar file this belongs to?
>
> I use *scalatest_2.11-2.2.6.jar *in my spark-shell
>
>  spark-shell --master spark://50.140.197.217:7077 --jars
> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'*
>
> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
> :28: error: object scalatest is not a member of package org
>  import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>

Issue with Spark shell and scalatest

2016-04-21 Thread Mich Talebzadeh

to Mario, Alonso, Luciano, user
Hi,

Following example in

https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532

Does anyone know which jar file this belongs to?

I use *scalatest_2.11-2.2.6.jar *in my spark-shell

 spark-shell --master spark://50.140.197.217:7077 --jars
,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,
/*home/hduser/jars/scalatest_2.11-2.2.6.jar'*

scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
:28: error: object scalatest is not a member of package org
 import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}

Thanks

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Re: Issue with Spark shell and scalatest

Issue with Spark shell and scalatest

16 matches

Site Navigation

Mail list logo

Footer information