You could try the following: create a minimal project using sbt or Maven,
add spark-streaming-twitter as a dependency, run sbt assembly (or mvn
package) on that to create a fat jar (with Spark as provided dependency),
and add that to the shell classpath when starting up.


On Tue, Jul 15, 2014 at 9:06 AM, Praveen Seluka <psel...@qubole.com> wrote:

> If you want to make Twitter* classes available in your shell, I believe
> you could do the following
> 1. Change the parent pom module ordering - Move external/twitter before
> assembly
> 2. In assembly/pom.xm, add external/twitter dependency - this will package
> twitter* into the assembly jar
>
> Now when spark-shell is launched, assembly jar is in classpath - hence
> twitter* too. I think this will work (remember trying this sometime back)
>
>
> On Tue, Jul 15, 2014 at 11:59 AM, Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> Hmm, I'd like to clarify something from your comments, Tathagata.
>>
>> Going forward, is Twitter Streaming functionality not supported from the
>> shell? What should users do if they'd like to process live Tweets from the
>> shell?
>>
>> Nick
>>
>>
>> On Mon, Jul 14, 2014 at 11:50 PM, Nicholas Chammas <
>> nicholas.cham...@gmail.com> wrote:
>>
>>> At some point, you were able to access TwitterUtils from spark shell
>>>> using Spark 1.0.0+ ?
>>>
>>>
>>> Yep.
>>>
>>>
>>>> If yes, then what change in Spark caused it to not work any more?
>>>
>>>
>>> It still works for me. I was just commenting on your remark that it
>>> doesn't work through the shell, which I now understand to apply to versions
>>> of Spark before 1.0.0.
>>>
>>>  Nick
>>>
>>
>>
>

Reply via email to