Re: How to access the off-heap representation of cached data in Spark 2.0

[Jacek Laskowski]

Hi Jim,

There's no C++ API in Spark to access the off-heap data. Moreover, I
also think "off-heap" has an overloaded meaning in Spark - for
tungsten and to persist your data off-heap (it's all about memory but
for different purposes and with client- and internal API).

That's my limited understanding of the things (and I'm not even sure
how trustworthy it is). Use with extreme caution.

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sat, May 28, 2016 at 5:29 PM, jpivar...@gmail.com
 wrote:
> Is this not the place to ask such questions? Where can I get a hint as to how
> to access the new off-heap cache, or C++ API, if it exists? I'm willing to
> do my own research, but I have to have a place to start. (In fact, this is
> the first step in that research.)
>
> Thanks,
> -- Jim
>
>
>
>
> --
> View this message in context: 
> http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-access-the-off-heap-representation-of-cached-data-in-Spark-2-0-tp17701p17717.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Spark Streaming - Twitter on Python current status

[Ricardo Almeida]

As far as I could understand...
1. Using Python (PySpark), the use of Twitter Streaming (TwitterUtils
)
as well as Customer Receivers is restricted to Scala and Java APIs on Spark
1.6.1;
2. Maven linking Twitter/spark-streaming-twitter_2.10 is being removed from
Spark Streaming core Scala/Java API (Maven Linking

)
3. There are no plans to support Twitter Streaming Python API (pyspark) on
Spark 2.0 or later
4. Twitter Streaming API usage over Python (Spark pyspark) is and will
continue being restricted to Twitter REST API


Could you please confirm if these 4 my assumption are correct or is there
any alternative way for using Twitter Streaming API on PySpark from Spark
2.0?

Re: How to access the off-heap representation of cached data in Spark 2.0

[jpivar...@gmail.com]

Is this not the place to ask such questions? Where can I get a hint as to how
to access the new off-heap cache, or C++ API, if it exists? I'm willing to
do my own research, but I have to have a place to start. (In fact, this is
the first step in that research.)

Thanks,
-- Jim




--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-access-the-off-heap-representation-of-cached-data-in-Spark-2-0-tp17701p17717.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org