[
https://issues.apache.org/jira/browse/SPARK-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evan Oman updated SPARK-14693:
------------------------------
Description:
All,
I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks
and my `ssc.start()` command is hanging.
I am using the following function to make my Spark Streaming Context:
{code:borderStyle=solid}
def creatingFunc(sc: SparkContext): StreamingContext =
{
// Create a StreamingContext
val ssc = new StreamingContext(sc,
Seconds(batchIntervalSeconds))
// Creata a Kinesis stream
val kinesisStream = KinesisUtils.createStream(ssc,
kinesisAppName, kinesisStreamName,
kinesisEndpointUrl,
RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName,
InitialPositionInStream.LATEST,
Seconds(kinesisCheckpointIntervalSeconds),
StorageLevel.MEMORY_AND_DISK_SER_2,
config.awsAccessKeyId, config.awsSecretKey)
kinesisStream.print()
ssc.remember(Minutes(1))
ssc.checkpoint(checkpointDir)
ssc
}
{code}
However when I run the following to start the streaming context:
{code:borderStyle=solid}
// Stop any existing StreamingContext
val stopActiveContext = true
if (stopActiveContext) {
StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
}
// Get or create a streaming context.
val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc))
// This starts the streaming context in the background.
ssc.start()
{code}
The last bit, `ssc.start()`, hangs indefinitely without issuing any log
messages. I am running this on a freshly spun up cluster with no other
notebooks attached so there aren't any other streaming contexts running.
Any thoughts?
Additionally, here are the libraries I am using (from my build.sbt file):
{code:borderStyle=solid}
"org.apache.spark" % "spark-core_2.10" % "1.6.0"
"org.apache.spark" % "spark-sql_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
{code}
was:
All,
I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks
and my `ssc.start()` command is hanging.
I am using the following function to make my Spark Streaming Context:
{code:scala|borderStyle=solid}
def creatingFunc(sc: SparkContext): StreamingContext =
{
// Create a StreamingContext
val ssc = new StreamingContext(sc,
Seconds(batchIntervalSeconds))
// Creata a Kinesis stream
val kinesisStream = KinesisUtils.createStream(ssc,
kinesisAppName, kinesisStreamName,
kinesisEndpointUrl,
RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName,
InitialPositionInStream.LATEST,
Seconds(kinesisCheckpointIntervalSeconds),
StorageLevel.MEMORY_AND_DISK_SER_2,
config.awsAccessKeyId, config.awsSecretKey)
kinesisStream.print()
ssc.remember(Minutes(1))
ssc.checkpoint(checkpointDir)
ssc
}
{code}
However when I run the following to start the streaming context:
{code:scala|borderStyle=solid}
// Stop any existing StreamingContext
val stopActiveContext = true
if (stopActiveContext) {
StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
}
// Get or create a streaming context.
val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc))
// This starts the streaming context in the background.
ssc.start()
{code}
The last bit, `ssc.start()`, hangs indefinitely without issuing any log
messages. I am running this on a freshly spun up cluster with no other
notebooks attached so there aren't any other streaming contexts running.
Any thoughts?
Additionally, here are the libraries I am using (from my build.sbt file):
{code:borderStyle=solid}
"org.apache.spark" % "spark-core_2.10" % "1.6.0"
"org.apache.spark" % "spark-sql_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0"
"org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
{code}
> Spark Streaming Context Hangs on Start
> --------------------------------------
>
> Key: SPARK-14693
> URL: https://issues.apache.org/jira/browse/SPARK-14693
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.0, 1.6.1
> Environment: Databricks Cloud
> Reporter: Evan Oman
>
> All,
> I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks
> and my `ssc.start()` command is hanging.
> I am using the following function to make my Spark Streaming Context:
> {code:borderStyle=solid}
> def creatingFunc(sc: SparkContext): StreamingContext =
> {
> // Create a StreamingContext
> val ssc = new StreamingContext(sc,
> Seconds(batchIntervalSeconds))
> // Creata a Kinesis stream
> val kinesisStream = KinesisUtils.createStream(ssc,
> kinesisAppName, kinesisStreamName,
> kinesisEndpointUrl,
> RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName,
> InitialPositionInStream.LATEST,
> Seconds(kinesisCheckpointIntervalSeconds),
> StorageLevel.MEMORY_AND_DISK_SER_2,
> config.awsAccessKeyId, config.awsSecretKey)
> kinesisStream.print()
> ssc.remember(Minutes(1))
> ssc.checkpoint(checkpointDir)
> ssc
> }
> {code}
> However when I run the following to start the streaming context:
> {code:borderStyle=solid}
> // Stop any existing StreamingContext
> val stopActiveContext = true
> if (stopActiveContext) {
> StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
> }
> // Get or create a streaming context.
> val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc))
> // This starts the streaming context in the background.
> ssc.start()
> {code}
> The last bit, `ssc.start()`, hangs indefinitely without issuing any log
> messages. I am running this on a freshly spun up cluster with no other
> notebooks attached so there aren't any other streaming contexts running.
> Any thoughts?
> Additionally, here are the libraries I am using (from my build.sbt file):
> {code:borderStyle=solid}
> "org.apache.spark" % "spark-core_2.10" % "1.6.0"
> "org.apache.spark" % "spark-sql_2.10" % "1.6.0"
> "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0"
> "org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
