I'm submitting this via 'dse spark-submit' but somehow, I don't see any loggings in my cluster or worker machines...
How can I find out ? My cluster is running DSE 4.6.1 with Spark enabled. My source is running Kafka 0.8.2.0 I'm launching my program on one of my DSE machines. Any insights much appreciated. Thanks. --------- cas1.dev% dse spark-submit --verbose --deploy-mode cluster --master spark://cas1.dev.kno.com:7077 --class com.kno.highlights.counting.service.HighlightConsumer --driver-class-path /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib --driver-library-path /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib --properties-file /tmp/highlights-counting.properties /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar --name HighlightConsumer Using properties file: /tmp/highlights-counting.properties Warning: Ignoring non-spark config property: checkpoint_directory=checkpointForHighlights Warning: Ignoring non-spark config property: zookeeper_port=2181 Warning: Ignoring non-spark config property: default_num_of_cores_per_topic=1 Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 Warning: Ignoring non-spark config property: kafka_consumer_group=highlight_consumer_group Warning: Ignoring non-spark config property: app_name=HighlightConsumer Warning: Ignoring non-spark config property: cassandra_keyspace=bookevents Warning: Ignoring non-spark config property: scheduler_mode=FIFO Warning: Ignoring non-spark config property: highlight_topic=highlight_topic Warning: Ignoring non-spark config property: cassandra_host=cas1.dev.kno.com Warning: Ignoring non-spark config property: checkpoint_interval=3 Warning: Ignoring non-spark config property: zookeeper_host=cas1.dev.kno.com Adding default property: spark_master=spark://cas1.dev.kno.com:7077 Warning: Ignoring non-spark config property: streaming_window=10 Using properties file: /tmp/highlights-counting.properties Warning: Ignoring non-spark config property: checkpoint_directory=checkpointForHighlights Warning: Ignoring non-spark config property: zookeeper_port=2181 Warning: Ignoring non-spark config property: default_num_of_cores_per_topic=1 Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 Warning: Ignoring non-spark config property: kafka_consumer_group=highlight_consumer_group Warning: Ignoring non-spark config property: app_name=HighlightConsumer Warning: Ignoring non-spark config property: cassandra_keyspace=bookevents Warning: Ignoring non-spark config property: scheduler_mode=FIFO Warning: Ignoring non-spark config property: highlight_topic=highlight_topic Warning: Ignoring non-spark config property: cassandra_host=cas1.dev.kno.com Warning: Ignoring non-spark config property: checkpoint_interval=3 Warning: Ignoring non-spark config property: zookeeper_host=cas1.dev.kno.com Adding default property: spark_master=spark://cas1.dev.kno.com:7077 Warning: Ignoring non-spark config property: streaming_window=10 Parsed arguments: master spark://cas1.dev.kno.com:7077 deployMode cluster executorMemory null executorCores null totalExecutorCores null propertiesFile /tmp/highlights-counting.properties extraSparkProperties Map() driverMemory null driverCores null driverExtraClassPath /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib driverExtraLibraryPath /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib driverExtraJavaOptions null supervise false queue null numExecutors null files null pyFiles null archives null mainClass com.kno.highlights.counting.service.HighlightConsumer primaryResource file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar name com.kno.highlights.counting.service.HighlightConsumer childArgs [--name HighlightConsumer] jars null verbose true Default properties from /tmp/highlights-counting.properties: spark_master -> spark://cas1.dev.kno.com:7077 Using properties file: /tmp/highlights-counting.properties Warning: Ignoring non-spark config property: checkpoint_directory=checkpointForHighlights Warning: Ignoring non-spark config property: zookeeper_port=2181 Warning: Ignoring non-spark config property: default_num_of_cores_per_topic=1 Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 Warning: Ignoring non-spark config property: kafka_consumer_group=highlight_consumer_group Warning: Ignoring non-spark config property: app_name=HighlightConsumer Warning: Ignoring non-spark config property: cassandra_keyspace=bookevents Warning: Ignoring non-spark config property: scheduler_mode=FIFO Warning: Ignoring non-spark config property: highlight_topic=highlight_topic Warning: Ignoring non-spark config property: cassandra_host=cas1.dev.kno.com Warning: Ignoring non-spark config property: checkpoint_interval=3 Warning: Ignoring non-spark config property: zookeeper_host=cas1.dev.kno.com Adding default property: spark_master=spark://cas1.dev.kno.com:7077 Warning: Ignoring non-spark config property: streaming_window=10 Main class: org.apache.spark.deploy.Client Arguments: launch spark://cas1.dev.kno.com:7077 file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar com.kno.highlights.counting.service.HighlightConsumer --name HighlightConsumer System properties: spark_master -> spark://cas1.dev.kno.com:7077 spark.driver.extraLibraryPath -> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib SPARK_SUBMIT -> true spark.app.name -> com.kno.highlights.counting.service.HighlightConsumer spark.jars -> file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar spark.master -> spark://cas1.dev.kno.com:7077 spark.driver.extraClassPath -> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib Classpath elements: Sending launch command to spark://cas1.dev.kno.com:7077 ------------ Here's my build.sbt: --------- import AssemblyKeys._ import NativePackagerHelper._ assemblySettings jarName in assembly := "kno-highlights-counting-service.jar" name := "kno-highlights-counting-service" version := "0.1" scalaVersion := "2.10.4" exportJars := true enablePlugins(JavaServerAppPackaging) resolvers ++= Seq( "spray repo" at "http://repo.spray.io/", "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/" ) packageOptions in (Compile, packageBin) += Package.ManifestAttributes( java.util.jar.Attributes.Name.SEALED -> "true" ) libraryDependencies ++= { Seq( // "org.apache.kafka" % "kafka_2.10" % "0.8.0" exclude("org.jboss.netty", "netty"), "org.apache.spark" % "spark-core_2.10" % "1.1.0" exclude("org.jboss.netty", "netty"), "org.apache.spark" % "spark-streaming_2.10" % "1.1.0" exclude("org.jboss.netty", "netty"), "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.1.0" exclude("org.jboss.netty", "netty"), "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0" exclude("org.jboss.netty", "netty"), "commons-io" % "commons-io" % "2.4", "org.apache.commons" % "commons-pool2" % "2.3", "ch.qos.logback" % "logback-classic" % "1.1.2", "io.spray" %% "spray-json" % "1.3.1", "com.typesafe" % "config" % "1.2.1" ) } seq(Revolver.settings: _*) atmosSettings ------- -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/No-logs-from-my-cluster-worker-running-DSE-4-6-1-tp22759.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org