Ramchandhar Rapolu created SPARK-19907: ------------------------------------------
Summary: Spark Submit Does not pick up the HBase Jars Key: SPARK-19907 URL: https://issues.apache.org/jira/browse/SPARK-19907 Project: Spark Issue Type: Task Components: DStreams, Spark Submit, YARN Affects Versions: 2.0.0 Environment: Linux, cloudera-jdk-1.7 Reporter: Ramchandhar Rapolu Priority: Blocker Using properties file: /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/conf/spark-defaults.conf Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer Adding default property: spark.yarn.jars=local:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/jars/*:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/hbase/jars/* Adding default property: spark.eventLog.enabled=true Adding default property: spark.hadoop.mapreduce.application.classpath= Adding default property: spark.shuffle.service.enabled=true Adding default property: spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native Adding default property: spark.yarn.historyServer.address=http://instance-26765.bigstep.io:18089 Adding default property: spark.ui.killEnabled=true Adding default property: spark.sql.hive.metastore.jars=${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/* Adding default property: spark.dynamicAllocation.schedulerBacklogTimeout=1 Adding default property: spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native Adding default property: spark.yarn.config.gatewayPath=/opt/cloudera/parcels Adding default property: spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../.. Adding default property: spark.submit.deployMode=client Adding default property: spark.shuffle.service.port=7337 Adding default property: spark.master=yarn Adding default property: spark.authenticate=false Adding default property: spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native Adding default property: spark.eventLog.dir=hdfs://instance-26765.bigstep.io:8020/user/spark/spark2ApplicationHistory Adding default property: spark.dynamicAllocation.enabled=true Adding default property: spark.sql.catalogImplementation=hive Adding default property: spark.hadoop.yarn.application.classpath= Adding default property: spark.dynamicAllocation.minExecutors=0 Adding default property: spark.dynamicAllocation.executorIdleTimeout=60 Adding default property: spark.sql.hive.metastore.version=1.1.0 Parsed arguments: master yarn deployMode client executorMemory 2g executorCores 1 totalExecutorCores null propertiesFile /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/conf/spark-defaults.conf driverMemory 4g driverCores null driverExtraClassPath null driverExtraLibraryPath /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native driverExtraJavaOptions null supervise false queue null numExecutors null files null pyFiles null archives null mainClass com.golfbreaks.spark.streaming.Test primaryResource file:/opt/golfbreaks/spark-jars/streaming-1.0-jar-with-dependencies.jar name com.golfbreaks.spark.streaming.Test childArgs [] jars file:/opt/golfbreaks/spark-jars/./lib/activation-1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/antlr4-runtime-4.5.3.jar,file:/opt/golfbreaks/spark-jars/./lib/aopalliance-repackaged-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/apacheds-i18n-2.0.0-M15.jar,file:/opt/golfbreaks/spark-jars/./lib/apacheds-kerberos-codec-2.0.0-M15.jar,file:/opt/golfbreaks/spark-jars/./lib/api-asn1-api-1.0.0-M20.jar,file:/opt/golfbreaks/spark-jars/./lib/api-util-1.0.0-M20.jar,file:/opt/golfbreaks/spark-jars/./lib/asm-3.1.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-1.7.6-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-mapred-1.7.6-cdh5.7.0-hadoop2.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-core-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-kms-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-s3-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/chill_2.11-0.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/chill-java-0.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-1.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-core-1.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-cli-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-codec-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-collections-3.2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-compiler-2.7.8.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-compress-1.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-configuration-1.6.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-crypto-1.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-daemon-1.0.13.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-digester-1.8.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-el-1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-httpclient-3.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-io-2.4.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-lang-2.6.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-lang3-3.3.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-logging-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-math-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-math3-3.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-net-2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/compress-lzf-1.0.3.jar,file:/opt/golfbreaks/spark-jars/./lib/core-3.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-client-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-framework-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-recipes-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/disruptor-3.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/fastutil-6.3.jar,file:/opt/golfbreaks/spark-jars/./lib/findbugs-annotations-1.3.9-1.jar,file:/opt/golfbreaks/spark-jars/./lib/gson-2.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/guava-12.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-annotations-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-auth-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-aws-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-client-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-common-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-core-2.6.0-mr1-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.9.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-api-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-client-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-server-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hamcrest-core-1.3.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-annotations-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-client-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop2-compat-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop-compat-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-prefix-tree-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-procedure-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-protocol-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-server-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-spark-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/high-scale-lib-1.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-api-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-locator-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-utils-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hsqldb-1.8.0.10.jar,file:/opt/golfbreaks/spark-jars/./lib/htrace-core-3.2.0-incubating.jar,file:/opt/golfbreaks/spark-jars/./lib/htrace-core4-4.0.1-incubating.jar,file:/opt/golfbreaks/spark-jars/./lib/httpclient-4.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/httpcore-4.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/ivy-2.4.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-annotations-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-core-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-core-asl-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-databind-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-jaxrs-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-mapper-asl-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-module-paranamer-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-module-scala_2.11-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-xc-1.8.3.jar,file:/opt/golfbreaks/spark-jars/./lib/jamon-runtime-2.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/janino-2.7.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jasper-compiler-5.5.23.jar,file:/opt/golfbreaks/spark-jars/./lib/jasper-runtime-5.5.23.jar,file:/opt/golfbreaks/spark-jars/./lib/javassist-3.18.1-GA.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.annotation-api-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.inject-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/java-xmlbuilder-0.4.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.servlet-api-3.1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.ws.rs-api-2.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jaxb-api-2.2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jaxb-impl-2.2.3-1.jar,file:/opt/golfbreaks/spark-jars/./lib/jcl-over-slf4j-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jcodings-1.0.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-client-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-common-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-core-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-core-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-guava-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-json-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-media-jaxb-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-server-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-server-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jets3t-0.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jettison-1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-sslengine-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-util-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/joni-2.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jsch-0.1.42.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-ast_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-core_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-jackson_2.11-3.2.11.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-native_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-scalap_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-2.1-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jsr305-1.3.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jul-to-slf4j-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/junit-4.12.jar,file:/opt/golfbreaks/spark-jars/./lib/kafka_2.11-0.9.0-kafka-2.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/kafka-clients-0.9.0-kafka-2.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/kryo-shaded-3.0.3.jar,file:/opt/golfbreaks/spark-jars/./lib/leveldbjni-all-1.8.jar,file:/opt/golfbreaks/spark-jars/./lib/log4j-1.2.17.jar,file:/opt/golfbreaks/spark-jars/./lib/lz4-1.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/mesos-0.21.1-shaded-protobuf.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-core-2.2.0.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-core-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-graphite-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-json-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-jvm-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/minlog-1.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/netty-3.8.0.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/netty-all-4.0.29.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/objenesis-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/oro-2.0.8.jar,file:/opt/golfbreaks/spark-jars/./lib/osgi-resource-locator-1.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/paranamer-2.8.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-avro-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-column-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-common-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-encoding-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-format-2.1.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-hadoop-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-jackson-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/protobuf-java-2.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/py4j-0.10.3.jar,file:/opt/golfbreaks/spark-jars/./lib/pyrolite-4.13.jar,file:/opt/golfbreaks/spark-jars/./lib/RoaringBitmap-0.5.11.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-library-2.11.8.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-parser-combinators_2.11-1.0.4.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-reflect-2.11.7.jar,file:/opt/golfbreaks/spark-jars/./lib/scalatest_2.11-2.2.6.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-xml_2.11-1.0.6.jar,file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5.jar,file:/opt/golfbreaks/spark-jars/./lib/slf4j-api-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/slf4j-log4j12-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/snappy-java-1.0.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-catalyst_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-core_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-launcher_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-network-common_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-network-shuffle_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-sketch_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-sql_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-streaming_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-streaming-kafka-0-8_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-tags_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-unsafe_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/stax-api-1.0-2.jar,file:/opt/golfbreaks/spark-jars/./lib/stream-2.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/univocity-parsers-2.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/unused-1.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/validation-api-1.1.0.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/xbean-asm5-shaded-4.4.jar,file:/opt/golfbreaks/spark-jars/./lib/xercesImpl-2.9.1.jar,file:/opt/golfbreaks/spark-jars/./lib/xml-apis-1.3.04.jar,file:/opt/golfbreaks/spark-jars/./lib/xmlenc-0.52.jar,file:/opt/golfbreaks/spark-jars/./lib/xz-1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/zkclient-0.7.jar,file:/opt/golfbreaks/spark-jars/./lib/zookeeper-3.4.5-cdh5.9.0.jar packages null packagesExclusions null repositories null verbose true Spark properties used, including those specified through --conf and those from the properties file /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/conf/spark-defaults.conf: spark.executor.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.driver.memory -> 4g spark.authenticate -> false spark.yarn.jars -> local:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/jars/*:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/hbase/jars/* spark.driver.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.yarn.historyServer.address -> http://instance-26765.bigstep.io:18089 spark.yarn.am.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.eventLog.enabled -> true spark.dynamicAllocation.schedulerBacklogTimeout -> 1 spark.yarn.config.gatewayPath -> /opt/cloudera/parcels spark.ui.killEnabled -> true spark.serializer -> org.apache.spark.serializer.KryoSerializer spark.shuffle.service.enabled -> true spark.hadoop.yarn.application.classpath -> spark.dynamicAllocation.minExecutors -> 0 spark.dynamicAllocation.executorIdleTimeout -> 60 spark.yarn.config.replacementPath -> {{HADOOP_COMMON_HOME}}/../../.. spark.sql.hive.metastore.version -> 1.1.0 spark.submit.deployMode -> client spark.shuffle.service.port -> 7337 spark.hadoop.mapreduce.application.classpath -> spark.eventLog.dir -> hdfs://instance-26765.bigstep.io:8020/user/spark/spark2ApplicationHistory spark.master -> yarn spark.dynamicAllocation.enabled -> true spark.sql.catalogImplementation -> hive spark.sql.hive.metastore.jars -> ${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/* Main class: com.golfbreaks.spark.streaming.Test Arguments: System properties: spark.executor.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.yarn.dist.jars -> file:/opt/golfbreaks/spark-jars/./lib/activation-1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/antlr4-runtime-4.5.3.jar,file:/opt/golfbreaks/spark-jars/./lib/aopalliance-repackaged-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/apacheds-i18n-2.0.0-M15.jar,file:/opt/golfbreaks/spark-jars/./lib/apacheds-kerberos-codec-2.0.0-M15.jar,file:/opt/golfbreaks/spark-jars/./lib/api-asn1-api-1.0.0-M20.jar,file:/opt/golfbreaks/spark-jars/./lib/api-util-1.0.0-M20.jar,file:/opt/golfbreaks/spark-jars/./lib/asm-3.1.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-1.7.6-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/avro-mapred-1.7.6-cdh5.7.0-hadoop2.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-core-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-kms-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-s3-1.10.6.jar,file:/opt/golfbreaks/spark-jars/./lib/chill_2.11-0.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/chill-java-0.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-1.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-core-1.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-cli-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-codec-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-collections-3.2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-compiler-2.7.8.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-compress-1.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-configuration-1.6.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-crypto-1.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-daemon-1.0.13.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-digester-1.8.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-el-1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-httpclient-3.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-io-2.4.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-lang-2.6.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-lang3-3.3.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-logging-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-math-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-math3-3.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/commons-net-2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/compress-lzf-1.0.3.jar,file:/opt/golfbreaks/spark-jars/./lib/core-3.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-client-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-framework-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/curator-recipes-2.7.1.jar,file:/opt/golfbreaks/spark-jars/./lib/disruptor-3.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/fastutil-6.3.jar,file:/opt/golfbreaks/spark-jars/./lib/findbugs-annotations-1.3.9-1.jar,file:/opt/golfbreaks/spark-jars/./lib/gson-2.8.0.jar,file:/opt/golfbreaks/spark-jars/./lib/guava-12.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-annotations-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-auth-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-aws-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-client-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-common-2.6.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-core-2.6.0-mr1-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.9.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-api-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-client-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-server-common-2.6.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hamcrest-core-1.3.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-annotations-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-client-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0-tests.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop2-compat-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop-compat-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-prefix-tree-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-procedure-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-protocol-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-server-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/hbase-spark-1.2.0-cdh5.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/high-scale-lib-1.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-api-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-locator-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hk2-utils-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/hsqldb-1.8.0.10.jar,file:/opt/golfbreaks/spark-jars/./lib/htrace-core-3.2.0-incubating.jar,file:/opt/golfbreaks/spark-jars/./lib/htrace-core4-4.0.1-incubating.jar,file:/opt/golfbreaks/spark-jars/./lib/httpclient-4.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/httpcore-4.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/ivy-2.4.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-annotations-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-core-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-core-asl-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-databind-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-jaxrs-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-mapper-asl-1.8.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-module-paranamer-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-module-scala_2.11-2.6.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jackson-xc-1.8.3.jar,file:/opt/golfbreaks/spark-jars/./lib/jamon-runtime-2.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/janino-2.7.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jasper-compiler-5.5.23.jar,file:/opt/golfbreaks/spark-jars/./lib/jasper-runtime-5.5.23.jar,file:/opt/golfbreaks/spark-jars/./lib/javassist-3.18.1-GA.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.annotation-api-1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.inject-2.4.0-b34.jar,file:/opt/golfbreaks/spark-jars/./lib/java-xmlbuilder-0.4.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.servlet-api-3.1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/javax.ws.rs-api-2.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jaxb-api-2.2.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jaxb-impl-2.2.3-1.jar,file:/opt/golfbreaks/spark-jars/./lib/jcl-over-slf4j-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/jcodings-1.0.8.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-client-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-common-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-core-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-core-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-guava-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-json-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-media-jaxb-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-server-1.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jersey-server-2.22.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jets3t-0.9.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jettison-1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-sslengine-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/jetty-util-6.1.26.cloudera.4.jar,file:/opt/golfbreaks/spark-jars/./lib/joni-2.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/jsch-0.1.42.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-ast_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-core_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-jackson_2.11-3.2.11.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-native_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/json4s-scalap_2.11-3.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-2.1-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/jsr305-1.3.9.jar,file:/opt/golfbreaks/spark-jars/./lib/jul-to-slf4j-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/junit-4.12.jar,file:/opt/golfbreaks/spark-jars/./lib/kafka_2.11-0.9.0-kafka-2.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/kafka-clients-0.9.0-kafka-2.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/kryo-shaded-3.0.3.jar,file:/opt/golfbreaks/spark-jars/./lib/leveldbjni-all-1.8.jar,file:/opt/golfbreaks/spark-jars/./lib/log4j-1.2.17.jar,file:/opt/golfbreaks/spark-jars/./lib/lz4-1.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/mesos-0.21.1-shaded-protobuf.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-core-2.2.0.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-core-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-graphite-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-json-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/metrics-jvm-3.1.2.jar,file:/opt/golfbreaks/spark-jars/./lib/minlog-1.3.0.jar,file:/opt/golfbreaks/spark-jars/./lib/netty-3.8.0.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/netty-all-4.0.29.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/objenesis-2.1.jar,file:/opt/golfbreaks/spark-jars/./lib/oro-2.0.8.jar,file:/opt/golfbreaks/spark-jars/./lib/osgi-resource-locator-1.0.1.jar,file:/opt/golfbreaks/spark-jars/./lib/paranamer-2.8.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-avro-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-column-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-common-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-encoding-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-format-2.1.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-hadoop-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/parquet-jackson-1.5.0-cdh5.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/protobuf-java-2.5.0.jar,file:/opt/golfbreaks/spark-jars/./lib/py4j-0.10.3.jar,file:/opt/golfbreaks/spark-jars/./lib/pyrolite-4.13.jar,file:/opt/golfbreaks/spark-jars/./lib/RoaringBitmap-0.5.11.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-library-2.11.8.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-parser-combinators_2.11-1.0.4.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-reflect-2.11.7.jar,file:/opt/golfbreaks/spark-jars/./lib/scalatest_2.11-2.2.6.jar,file:/opt/golfbreaks/spark-jars/./lib/scala-xml_2.11-1.0.6.jar,file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5-6.1.14.jar,file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5.jar,file:/opt/golfbreaks/spark-jars/./lib/slf4j-api-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/slf4j-log4j12-1.7.5.jar,file:/opt/golfbreaks/spark-jars/./lib/snappy-java-1.0.4.1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-catalyst_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-core_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-launcher_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-network-common_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-network-shuffle_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-sketch_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-sql_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-streaming_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-streaming-kafka-0-8_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-tags_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/spark-unsafe_2.11-2.0.0.cloudera1.jar,file:/opt/golfbreaks/spark-jars/./lib/stax-api-1.0-2.jar,file:/opt/golfbreaks/spark-jars/./lib/stream-2.7.0.jar,file:/opt/golfbreaks/spark-jars/./lib/univocity-parsers-2.1.1.jar,file:/opt/golfbreaks/spark-jars/./lib/unused-1.0.0.jar,file:/opt/golfbreaks/spark-jars/./lib/validation-api-1.1.0.Final.jar,file:/opt/golfbreaks/spark-jars/./lib/xbean-asm5-shaded-4.4.jar,file:/opt/golfbreaks/spark-jars/./lib/xercesImpl-2.9.1.jar,file:/opt/golfbreaks/spark-jars/./lib/xml-apis-1.3.04.jar,file:/opt/golfbreaks/spark-jars/./lib/xmlenc-0.52.jar,file:/opt/golfbreaks/spark-jars/./lib/xz-1.0.jar,file:/opt/golfbreaks/spark-jars/./lib/zkclient-0.7.jar,file:/opt/golfbreaks/spark-jars/./lib/zookeeper-3.4.5-cdh5.9.0.jar spark.driver.memory -> 4g spark.executor.memory -> 2g spark.authenticate -> false spark.yarn.jars -> local:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/jars/*:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/hbase/jars/* spark.driver.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.yarn.historyServer.address -> http://instance-26765.bigstep.io:18089 spark.yarn.am.extraLibraryPath -> /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/lib/native spark.eventLog.enabled -> true spark.dynamicAllocation.schedulerBacklogTimeout -> 1 SPARK_SUBMIT -> true spark.yarn.config.gatewayPath -> /opt/cloudera/parcels spark.ui.killEnabled -> true spark.serializer -> org.apache.spark.serializer.KryoSerializer spark.app.name -> com.golfbreaks.spark.streaming.Test spark.shuffle.service.enabled -> true spark.hadoop.yarn.application.classpath -> spark.dynamicAllocation.minExecutors -> 0 spark.dynamicAllocation.executorIdleTimeout -> 60 spark.yarn.config.replacementPath -> {{HADOOP_COMMON_HOME}}/../../.. spark.sql.hive.metastore.version -> 1.1.0 spark.jars -> file:/opt/golfbreaks/spark-jars/streaming-1.0-jar-with-dependencies.jar spark.submit.deployMode -> client spark.shuffle.service.port -> 7337 spark.hadoop.mapreduce.application.classpath -> spark.eventLog.dir -> hdfs://instance-26765.bigstep.io:8020/user/spark/spark2ApplicationHistory spark.master -> yarn spark.dynamicAllocation.enabled -> true spark.sql.catalogImplementation -> hive spark.executor.cores -> 1 spark.sql.hive.metastore.jars -> ${env:HADOOP_COMMON_HOME}/../hive/lib/*:${env:HADOOP_COMMON_HOME}/client/* Classpath elements: file:/opt/golfbreaks/spark-jars/streaming-1.0-jar-with-dependencies.jar file:/opt/golfbreaks/spark-jars/./lib/activation-1.1.jar file:/opt/golfbreaks/spark-jars/./lib/antlr4-runtime-4.5.3.jar file:/opt/golfbreaks/spark-jars/./lib/aopalliance-repackaged-2.4.0-b34.jar file:/opt/golfbreaks/spark-jars/./lib/apacheds-i18n-2.0.0-M15.jar file:/opt/golfbreaks/spark-jars/./lib/apacheds-kerberos-codec-2.0.0-M15.jar file:/opt/golfbreaks/spark-jars/./lib/api-asn1-api-1.0.0-M20.jar file:/opt/golfbreaks/spark-jars/./lib/api-util-1.0.0-M20.jar file:/opt/golfbreaks/spark-jars/./lib/asm-3.1.jar file:/opt/golfbreaks/spark-jars/./lib/avro-1.7.6-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/avro-ipc-1.7.6-cdh5.7.0-tests.jar file:/opt/golfbreaks/spark-jars/./lib/avro-mapred-1.7.6-cdh5.7.0-hadoop2.jar file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-core-1.10.6.jar file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-kms-1.10.6.jar file:/opt/golfbreaks/spark-jars/./lib/aws-java-sdk-s3-1.10.6.jar file:/opt/golfbreaks/spark-jars/./lib/chill_2.11-0.8.0.jar file:/opt/golfbreaks/spark-jars/./lib/chill-java-0.8.0.jar file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-1.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/commons-beanutils-core-1.8.0.jar file:/opt/golfbreaks/spark-jars/./lib/commons-cli-1.2.jar file:/opt/golfbreaks/spark-jars/./lib/commons-codec-1.9.jar file:/opt/golfbreaks/spark-jars/./lib/commons-collections-3.2.2.jar file:/opt/golfbreaks/spark-jars/./lib/commons-compiler-2.7.8.jar file:/opt/golfbreaks/spark-jars/./lib/commons-compress-1.4.1.jar file:/opt/golfbreaks/spark-jars/./lib/commons-configuration-1.6.jar file:/opt/golfbreaks/spark-jars/./lib/commons-crypto-1.0.0.jar file:/opt/golfbreaks/spark-jars/./lib/commons-daemon-1.0.13.jar file:/opt/golfbreaks/spark-jars/./lib/commons-digester-1.8.jar file:/opt/golfbreaks/spark-jars/./lib/commons-el-1.0.jar file:/opt/golfbreaks/spark-jars/./lib/commons-httpclient-3.1.jar file:/opt/golfbreaks/spark-jars/./lib/commons-io-2.4.jar file:/opt/golfbreaks/spark-jars/./lib/commons-lang-2.6.jar file:/opt/golfbreaks/spark-jars/./lib/commons-lang3-3.3.2.jar file:/opt/golfbreaks/spark-jars/./lib/commons-logging-1.2.jar file:/opt/golfbreaks/spark-jars/./lib/commons-math-2.1.jar file:/opt/golfbreaks/spark-jars/./lib/commons-math3-3.4.1.jar file:/opt/golfbreaks/spark-jars/./lib/commons-net-2.2.jar file:/opt/golfbreaks/spark-jars/./lib/compress-lzf-1.0.3.jar file:/opt/golfbreaks/spark-jars/./lib/core-3.1.1.jar file:/opt/golfbreaks/spark-jars/./lib/curator-client-2.7.1.jar file:/opt/golfbreaks/spark-jars/./lib/curator-framework-2.7.1.jar file:/opt/golfbreaks/spark-jars/./lib/curator-recipes-2.7.1.jar file:/opt/golfbreaks/spark-jars/./lib/disruptor-3.3.0.jar file:/opt/golfbreaks/spark-jars/./lib/fastutil-6.3.jar file:/opt/golfbreaks/spark-jars/./lib/findbugs-annotations-1.3.9-1.jar file:/opt/golfbreaks/spark-jars/./lib/gson-2.8.0.jar file:/opt/golfbreaks/spark-jars/./lib/guava-12.0.1.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-annotations-2.6.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-auth-2.6.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-aws-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-client-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-common-2.6.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-core-2.6.0-mr1-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-hdfs-2.6.0-cdh5.9.0-tests.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-api-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-client-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-common-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hadoop-yarn-server-common-2.6.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/hamcrest-core-1.3.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-annotations-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-client-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-common-1.2.0-cdh5.9.0-tests.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop2-compat-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-hadoop-compat-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-prefix-tree-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-procedure-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-protocol-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-server-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/hbase-spark-1.2.0-cdh5.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/high-scale-lib-1.1.1.jar file:/opt/golfbreaks/spark-jars/./lib/hk2-api-2.4.0-b34.jar file:/opt/golfbreaks/spark-jars/./lib/hk2-locator-2.4.0-b34.jar file:/opt/golfbreaks/spark-jars/./lib/hk2-utils-2.4.0-b34.jar file:/opt/golfbreaks/spark-jars/./lib/hsqldb-1.8.0.10.jar file:/opt/golfbreaks/spark-jars/./lib/htrace-core-3.2.0-incubating.jar file:/opt/golfbreaks/spark-jars/./lib/htrace-core4-4.0.1-incubating.jar file:/opt/golfbreaks/spark-jars/./lib/httpclient-4.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/httpcore-4.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/ivy-2.4.0.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-annotations-2.6.5.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-core-2.6.5.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-core-asl-1.8.8.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-databind-2.6.5.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-jaxrs-1.8.8.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-mapper-asl-1.8.8.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-module-paranamer-2.6.5.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-module-scala_2.11-2.6.5.jar file:/opt/golfbreaks/spark-jars/./lib/jackson-xc-1.8.3.jar file:/opt/golfbreaks/spark-jars/./lib/jamon-runtime-2.4.1.jar file:/opt/golfbreaks/spark-jars/./lib/janino-2.7.8.jar file:/opt/golfbreaks/spark-jars/./lib/jasper-compiler-5.5.23.jar file:/opt/golfbreaks/spark-jars/./lib/jasper-runtime-5.5.23.jar file:/opt/golfbreaks/spark-jars/./lib/javassist-3.18.1-GA.jar file:/opt/golfbreaks/spark-jars/./lib/javax.annotation-api-1.2.jar file:/opt/golfbreaks/spark-jars/./lib/javax.inject-2.4.0-b34.jar file:/opt/golfbreaks/spark-jars/./lib/java-xmlbuilder-0.4.jar file:/opt/golfbreaks/spark-jars/./lib/javax.servlet-api-3.1.0.jar file:/opt/golfbreaks/spark-jars/./lib/javax.ws.rs-api-2.0.1.jar file:/opt/golfbreaks/spark-jars/./lib/jaxb-api-2.2.2.jar file:/opt/golfbreaks/spark-jars/./lib/jaxb-impl-2.2.3-1.jar file:/opt/golfbreaks/spark-jars/./lib/jcl-over-slf4j-1.7.5.jar file:/opt/golfbreaks/spark-jars/./lib/jcodings-1.0.8.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-client-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-common-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-container-servlet-core-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-core-1.9.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-guava-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-json-1.9.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-media-jaxb-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-server-1.9.jar file:/opt/golfbreaks/spark-jars/./lib/jersey-server-2.22.2.jar file:/opt/golfbreaks/spark-jars/./lib/jets3t-0.9.0.jar file:/opt/golfbreaks/spark-jars/./lib/jettison-1.1.jar file:/opt/golfbreaks/spark-jars/./lib/jetty-6.1.26.cloudera.4.jar file:/opt/golfbreaks/spark-jars/./lib/jetty-sslengine-6.1.26.cloudera.4.jar file:/opt/golfbreaks/spark-jars/./lib/jetty-util-6.1.26.cloudera.4.jar file:/opt/golfbreaks/spark-jars/./lib/joni-2.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/jsch-0.1.42.jar file:/opt/golfbreaks/spark-jars/./lib/json4s-ast_2.11-3.5.0.jar file:/opt/golfbreaks/spark-jars/./lib/json4s-core_2.11-3.5.0.jar file:/opt/golfbreaks/spark-jars/./lib/json4s-jackson_2.11-3.2.11.jar file:/opt/golfbreaks/spark-jars/./lib/json4s-native_2.11-3.5.0.jar file:/opt/golfbreaks/spark-jars/./lib/json4s-scalap_2.11-3.5.0.jar file:/opt/golfbreaks/spark-jars/./lib/jsp-2.1-6.1.14.jar file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1-6.1.14.jar file:/opt/golfbreaks/spark-jars/./lib/jsp-api-2.1.jar file:/opt/golfbreaks/spark-jars/./lib/jsr305-1.3.9.jar file:/opt/golfbreaks/spark-jars/./lib/jul-to-slf4j-1.7.5.jar file:/opt/golfbreaks/spark-jars/./lib/junit-4.12.jar file:/opt/golfbreaks/spark-jars/./lib/kafka_2.11-0.9.0-kafka-2.0.0.jar file:/opt/golfbreaks/spark-jars/./lib/kafka-clients-0.9.0-kafka-2.0.0.jar file:/opt/golfbreaks/spark-jars/./lib/kryo-shaded-3.0.3.jar file:/opt/golfbreaks/spark-jars/./lib/leveldbjni-all-1.8.jar file:/opt/golfbreaks/spark-jars/./lib/log4j-1.2.17.jar file:/opt/golfbreaks/spark-jars/./lib/lz4-1.3.0.jar file:/opt/golfbreaks/spark-jars/./lib/mesos-0.21.1-shaded-protobuf.jar file:/opt/golfbreaks/spark-jars/./lib/metrics-core-2.2.0.jar file:/opt/golfbreaks/spark-jars/./lib/metrics-core-3.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/metrics-graphite-3.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/metrics-json-3.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/metrics-jvm-3.1.2.jar file:/opt/golfbreaks/spark-jars/./lib/minlog-1.3.0.jar file:/opt/golfbreaks/spark-jars/./lib/netty-3.8.0.Final.jar file:/opt/golfbreaks/spark-jars/./lib/netty-all-4.0.29.Final.jar file:/opt/golfbreaks/spark-jars/./lib/objenesis-2.1.jar file:/opt/golfbreaks/spark-jars/./lib/oro-2.0.8.jar file:/opt/golfbreaks/spark-jars/./lib/osgi-resource-locator-1.0.1.jar file:/opt/golfbreaks/spark-jars/./lib/paranamer-2.8.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-avro-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-column-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-common-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-encoding-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-format-2.1.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-hadoop-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/parquet-jackson-1.5.0-cdh5.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/protobuf-java-2.5.0.jar file:/opt/golfbreaks/spark-jars/./lib/py4j-0.10.3.jar file:/opt/golfbreaks/spark-jars/./lib/pyrolite-4.13.jar file:/opt/golfbreaks/spark-jars/./lib/RoaringBitmap-0.5.11.jar file:/opt/golfbreaks/spark-jars/./lib/scala-library-2.11.8.jar file:/opt/golfbreaks/spark-jars/./lib/scala-parser-combinators_2.11-1.0.4.jar file:/opt/golfbreaks/spark-jars/./lib/scala-reflect-2.11.7.jar file:/opt/golfbreaks/spark-jars/./lib/scalatest_2.11-2.2.6.jar file:/opt/golfbreaks/spark-jars/./lib/scala-xml_2.11-1.0.6.jar file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5-6.1.14.jar file:/opt/golfbreaks/spark-jars/./lib/servlet-api-2.5.jar file:/opt/golfbreaks/spark-jars/./lib/slf4j-api-1.7.5.jar file:/opt/golfbreaks/spark-jars/./lib/slf4j-log4j12-1.7.5.jar file:/opt/golfbreaks/spark-jars/./lib/snappy-java-1.0.4.1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-catalyst_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-core_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-launcher_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-network-common_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-network-shuffle_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-sketch_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-sql_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-streaming_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-streaming-kafka-0-8_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-tags_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/spark-unsafe_2.11-2.0.0.cloudera1.jar file:/opt/golfbreaks/spark-jars/./lib/stax-api-1.0-2.jar file:/opt/golfbreaks/spark-jars/./lib/stream-2.7.0.jar file:/opt/golfbreaks/spark-jars/./lib/univocity-parsers-2.1.1.jar file:/opt/golfbreaks/spark-jars/./lib/unused-1.0.0.jar file:/opt/golfbreaks/spark-jars/./lib/validation-api-1.1.0.Final.jar file:/opt/golfbreaks/spark-jars/./lib/xbean-asm5-shaded-4.4.jar file:/opt/golfbreaks/spark-jars/./lib/xercesImpl-2.9.1.jar file:/opt/golfbreaks/spark-jars/./lib/xml-apis-1.3.04.jar file:/opt/golfbreaks/spark-jars/./lib/xmlenc-0.52.jar file:/opt/golfbreaks/spark-jars/./lib/xz-1.0.jar file:/opt/golfbreaks/spark-jars/./lib/zkclient-0.7.jar file:/opt/golfbreaks/spark-jars/./lib/zookeeper-3.4.5-cdh5.9.0.jar 17/03/10 19:19:01 INFO spark.SparkContext: Running Spark version 2.0.0.cloudera1 17/03/10 19:19:01 INFO spark.SecurityManager: Changing view acls to: hdfs 17/03/10 19:19:01 INFO spark.SecurityManager: Changing modify acls to: hdfs 17/03/10 19:19:01 INFO spark.SecurityManager: Changing view acls groups to: 17/03/10 19:19:01 INFO spark.SecurityManager: Changing modify acls groups to: 17/03/10 19:19:01 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); groups with view permissions: Set(); users with modify permissions: Set(hdfs); groups with modify permissions: Set() 17/03/10 19:19:02 INFO util.Utils: Successfully started service 'sparkDriver' on port 44306. 17/03/10 19:19:02 INFO spark.SparkEnv: Registering MapOutputTracker 17/03/10 19:19:02 INFO spark.SparkEnv: Registering BlockManagerMaster 17/03/10 19:19:02 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-26ec5da2-b735-4402-832b-10446069948d 17/03/10 19:19:02 INFO memory.MemoryStore: MemoryStore started with capacity 2004.6 MB 17/03/10 19:19:02 INFO spark.SparkEnv: Registering OutputCommitCoordinator 17/03/10 19:19:02 INFO util.log: Logging initialized @2045ms 17/03/10 19:19:02 INFO server.Server: jetty-9.2.z-SNAPSHOT 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7c310a67{/jobs,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77a22fe3{/jobs/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@38f61d71{/jobs/job,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@260702ee{/jobs/job/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@16516ac3{/stages,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b3833a7{/stages/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@66c73750{/stages/stage,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@210099e7{/stages/stage/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2e781d9e{/stages/pool,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fe087e7{/stages/pool/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1dfa77a9{/storage,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@26c6079f{/storage/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@131e6b9c{/storage/rdd,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@322ad892{/storage/rdd/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1af072f9{/environment,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@200c4740{/environment/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@619cb30{/executors,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@76abf71{/executors/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@713e9784{/executors/threadDump,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@444d9531{/executors/threadDump/json,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@417de6ff{/static,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@30c890f0{/,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3fa39595{/api,null,AVAILABLE} 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@cb189d7{/stages/stage/kill,null,AVAILABLE} 17/03/10 19:19:02 INFO server.ServerConnector: Started ServerConnector@2323dd3b{HTTP/1.1}{0.0.0.0:4040} 17/03/10 19:19:02 INFO server.Server: Started @2142ms 17/03/10 19:19:02 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 17/03/10 19:19:02 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://84.40.61.250:4040 17/03/10 19:19:02 INFO spark.SparkContext: Added JAR file:/opt/golfbreaks/spark-jars/streaming-1.0-jar-with-dependencies.jar at spark://84.40.61.250:44306/jars/streaming-1.0-jar-with-dependencies.jar with timestamp 1489173542595 17/03/10 19:19:02 INFO executor.Executor: Starting executor ID driver on host localhost 17/03/10 19:19:02 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40903. 17/03/10 19:19:02 INFO netty.NettyBlockTransferService: Server created on 84.40.61.250:40903 17/03/10 19:19:02 INFO storage.BlockManager: external shuffle service port = 7337 17/03/10 19:19:02 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 84.40.61.250, 40903) 17/03/10 19:19:02 INFO storage.BlockManagerMasterEndpoint: Registering block manager 84.40.61.250:40903 with 2004.6 MB RAM, BlockManagerId(driver, 84.40.61.250, 40903) 17/03/10 19:19:02 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 84.40.61.250, 40903) 17/03/10 19:19:02 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2c94e76a{/metrics/json,null,AVAILABLE} 17/03/10 19:19:03 INFO scheduler.EventLoggingListener: Logging events to hdfs://instance-26765.bigstep.io:8020/user/spark/spark2ApplicationHistory/local-1489173542628 17/03/10 19:19:03 INFO scheduler.ReceiverTracker: Starting 1 receivers 17/03/10 19:19:03 INFO scheduler.ReceiverTracker: ReceiverTracker started 17/03/10 19:19:03 INFO kafka.KafkaInputDStream: Slide time = 1000 ms 17/03/10 19:19:03 INFO kafka.KafkaInputDStream: Storage level = Serialized 1x Replicated 17/03/10 19:19:03 INFO kafka.KafkaInputDStream: Checkpoint interval = null 17/03/10 19:19:03 INFO kafka.KafkaInputDStream: Remember interval = 1000 ms 17/03/10 19:19:03 INFO kafka.KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@732e70a0 17/03/10 19:19:03 INFO dstream.FilteredDStream: Slide time = 1000 ms 17/03/10 19:19:03 INFO dstream.FilteredDStream: Storage level = Serialized 1x Replicated 17/03/10 19:19:03 INFO dstream.FilteredDStream: Checkpoint interval = null 17/03/10 19:19:03 INFO dstream.FilteredDStream: Remember interval = 1000 ms 17/03/10 19:19:03 INFO dstream.FilteredDStream: Initialized and validated org.apache.spark.streaming.dstream.FilteredDStream@4f11189a 17/03/10 19:19:03 INFO dstream.MappedDStream: Slide time = 1000 ms 17/03/10 19:19:03 INFO dstream.MappedDStream: Storage level = Serialized 1x Replicated 17/03/10 19:19:03 INFO dstream.MappedDStream: Checkpoint interval = null 17/03/10 19:19:03 INFO dstream.MappedDStream: Remember interval = 1000 ms 17/03/10 19:19:03 INFO dstream.MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@4ccc69fe 17/03/10 19:19:03 INFO dstream.TransformedDStream: Slide time = 1000 ms 17/03/10 19:19:03 INFO dstream.TransformedDStream: Storage level = Serialized 1x Replicated 17/03/10 19:19:03 INFO dstream.TransformedDStream: Checkpoint interval = null 17/03/10 19:19:03 INFO dstream.TransformedDStream: Remember interval = 1000 ms 17/03/10 19:19:03 INFO dstream.TransformedDStream: Initialized and validated org.apache.spark.streaming.dstream.TransformedDStream@63a2aa36 17/03/10 19:19:03 INFO dstream.ForEachDStream: Slide time = 1000 ms 17/03/10 19:19:03 INFO dstream.ForEachDStream: Storage level = Serialized 1x Replicated 17/03/10 19:19:03 INFO dstream.ForEachDStream: Checkpoint interval = null 17/03/10 19:19:03 INFO dstream.ForEachDStream: Remember interval = 1000 ms 17/03/10 19:19:03 INFO dstream.ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@1c5f5c87 17/03/10 19:19:03 INFO util.RecurringTimer: Started timer for JobGenerator at time 1489173544000 17/03/10 19:19:03 INFO scheduler.JobGenerator: Started JobGenerator at 1489173544000 ms 17/03/10 19:19:03 INFO scheduler.JobScheduler: Started JobScheduler 17/03/10 19:19:03 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@37f7ce71{/streaming,null,AVAILABLE} 17/03/10 19:19:03 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7ee19841{/streaming/json,null,AVAILABLE} 17/03/10 19:19:03 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e6fe928{/streaming/batch,null,AVAILABLE} 17/03/10 19:19:03 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@36099a63{/streaming/batch/json,null,AVAILABLE} 17/03/10 19:19:03 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f97c0e9{/static/streaming,null,AVAILABLE} 17/03/10 19:19:03 INFO streaming.StreamingContext: StreamingContext started 17/03/10 19:19:03 INFO scheduler.ReceiverTracker: Receiver 0 started 17/03/10 19:19:03 INFO scheduler.DAGScheduler: Got job 0 (start at Test.scala:92) with 1 output partitions 17/03/10 19:19:03 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (start at Test.scala:92) 17/03/10 19:19:03 INFO scheduler.DAGScheduler: Parents of final stage: List() 17/03/10 19:19:03 INFO scheduler.DAGScheduler: Missing parents: List() 17/03/10 19:19:03 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:610), which has no missing parents 17/03/10 19:19:04 ERROR scheduler.JobScheduler: Error generating jobs for time 1489173544000 ms java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1087) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1078) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:88) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:53) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:680) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:678) at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:46) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415) at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:335) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:333) at scala.Option.orElse(Option.scala:289) at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:330) at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:48) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:117) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:116) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:116) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:249) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:247) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:247) at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:183) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) ... 44 more Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1087) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1078) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:88) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:53) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:680) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:678) at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:46) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415) at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:335) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:333) at scala.Option.orElse(Option.scala:289) at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:330) at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:48) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:117) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:116) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:116) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:249) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:247) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:247) at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:183) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) ... 44 more 17/03/10 19:19:04 INFO streaming.StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook 17/03/10 19:19:04 INFO scheduler.ReceiverTracker: Sent stop signal to all 1 receivers 17/03/10 19:19:04 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 71.3 KB, free 2004.5 MB) 17/03/10 19:19:04 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.9 KB, free 2004.5 MB) 17/03/10 19:19:04 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 84.40.61.250:40903 (size: 25.9 KB, free: 2004.6 MB) 17/03/10 19:19:04 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1012 17/03/10 19:19:04 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:610) 17/03/10 19:19:04 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 17/03/10 19:19:04 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7149 bytes) 17/03/10 19:19:04 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 17/03/10 19:19:04 INFO executor.Executor: Fetching spark://84.40.61.250:44306/jars/streaming-1.0-jar-with-dependencies.jar with timestamp 1489173542595 17/03/10 19:19:04 INFO client.TransportClientFactory: Successfully created connection to /84.40.61.250:44306 after 47 ms (0 ms spent in bootstraps) 17/03/10 19:19:04 INFO util.Utils: Fetching spark://84.40.61.250:44306/jars/streaming-1.0-jar-with-dependencies.jar to /tmp/spark-0240c93e-4bd3-4c59-b56c-40c5b6499c44/userFiles-86a0ab49-294c-40f2-88a5-c6ff6514116b/fetchFileTemp3828224251343929630.tmp 17/03/10 19:19:05 ERROR scheduler.JobScheduler: Error generating jobs for time 1489173545000 ms java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:232) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1087) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1078) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:88) at com.golfbreaks.spark.streaming.Test$$anonfun$4.apply(Test.scala:53) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21.apply(DStream.scala:666) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:680) at org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5.apply(DStream.scala:678) at org.apache.spark.streaming.dstream.TransformedDStream.compute(TransformedDStream.scala:46) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:341) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:340) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415) at org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalProperties(TransformedDStream.scala:65) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:335) at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:333) at scala.Option.orElse(Option.scala:289) at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:330) at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:48) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:117) at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:116) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:116) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:249) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$3.apply(JobGenerator.scala:247) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:247) at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:183) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89) at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) ... 44 more 17/03/10 19:19:05 INFO executor.Executor: Adding file:/tmp/spark-0240c93e-4bd3-4c59-b56c-40c5b6499c44/userFiles-86a0ab49-294c-40f2-88a5-c6ff6514116b/streaming-1.0-jar-with-dependencies.jar to class loader 17/03/10 19:19:05 INFO util.RecurringTimer: Started timer for BlockGenerator at time 1489173545400 17/03/10 19:19:05 INFO receiver.BlockGenerator: Started BlockGenerator 17/03/10 19:19:05 INFO receiver.BlockGenerator: Started block pushing thread 17/03/10 19:19:05 INFO receiver.ReceiverSupervisorImpl: Stopping receiver with message: Registered unsuccessfully because Driver refused to start receiver 0: 17/03/10 19:19:05 WARN receiver.ReceiverSupervisorImpl: Skip stopping receiver because it has not yet stared 17/03/10 19:19:05 INFO receiver.BlockGenerator: Stopping BlockGenerator 17/03/10 19:19:05 INFO util.RecurringTimer: Stopped timer for BlockGenerator after time 1489173545600 17/03/10 19:19:05 INFO receiver.BlockGenerator: Waiting for block pushing thread to terminate 17/03/10 19:19:05 INFO receiver.BlockGenerator: Pushing out the last 0 blocks 17/03/10 19:19:05 INFO receiver.BlockGenerator: Stopped block pushing thread 17/03/10 19:19:05 INFO receiver.BlockGenerator: Stopped BlockGenerator 17/03/10 19:19:05 INFO receiver.ReceiverSupervisorImpl: Waiting for receiver to be stopped 17/03/10 19:19:05 INFO receiver.ReceiverSupervisorImpl: Stopped receiver without error 17/03/10 19:19:05 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 787 bytes result sent to driver 17/03/10 19:19:05 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1313 ms on localhost (executor driver) (1/1) 17/03/10 19:19:05 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/03/10 19:19:05 INFO scheduler.DAGScheduler: ResultStage 0 (start at Test.scala:92) finished in 1.335 s 17/03/10 19:19:05 INFO scheduler.ReceiverTracker: All of the receivers have deregistered successfully 17/03/10 19:19:05 INFO scheduler.ReceiverTracker: ReceiverTracker stopped 17/03/10 19:19:05 INFO scheduler.JobGenerator: Stopping JobGenerator immediately 17/03/10 19:19:05 INFO util.RecurringTimer: Stopped timer for JobGenerator after time 1489173545000 17/03/10 19:19:05 INFO scheduler.JobGenerator: Stopped JobGenerator 17/03/10 19:19:05 INFO scheduler.JobScheduler: Stopped JobScheduler 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@37f7ce71{/streaming,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5e6fe928{/streaming/batch,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7f97c0e9{/static/streaming,null,UNAVAILABLE} 17/03/10 19:19:05 INFO streaming.StreamingContext: StreamingContext stopped successfully 17/03/10 19:19:05 INFO spark.SparkContext: Invoking stop() from shutdown hook 17/03/10 19:19:05 INFO server.ServerConnector: Stopped ServerConnector@2323dd3b{HTTP/1.1}{0.0.0.0:4040} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@cb189d7{/stages/stage/kill,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3fa39595{/api,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@30c890f0{/,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@417de6ff{/static,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@444d9531{/executors/threadDump/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@713e9784{/executors/threadDump,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@76abf71{/executors/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@619cb30{/executors,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@200c4740{/environment/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1af072f9{/environment,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@322ad892{/storage/rdd/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@131e6b9c{/storage/rdd,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@26c6079f{/storage/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1dfa77a9{/storage,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4fe087e7{/stages/pool/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2e781d9e{/stages/pool,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@210099e7{/stages/stage/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@66c73750{/stages/stage,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3b3833a7{/stages/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@16516ac3{/stages,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@260702ee{/jobs/job/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@38f61d71{/jobs/job,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@77a22fe3{/jobs/json,null,UNAVAILABLE} 17/03/10 19:19:05 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7c310a67{/jobs,null,UNAVAILABLE} 17/03/10 19:19:05 INFO ui.SparkUI: Stopped Spark web UI at http://84.40.61.250:4040 17/03/10 19:19:05 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/03/10 19:19:05 INFO memory.MemoryStore: MemoryStore cleared 17/03/10 19:19:05 INFO storage.BlockManager: BlockManager stopped 17/03/10 19:19:05 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 17/03/10 19:19:05 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/03/10 19:19:05 INFO spark.SparkContext: Successfully stopped SparkContext 17/03/10 19:19:05 INFO util.ShutdownHookManager: Shutdown hook called 17/03/10 19:19:05 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0240c93e-4bd3-4c59-b56c-40c5b6499c44 -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org