Hello,I'm working with this set of technologies (Spark 1.6.0 and Ignite 1.6.0
) and Im having some issues, the most troublesome is this exception:
6/10/20 22:09:15 ERROR DAGSchedulerEventProcessLoop:
DAGSchedulerEventProcessLoop failed; shutting down
SparkContextjava.lang.NumberFormatException: For input string: "1%1" at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492) at
java.lang.Integer.parseInt(Integer.java:527) at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229) at
scala.collection.immutable.StringOps.toInt(StringOps.scala:31) at
org.apache.spark.util.Utils$.parseHostPort(Utils.scala:877) at
org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:37)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:208)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:187)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:187)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:166)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at
org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:165) at
org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:200)
at
org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:164)
at
org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1052)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:921)
at
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:861)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1607)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
I'm using the XML config to use the DiscoverySPI from S3
<property name="discoverySpi"><bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"><property
name="ipFinder"><bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder"><property
name="awsCredentials" ref="aws.creds"/><property
name="bucketName"
value="embedded-ignite-discovery"/></bean></property>
But that doesn't work, So I changed the scala code to do the same inside the
code:
val cfg = new IgniteConfiguration with Serializable//val clientCfg = new
ClientConfiguration()val ipFinder = new CustomTcpDiscoveryS3IpFinderval
discoverySPI = new TcpDiscoverySpival accessKey =
configHelper.envOrElseConfig("aws.security.credentials.access_key")val
secretKey =
configHelper.envOrElseConfig("aws.security.credentials.secret_key")val
awsCredentials = new BasicAWSCredentials(accessKey,
secretKey)ipFinder.setAwsCredentials(awsCredentials)ipFinder.setBucketName(configHelper.envOrElseConfig("ignite.aws.s3.bucket_name"))discoverySPI.setIpFinder(ipFinder)
But that fails too, it throws the same exception with the "1%1",When I check
the bucket I can see a file called "0:0:0:0:0:0:0:1:1%1" and I think that is
the IPv6 address of the loopback interface and in some way spark or ignite
is failing to handle that address.In the attempt log I can see this:
>>> +----------------------------------------------------------------------+>>>
>>> Ignite ver.
>>> 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031>>>
>>> +----------------------------------------------------------------------+>>>
>>> OS name: Linux 4.1.17-22.30.amzn1.x86_64 amd64>>> CPU(s): 4>>> Heap:
>>> 11.0GB>>> VM name: 6268@ip-172-31-41-59>>> Local node
>>> [ID=B284DCEF-BB88-4AF1-BAA2-C7F337A8E579, order=2, clientMode=true]>>>
>>> Local node addresses: [ip-172-31-41-59.ec2.internal/*0:0:0:0:0:0:0:1%1*,
>>> /127.0.0.1, /172.31.41.59]>>> Local ports: TCP:11211 TCP:47100 TCP:48100
Notice the "*0:0:0:0:0:0:0:1%1*"I also tried to do a:
cfg.setDiscoverySpi(discoverySPI)cfg.setLocalHost("127.0.0.1")cfg.setClientMode(true)cfg.setPeerClassLoadingEnabled(false)
But in that case it hangs in:
16/10/20 22:49:27 INFO IgniteKernal: Security status [authentication=off,
tls/ssl=off]16/10/20 22:49:27 INFO GridTcpRestProtocol: Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0,
port=11211]16/10/20 22:49:28 WARN CustomTcpDiscoveryS3IpFinder: Amazon
client configuration is not set (will use default).16/10/20 22:49:28 WARN
TcpDiscoverySpi: Failed to connect to any address from IP finder (will retry
to join topology every 2 secs): [/127.0.0.1:47500]16/10/20 22:51:36 INFO
ContextCleaner: Cleaned accumulator 1
Any help is really appreciated.Thanks,P.S Sorry if the post appears twice
but I sent the post to the mailing list some days ago and it doesn't show in
here and I really need your help.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
