Hi, I am running eagle on a local OSX cluster and for step 2 of the offline user profile training, I run the command “bin/eagle-userprofile-scheduler.sh –site sandbox start” as explained. However, I am faced with an error with the attached stack trace. The exception is “Exception in thread "main" java.lang.IllegalArgumentException: Illegal character in authority at index 7: hdfs://NN URL:8020” . Any help on this would be very much appreciated. Thanks in advance for your time.
Best, Peter
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/07/22 17:19:59 INFO SparkContext: Running Spark version 1.6.2 16/07/22 17:20:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/07/22 17:20:00 WARN Utils: Your hostname, LM-SJN-21002944 resolves to a loopback address: 127.0.0.1; using 10.225.90.61 instead (on interface en0) 16/07/22 17:20:00 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 16/07/22 17:20:00 INFO SecurityManager: Changing view acls to: petkim 16/07/22 17:20:00 INFO SecurityManager: Changing modify acls to: petkim 16/07/22 17:20:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(petkim); users with modify permissions: Set(petkim) 16/07/22 17:20:00 INFO Utils: Successfully started service 'sparkDriver' on port 52369. 16/07/22 17:20:01 INFO Slf4jLogger: Slf4jLogger started 16/07/22 17:20:01 INFO Remoting: Starting remoting 16/07/22 17:20:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:52370] 16/07/22 17:20:01 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 52370. 16/07/22 17:20:01 INFO SparkEnv: Registering MapOutputTracker 16/07/22 17:20:01 INFO SparkEnv: Registering BlockManagerMaster 16/07/22 17:20:01 INFO DiskBlockManager: Created local directory at /private/var/folders/51/v_6pr2l13m3dyvc368zx055w3kfvby/T/blockmgr-29e02fbb-2d02-4d2c-a1bb-11952dfb6bab 16/07/22 17:20:01 INFO MemoryStore: MemoryStore started with capacity 511.5 MB 16/07/22 17:20:01 INFO SparkEnv: Registering OutputCommitCoordinator 16/07/22 17:20:01 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/07/22 17:20:01 INFO SparkUI: Started SparkUI at http://10.225.90.61:4040 16/07/22 17:20:01 INFO HttpFileServer: HTTP File server directory is /private/var/folders/51/v_6pr2l13m3dyvc368zx055w3kfvby/T/spark-327a0312-0033-4ef0-a46b-94fc937e70ce/httpd-403f4245-5fd8-41ec-9e98-cf8ecd13c58d 16/07/22 17:20:01 INFO HttpServer: Starting HTTP Server 16/07/22 17:20:01 INFO Utils: Successfully started service 'HTTP file server' on port 52371. 16/07/22 17:20:01 INFO SparkContext: Added JAR file:/Users/petkim/Documents/ApacheEagle-2.0/apache-eagle-0.4.0-incubating/bin/../lib/userprofile/eagle-security-userprofile-training-0.4.0-incubating-assembly.jar at http://10.225.90.61:52371/jars/eagle-security-userprofile-training-0.4.0-incubating-assembly.jar with timestamp 1469233201736 16/07/22 17:20:01 INFO Executor: Starting executor ID driver on host localhost 16/07/22 17:20:01 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52376. 16/07/22 17:20:01 INFO NettyBlockTransferService: Server created on 52376 16/07/22 17:20:01 INFO BlockManagerMaster: Trying to register BlockManager 16/07/22 17:20:01 INFO BlockManagerMasterEndpoint: Registering block manager localhost:52376 with 511.5 MB RAM, BlockManagerId(driver, localhost, 52376) 16/07/22 17:20:01 INFO BlockManagerMaster: Registered BlockManager 16/07/22 17:20:01 INFO UserProfileTrainingApp: Registering UserProfileEigenModeler(getfileinfo,open,listStatus,setTimes,setPermission,rename,mkdirs,create,setReplication,contentSummary,delete,setOwner,fsck) 16/07/22 17:20:01 INFO UserProfileTrainingApp: Registering UserProfileKDEModeler(getfileinfo,open,listStatus,setTimes,setPermission,rename,mkdirs,create,setReplication,contentSummary,delete,setOwner,fsck) 16/07/22 17:20:01 INFO UserProfileTrainingApp: Registering UserProfileEagleServiceSink(localhost,9099, admin, secret) 16/07/22 17:20:01 INFO UserProfileTrainingApp: Starting to run 16/07/22 17:20:01 INFO AuditLogTrainingSparkJob: Starting UserProfileTraining 16/07/22 17:20:02 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 153.6 KB, free 153.6 KB) 16/07/22 17:20:02 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 14.0 KB, free 167.6 KB) 16/07/22 17:20:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:52376 (size: 14.0 KB, free: 511.5 MB) 16/07/22 17:20:02 INFO SparkContext: Created broadcast 0 from textFile at AuditLogTrainingSparkJob.scala:75 Exception in thread "main" java.lang.IllegalArgumentException: Illegal character in authority at index 7: hdfs://NN URL:8020 at java.net.URI.create(URI.java:859) at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169) at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656) at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:436) at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:409) at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015) at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176) at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176) at scala.Option.map(Option.scala:145) at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:330) at org.apache.eagle.security.userprofile.job.AuditLogTrainingSparkJob.buildDAG(AuditLogTrainingSparkJob.scala:81) at org.apache.eagle.security.userprofile.job.AuditLogTrainingSparkJob.run(AuditLogTrainingSparkJob.scala:53) at org.apache.eagle.security.userprofile.UserProfileTrainingApp$$anonfun$1.apply(UserProfileTrainingApp.scala:55) at org.apache.eagle.security.userprofile.UserProfileTrainingApp$$anonfun$1.apply(UserProfileTrainingApp.scala:32) at org.apache.eagle.security.userprofile.UserProfileJobFactory$.AuditlogTrainingSparkJob(UserProfileJobFactory.scala:28) at org.apache.eagle.security.userprofile.UserProfileTrainingApp.<init>(UserProfileTrainingApp.scala:32) at org.apache.eagle.security.userprofile.UserProfileTrainingCLI$.main(UserProfileTrainingCLI.scala:51) at org.apache.eagle.security.userprofile.UserProfileTrainingCLI.main(UserProfileTrainingCLI.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.net.URISyntaxException: Illegal character in authority at index 7: hdfs://NN URL:8020 at java.net.URI$Parser.fail(URI.java:2829) at java.net.URI$Parser.parseAuthority(URI.java:3167) at java.net.URI$Parser.parseHierarchical(URI.java:3078) at java.net.URI$Parser.parse(URI.java:3034) at java.net.URI.<init>(URI.java:595) at java.net.URI.create(URI.java:857) ... 60 more 16/07/22 17:20:02 INFO SparkContext: Invoking stop() from shutdown hook 16/07/22 17:20:02 INFO SparkUI: Stopped Spark web UI at http://10.225.90.61:4040 16/07/22 17:20:02 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/07/22 17:20:02 INFO MemoryStore: MemoryStore cleared 16/07/22 17:20:02 INFO BlockManager: BlockManager stopped 16/07/22 17:20:02 INFO BlockManagerMaster: BlockManagerMaster stopped 16/07/22 17:20:02 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
