[
https://issues.apache.org/jira/browse/MAHOUT-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969759#comment-13969759
]
Pat Ferrel commented on MAHOUT-1464:
------------------------------------
Running from my dev machine in IDEA against a remote cluster I can read input
from HDFS and the computation on Spark seems to complete correctly but the
write fails.
input, output,
mahoutSparkContext(masterUrl = , Success?
HDFS:/occam4/user/pat/xrsj, HDFS:/occam4/user/pat/tmp, spark://occam4:7077,
no, computation competes, but output
gives a
failure to connect message
while
trying to write to HDFS
Again this may be a URI error in the output path or some config problem. From
the shell I can examine HDFS and all is as expected. Hadoop jobs that I launch
from the command line work against the cluster correctly.
14/04/15 10:04:02 INFO storage.MemoryStore: Block broadcast_3 stored as values
to memory (estimated size 385.2 KB, free 4.7 GB)
14/04/15 10:04:02 INFO rdd.FlatMappedRDD: Removing RDD 16 from persistence list
14/04/15 10:04:02 INFO storage.BlockManager: Removing RDD 16
14/04/15 10:04:02 INFO rdd.FlatMappedRDD: Removing RDD 7 from persistence list
14/04/15 10:04:02 INFO storage.BlockManager: Removing RDD 7
14/04/15 10:04:09 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:10 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:11 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:12 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 3 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:13 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 4 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:14 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 5 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:15 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 6 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:16 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 7 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:17 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 8 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/15 10:04:18 INFO ipc.Client: Retrying connect to server:
occam4/192.168.0.14:8020. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
Exception in thread "main" java.net.ConnectException: Call to
occam4/192.168.0.14:8020 failed on connection exception:
java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at
org.apache.hadoop.mapred.SparkHadoopWriter$.createPathFromString(SparkHadoopWriter.scala:193)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:685)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:572)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:894)
at
org.apache.mahout.cf.examples.RecommendationExamplesHelper$.saveIndicatorMatrix(Recommendations.scala:178)
at
org.apache.mahout.cf.examples.RunCrossCooccurrenceAnalysisOnEpinions$.main(Recommendations.scala:111)
at
org.apache.mahout.cf.examples.RunCrossCooccurrenceAnalysisOnEpinions.main(Recommendations.scala)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
at org.apache.hadoop.ipc.Client.call(Client.java:1093)
... 26 more
Disconnected from the target VM, address: '127.0.0.1:59483', transport: 'socket'
> Cooccurrence Analysis on Spark
> ------------------------------
>
> Key: MAHOUT-1464
> URL: https://issues.apache.org/jira/browse/MAHOUT-1464
> Project: Mahout
> Issue Type: Improvement
> Components: Collaborative Filtering
> Environment: hadoop, spark
> Reporter: Pat Ferrel
> Assignee: Sebastian Schelter
> Fix For: 1.0
>
> Attachments: MAHOUT-1464.patch, MAHOUT-1464.patch, MAHOUT-1464.patch,
> MAHOUT-1464.patch, MAHOUT-1464.patch, MAHOUT-1464.patch, run-spark-xrsj.sh
>
>
> Create a version of Cooccurrence Analysis (RowSimilarityJob with LLR) that
> runs on Spark. This should be compatible with Mahout Spark DRM DSL so a DRM
> can be used as input.
> Ideally this would extend to cover MAHOUT-1422. This cross-cooccurrence has
> several applications including cross-action recommendations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
