Sorry, I just realized our previous conversation on this question was done via
private email and not to user@flink.apache.org
Forwarding the previous content of the conversation back to the mailing list:
On March 30, 2017 at 4:15:46 PM, rimin...@sina.cn (rimin...@sina.cn) wrote:
the job can run success,but the result is error.
the record 1 and the record 14 is same,so the vector compute cos value is 1,but
on the yarn the value is not 1,and others are different from the result which
run on local.
so,i guess,the step:
1 val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
2 val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
3 val rescomm = computeCosSims
(dataVec)//DataSet[(String,Array[(String,Double)])]
the record is from 1,2,3;but the step 3 must start when step 2 is end,because
step 3 compute all record cos sim value must use all data.so is there some
operate can set the step 3 start when step 2 is end.
----- 原始邮件 -----
发件人："Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
收件人：rimin...@sina.cn
主题：Re: flink one transformation end,the next transformation start
日期：2017年03月30日 15点54分
Hi,
What exactly is the error you’re running into on YARN? You should be able to
find them in the TM logs.
It’ll be helpful to understand the problem if you can provide them (just the
relevant parts of the error will do).
Otherwise, I currently can not tell much from the information here.
Cheers,
Gordon
On March 30, 2017 at 3:33:53 PM, rimin...@sina.cn (rimin...@sina.cn) wrote:
hi,all,
i run a job,it is :
---------------------------------------------------------
val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]
but when run on the yarn cluster,the result was error,the job can success;and
run on the local,in eclipse on my computer,the result is correct.
so,i run twice,
first:
val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
dataVec.writeAsText("hdfs///vec")//the vector is correct,
second:
val readVec =
env.readTextFile("hdfs:///vec").map(...)//DataSet[(String,Int,Array[(Int,Double)])]
val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]
and the result is correct,is the same as on local,in eclispe.
----------------------------------
someone can solve the problem?
