Hi Andrea,

I’m not expert of Scala but It seems about closure cleaning problem. Scala App 
trait extends DelayedInit trait to initialize object. But Flink serialization 
stack doesn’t handle this special initialization. (It is just my opinion, not 
verified.)

To run TFIDFNPE safely, you need to just change tokenize and uniqueWords to 
method like following:

```
def tokenize = …
def uniqueWords = ...
```

With this change, I tested that TFIDFNPE works safely in Flink 0.10.1 cluster.

About TFIDF object, you should avoid overriding main method if the object is 
derived by App trait. It is also related DelaytedInit mechanism [1].

[1]: https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31

> On Jan 19, 2016, at 12:08 AM, Andrea Sella <[email protected]> wrote:
> 
> Hi,
> 
> I was implementing TF-IDF example of flink-training when I faced a problem 
> with NPE during the deploy of my Job.
> 
> Source code: https://github.com/alkagin/flink-tfidf-example
> 
> I used 0.10.1 version and started in local mode.
> During the deploy of TFIDFNPE Job, which it extends App, Flink throws 
> NullPointerException on both flatMap functions. 
> If I include the tokenize function into the closures of flatMap functions, 
> the Job works fine; see example TFIDFApp.
> To avoid this unexpected behavior I don't have use Scala App trait, see 
> TFIDF, but why?
> 
> 
> Thanks,
> Andrea
> 


Regards,
Chiwan Park

Reply via email to