Re: when use broadcast variable and run on bigdata display this error please help

[Chiwan Park]

Hi hagersaleh,

You should know why the error occurred with large scale data. Broadcast 
variables can handle only data of which size is fit for single machine.

I meant that using an external system such as Redis, HBase, …, etc. The 
connection with the external system could be initialized in `open` method of 
rich functions such as `RichFilterFunction`, `RichFlatMapFunction`, …, etc..

You can choose another solution which Stephen said. He said that rethink your 
approach. I think that rethinking your algorithm would be better than my 
suggestion.

From your code, I don’t understand why you want to use broadcast variable. You 
can do same thing with filter and join operations. Here is my implementation 
[1].


Regards,
Chiwan Park

[1] https://gist.github.com/chiwanpark/a0b0269c9a9b058d15d3


> On Sep 4, 2015, at 3:51 AM, hagersaleh  wrote:
> 
> Hi Chiwan Park 
> not understand this solution please explain more
> 
> 
> 
> --
> View this message in context: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2676.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at 
> Nabble.com.

Re: when use broadcast variable and run on bigdata display this error please help

[hagersaleh]

Hi Chiwan Park 
not understand this solution please explain more



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2676.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: when use broadcast variable and run on bigdata display this error please help

[hagersaleh]

When to use broadcast variable?

Distribute data with a broadcast variable when

The data is large
The data has been produced by some form of computation and is already a
DataSet (distributed result)
Typical use case: Redistribute intermediate results, such as trained
models


from link
https://cwiki.apache.org/confluence/display/FLINK/Variables+Closures+vs.+Broadcast+Variables



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2488.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: when use broadcast variable and run on bigdata display this error please help

[hagersaleh]

why this is not good broadcast variable use in bigdata



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2468.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: when use broadcast variable and run on bigdata display this error please help

[Rico Bergmann]

As you can see from the exceptions your broadcast variable is too large to fit 
into the main memory. 

I think storing that amount of data in a broadcast variable is not the best 
approach. Try to use a dataset for this, I would suggest. 



 Am 20.08.2015 um 11:56 schrieb hagersaleh loveallah1...@yahoo.com:
 
 please help
 
 
 
 --
 View this message in context: 
 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2461.html
 Sent from the Apache Flink User Mailing List archive. mailing list archive at 
 Nabble.com.