Re: Any role for volunteering

[Matthias J. Sax]

Hi Deepak,

the Flink community is always open to new people who want to contribute
to the project. Please subscribe to the user- and dev-mailing list as a
starting point: https://flink.apache.org/community.html#mailing-lists

Furthermore, please read the following docs:
https://flink.apache.org/how-to-contribute.html
https://flink.apache.org/contribute-code.html

It explains the process the Flink community follows and you have to
follow, too.

The best way to get started with coding is to look into open tickets:
https://issues.apache.org/jira/browse/FLINK

If you find anything you want to work on, let us know.


-Matthias


On 12/04/2015 06:58 PM, Deepak Sharma wrote:
> Hi All
> Sorry for spamming your inbox.
> I am really keen to work on a big data project full time(preferably
> remote from India) , if not I am open to volunteering as well.
> Please do let me know if there is any such opportunity available
> 
> -- 
> Thanks
> Deepak



signature.asc
Description: OpenPGP digital signature

Re: [IE] Re: passing environment variables to flink program

[Robert Metzger]

Just a little note, the feature requested in FLINK-2954
 has been implemented and
is available in 1.0-SNAPSHOT now.

Please let me know if its working as expected.

On Wed, Nov 4, 2015 at 6:35 PM, Jian Jiang  wrote:

> I have to continue the evaluation so I managed to patch a fix to my local
> build. In case someone cannot wait here is what I did:
>
> 1. In the flink-core's ConfigConstants.java add
>
> public static final String FLINK_JVM_OPTIONS = "env.java.opts"; <--
> existing
> public static final String FLINK_CUSTOM_ENV_VARS = "env.custom.env.vars";
>
> 2. In the flink-conf.yaml now you can add environment variables separated
> by
> "|":
>
> env.custom.env.vars: name1=value1|name2=value2|name3=value3 
>
> 3. In the flink-yarn's YarnJobManager.scala's
> createContainerLaunchContext()
> method add the following between the two existing lines:
>
> Utils.setupEnv(yarnConf, containerEnv) <-- existing
>
> val customEnvOpts =
> flinkConfiguration.getString(ConfigConstants.FLINK_CUSTOM_ENV_VARS, "")
>
> if(customEnvOpts!=null)
> {
> val variables = customEnvOpts.split("\\|");
> if(variables!=null&>0)
> {
> for(variable <- variables)
> {
> val pair = variable.split("=");
> containerEnv.put(pair(0), pair(1));
> }
> }
> }
> containerEnv.put(FlinkYarnClientBase.ENV_CLIENT_USERNAME,
> yarnClientUsername) <-- existing
>
> Now the container running my program sees the env variables.
>
> Note this is temp work around for local personally and should be discarded
> once Flink 1.0 comes out.
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/passing-environment-variables-to-flink-program-tp3337p3367.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Re: Read Kafka topic from the beginning

[Robert Metzger]

Hi Vladimir,

Does current Kafka Consumer implementation allow to read all messages in a
> topic from the beginning or from a specific offset.


For reading from the beginning, setting "auto.offset.reset" to "smallest"
will do the job.
Reading from a specific offset is not yet supported yet, but it is very
easy to implement on top of: https://github.com/apache/flink/pull/1437,
that's why I filed a JIRA for this feature:
https://issues.apache.org/jira/browse/FLINK-3123
I'll implement it once the pull request is merged. Expect it in the course
of next week.


On Thu, Dec 3, 2015 at 11:55 PM, Stephan Ewen  wrote:

> Hi Vladimir!
>
> The Kafka Consumer can start from any offset internally (it does that for
> example when recovering a failure).
>
> Should be fairly straightforward to set that offset field initially from a
> parameter. The FlinkKafkaConsumer is part of the user jars anyways. If you
> want, you can give it a try to create a modified version that accepts that
> parameter, and then package that instead of the standard one.
>
> Greetings,
> Stephan
>
>
> On Thu, Dec 3, 2015 at 7:07 PM, Maximilian Michels  wrote:
>
>> Hi Vladimir,
>>
>> Did you pass the properties to the FlinkKafkaConsumer?
>>
>> Cheers,
>> Max
>>
>> On Thu, Dec 3, 2015 at 7:06 PM, Vladimir Stoyak 
>> wrote:
>> > Gave it a try, but does not seem to help. Is it working for you?
>> >
>> > Thanks
>> >
>> > Sent from my iPhone
>> >
>> >> On Dec 3, 2015, at 6:11 PM, Vladimir Stoyak  wrote:
>> >>
>> >> As far as I know "auto.offset.reset" what to do if offset it not
>> available or out of bound?
>> >>
>> >> Vladimir
>> >>
>> >>
>> >> On Thursday, December 3, 2015 5:58 PM, Maximilian Michels <
>> m...@apache.org> wrote:
>> >> Hi Vladimir,
>> >>
>> >> You may supply Kafka consumer properties when you create the
>> FlinkKafkaConsumer.
>> >>
>> >> Properties props = new Properties();
>> >>
>> >> // start from largest offset - DEFAULT
>> >> props.setProperty("auto.offset.reset", "largest");
>> >> // start from smallest offset
>> >> props.setProperty("auto.offset.reset", "smallest");
>> >>
>> >> I don't think it is possible to start from a specific offset. The
>> >> offset is only unique per partition. You could modify the offsets in
>> >> the Zookeeper state but you really have to know what you're doing
>> >> then.
>> >>
>> >> Best regards,
>> >> Max
>> >>
>> >>
>> >>
>> >>> On Thu, Dec 3, 2015 at 4:01 PM, Vladimir Stoyak 
>> wrote:
>> >>> I see that Flink 0.10.1 now supports Keyed Schemas which allows us to
>> rely on Kafka topics set to "compact" retention for data persistence.
>> >>>
>> >>> In our topology we wanted to set some topics with Log Compactions
>> enabled and read topic from the beginning when the topology starts or
>> component recovers. Does current Kafka Consumer implementation allow to
>> read all messages in a topic from the beginning or from a specific offset.
>> >>>
>> >>> Thanks,
>> >>> Vladimir
>>
>
>