Re: 1.2-SNAPSHOT documentation

[Timo Walther]

Hi Ken,

the docs on the website need to be built manually at the moment. So they 
might be out of sync.
If you want the most recent documentation you can checkout the git 
repository and built the docs:


git clone https://github.com/apache/flink.git
./flink/docs/build_docs.sh

Regards,
Timo


Am 04/12/16 um 02:42 schrieb Ken Krugler:

Hi all,

Quick question - how quickly are the online Javadocs updated to match snapshot 
builds?

E.g. 
https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/datastream/KeyedStream.html
 


says it’s the 1.2-SNAPSHOT documentation, but it doesn’t match the current 
1.2-SNAPSHOT jars that I’m downloading.

In particular, the RichTimelyFlatMapFunction is still in the documentation, but 
not the ProcessFunction.

So should I be using a different link for snapshot Javadocs?

Thanks,

— Ken

--
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr







--
Freundliche Grüße / Kind Regards

Timo Walther

Follow me: @twalthr
https://www.linkedin.com/in/twalthr

答复: [jira] [Created] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[liuxinchun]

Where can I find the usage of CheckpointedFunction? And I found the ckeckpoint 
problem in the following issue:

https://issues.apache.org/jira/browse/FLINK-5221


-邮件原件-
发件人: Aljoscha Krettek (JIRA) [mailto:j...@apache.org] 
发送时间: 2016年12月5日 8:11
收件人: dev@flink.apache.org
主题: [jira] [Created] (FLINK-5250) Make AbstractUdfStreamOperator aware of 
WrappingFunction

Aljoscha Krettek created FLINK-5250:
---

 Summary: Make AbstractUdfStreamOperator aware of WrappingFunction
 Key: FLINK-5250
 URL: https://issues.apache.org/jira/browse/FLINK-5250
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Reporter: Aljoscha Krettek


Right now, when using a {{WrappingFunction}}, as happens for {{WindowFunction}} 
and also for some functions in the Scala API then using custom interfaces is 
not possible. These custom interfaces are, for example, the checkpointing 
functions such as {{ListCheckpointed}} and {{CheckpointedFunction}}.

We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so that 
it can correctly handle the case where wrapped user functions implement on of 
these interfaces.

Also, in the Scala API we have some custom functions that mimic 
{{WrappingFunction}} behaviour. These should be moved to use 
{{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5253) Remove special treatment of "dynamic properties"

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-5253:
---

 Summary: Remove special treatment of "dynamic properties"
 Key: FLINK-5253
 URL: https://issues.apache.org/jira/browse/FLINK-5253
 Project: Flink
  Issue Type: Sub-task
  Components: YARN
 Environment: {{flip-6}} feature branch
Reporter: Stephan Ewen


The YARN client accepts configuration keys as command line parameters.

Currently these are send to the AppMaster and TaskManager as "dynamic 
properties", encoded in a special way via environment variables.

The mechanism is quite fragile. We should simplify it:

  - The YARN client takes the local {{flink-conf.yaml}} as the base.
  - It overwrite config entries with command line properties when preparing the 
configuration to be shipped to YARN container processes (JM / TM)
  - No additional handling neccessary




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5252) Migrate all YARN related configurations to ConfigOptions

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-5252:
---

 Summary: Migrate all YARN related configurations to ConfigOptions
 Key: FLINK-5252
 URL: https://issues.apache.org/jira/browse/FLINK-5252
 Project: Flink
  Issue Type: Sub-task
  Components: YARN
 Environment: {{flip-6}} feature branch
Reporter: Stephan Ewen


Since a few months, we started using a more elaborate way of defining 
configuration options, together with default values and deprecated keys.

The options are defined via the {{ConfigOptions}} class. A good example how to 
use it is in the {{HigHAvailabilityOptions}}.

All Yarn configuration options should be defined that way as we..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5250) Make AbstractUdfStreamOperator aware of WrappingFunction

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-5250:
---

 Summary: Make AbstractUdfStreamOperator aware of WrappingFunction
 Key: FLINK-5250
 URL: https://issues.apache.org/jira/browse/FLINK-5250
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Reporter: Aljoscha Krettek


Right now, when using a {{WrappingFunction}}, as happens for {{WindowFunction}} 
and also for some functions in the Scala API then using custom interfaces is 
not possible. These custom interfaces are, for example, the checkpointing 
functions such as {{ListCheckpointed}} and {{CheckpointedFunction}}.

We should teach {{AbstractUdfStreamOperator}} about {{WrapingFunction}} so that 
it can correctly handle the case where wrapped user functions implement on of 
these interfaces.

Also, in the Scala API we have some custom functions that mimic 
{{WrappingFunction}} behaviour. These should be moved to use 
{{WrappingFunction}}, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Savepoints seem to not work properly on release-1.1 branch

[Gyula Fóra]

I tried it, works fine!

Ufuk Celebi  ezt írta (időpont: 2016. dec. 4., V, 18:52):

> Thanks. It has been merged now. :-)
>
> On 4 December 2016 at 17:24:30, Gyula Fóra (gyula.f...@gmail.com) wrote:
> > Hi,
> > Thanks for being so quick, I will try the fix as soon as it is merged. :)
> > Gyula
> >
> > Ufuk Celebi ezt írta (időpont: 2016. dec. 4., V, 12:02):
> >
> > > Thanks for reporting this Gyula. A fix is available in #2930. We are
> going
> > > to merge that today.
> > >
> > > – Ufuk
> > >
> > > On 3 December 2016 at 09:56:21, Gyula Fóra (gyula.f...@gmail.com)
> wrote:
> > > > Hi,
> > > >
> > > > I think something is not behaving correctly with savepoint/checkpoint
> > > > discard on the release-1.1 branch. If I build the release-1.1.4-rc
> branch
> > > > everything works correctly but on the latest 1.1 branch it seems like
> > > > savepoints are discarded like checkpoints and cannot be restored
> after
> > > the
> > > > job is stopped.
> > > >
> > > > I can take a savepoints, I see it complete. But when I try to restore
> > > from
> > > > it doesnt find the checkpoint folder anymore. I know that there are
> tests
> > > > for this behaviour but it still seems to happen when I run it on
> yarn.
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > >
> > >
> >
>
>

Re: Savepoints seem to not work properly on release-1.1 branch

[Ufuk Celebi]

Thanks. It has been merged now. :-)

On 4 December 2016 at 17:24:30, Gyula Fóra (gyula.f...@gmail.com) wrote:
> Hi,
> Thanks for being so quick, I will try the fix as soon as it is merged. :)
> Gyula
>  
> Ufuk Celebi ezt írta (időpont: 2016. dec. 4., V, 12:02):
>  
> > Thanks for reporting this Gyula. A fix is available in #2930. We are going
> > to merge that today.
> >
> > – Ufuk
> >
> > On 3 December 2016 at 09:56:21, Gyula Fóra (gyula.f...@gmail.com) wrote:
> > > Hi,
> > >
> > > I think something is not behaving correctly with savepoint/checkpoint
> > > discard on the release-1.1 branch. If I build the release-1.1.4-rc branch
> > > everything works correctly but on the latest 1.1 branch it seems like
> > > savepoints are discarded like checkpoints and cannot be restored after
> > the
> > > job is stopped.
> > >
> > > I can take a savepoints, I see it complete. But when I try to restore
> > from
> > > it doesnt find the checkpoint folder anymore. I know that there are tests
> > > for this behaviour but it still seems to happen when I run it on yarn.
> > >
> > > Cheers,
> > > Gyula
> > >
> >
> >
>  

Re: Savepoints seem to not work properly on release-1.1 branch

[Gyula Fóra]

Hi,
Thanks for being so quick, I will try the fix as soon as it is merged. :)
Gyula

Ufuk Celebi  ezt írta (időpont: 2016. dec. 4., V, 12:02):

> Thanks for reporting this Gyula. A fix is available in #2930. We are going
> to merge that today.
>
> – Ufuk
>
> On 3 December 2016 at 09:56:21, Gyula Fóra (gyula.f...@gmail.com) wrote:
> > Hi,
> >
> > I think something is not behaving correctly with savepoint/checkpoint
> > discard on the release-1.1 branch. If I build the release-1.1.4-rc branch
> > everything works correctly but on the latest 1.1 branch it seems like
> > savepoints are discarded like checkpoints and cannot be restored after
> the
> > job is stopped.
> >
> > I can take a savepoints, I see it complete. But when I try to restore
> from
> > it doesnt find the checkpoint folder anymore. I know that there are tests
> > for this behaviour but it still seems to happen when I run it on yarn.
> >
> > Cheers,
> > Gyula
> >
>
>

[jira] [Created] (FLINK-5249) description of datastream rescaling doesn't match the figure

[David Anderson (JIRA)]

David Anderson created FLINK-5249:
-

 Summary: description of datastream rescaling doesn't match the 
figure
 Key: FLINK-5249
 URL: https://issues.apache.org/jira/browse/FLINK-5249
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: David Anderson
Priority: Minor


In 
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html#physical-partitioning
 the figure shows a parallelism of 2 rescaling to 6 and back to 2 -- while the 
text assumes 4, rather than 6. This makes the explanation very confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5248) SavepointITCase doesn't catch savepoint restore failure

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-5248:
--

 Summary: SavepointITCase doesn't catch savepoint restore failure
 Key: FLINK-5248
 URL: https://issues.apache.org/jira/browse/FLINK-5248
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.1.3
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi
Priority: Critical


The savepoint IT case does not fail whem restoring from a savepoints fails on 
the task side. It only checks that the deployment descriptors are as expected 
when resubmitting. This is a pretty bad shortcoming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Savepoints seem to not work properly on release-1.1 branch

[Ufuk Celebi]

Thanks for reporting this Gyula. A fix is available in #2930. We are going to 
merge that today.

– Ufuk

On 3 December 2016 at 09:56:21, Gyula Fóra (gyula.f...@gmail.com) wrote:
> Hi,
>  
> I think something is not behaving correctly with savepoint/checkpoint
> discard on the release-1.1 branch. If I build the release-1.1.4-rc branch
> everything works correctly but on the latest 1.1 branch it seems like
> savepoints are discarded like checkpoints and cannot be restored after the
> job is stopped.
>  
> I can take a savepoints, I see it complete. But when I try to restore from
> it doesnt find the checkpoint folder anymore. I know that there are tests
> for this behaviour but it still seems to happen when I run it on yarn.
>  
> Cheers,
> Gyula
>