[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1428


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-179926670
  
I went through all the classes in the relevant modules outside `flink-core` 
with @fhueske and annotated them properly.
I've rebased the changes and checked with the additions in between.

I'm going to merge this PR in the next ~12 hours if nobody objects.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-173511286
  
Cool, thank you.
I'll rebase the PR, and go over the APIs again as well


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-170333018
  
Thanks for the update! 
I'd like to do another pass over the DataSet Java and Scala classes next 
week.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-169335005
  
Thanks a lot for looking into this!

In the `ExecutionConfig` I've made the following methods experimental:
- setAutoWatermarkInterval
- enableTimestamps
- disableTimestamps
- areTimestampsEnabled
- getAutoWatermarkInterval
- setCodeAnalysisMode
- getCodeAnalysisMode

Regarding the `ExecutionMode`, I'll bring the issue to the mailing list.

`JobExecutionResult.getIntCounterResult()` marked Experimental and 
Deprecated

I also remember that there was an issue like this with the `Partitioner`. 
I'll also bring this to the mailing list.

I removed the `@Public` annotation from `DefaultInputSplitAssigner`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-165848516
  
Do we want to make the IOFormats Public as well?
Most of them are hidden by helper methods of the `ExecutionEnvironment`. 
However, they can also be used as regular input formats via 
`ExecutionEnvironment.createInput()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-165171199
  
Hi,

I had a look and have some comments as well

- `ExecutionConfig`: Do we want to make the whole class public? Are we sure 
that we do not change the configuration of certain streaming features such as 
`enableTimestamps()`, `disableTimestamps()`, `setAutoWatermarkInterval()`? I 
find these options in combination with `TimeCharacteristics` very confusing and 
I'm not sure if we should mark them already stable. I talked to @aljoscha and 
he's not satisfied with the current implementation as well.
- `ExecutionMode`: IMO, the name is quite generic for it's purpose. The 
mode is is only relevant for network shuffles in the batch programs.
- `JobExecutionResult.getIntCounterResult()`: Do we need and want to keep 
this method? IMO, it should be rather deprecated than marked as stable. 
- `Partitioner`: Does it make sense to extend the `partition(K key, int 
numPartition)` method by a paramter for the current partitionId? I remember I 
had some use cases for this, but forgot about the details.

We need to carefully check the class hierarchy of public/stable classes. 
For example, if we make `DefaultInputSplitAssigner` a `Public` interface, we 
don't need to make `InputSplitAssigner` `Experimental` because we cannot touch 
it then. There might be more of these cases in this PR (and PR #1428). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164762014
  
Thank you all for the comments.
I'll rename the interfaces to `@Public` and `@Experimental`.

I'll keep the annotation on the `ConfigConstants` class to ensure that we 
are not breaking the configuration parameters.
The tool will detect changes to the variable names, not to the values.
Adding @Deprecated annotations is not a breaking change, so we can rename 
configs by adding the new name and Deprecating the old one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164764484
  
I didn't make `MemorySegment` a public API. Lets declare it in the pull 
request as stable.
I decided to mark `XORShiftRandom` and `NumberSequenceIterator` as stable 
APIs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164768159
  
> Are inner classes of a class tagged as PublicInterface also marked as 
public/stable?

Yes!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164777618
  
I addressed all issues you've mentioned and rebased the code to the current 
master. I would like to merge this soon!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-164781638
  
Thank you for your patch. I've added it to the PR.

Regarding your question: I would keep name changes separate from this PR. 
Once this one is merged, we can start changing names.


Once #1427 is merged, I'll rebase this PR and we can finalize it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[hsaputra]

Github user hsaputra commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164665117
  
Since this is primarily used to tag Interface or methods, could we just 
call the annotations as Public and Experimental rather than PublicInterface and 
PublicExperimental ? 

Seemed redundant to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[StephanEwen]

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-164288665
  
Thanks for the initiative, Robert. I had a look and here are some thoughts:

### General Comments
  
  - The interface declarations should declare their targets:
- @PublicInterface should have `@Target(ElementType.TYPE)`
- @PublicExperimental should have `@Target({ ElementType.TYPE, 
ElementType.METHOD })`

  - How about shortening the name `@PublicInterface` to `@Public` or 
`@PublicAPI`


### Stability decisions to make before 1.0

  - We need to think about whether we change the RichGroupReduceFunction to 
implement the regular combine function (combining into one value) rather than 
the group combine function. While API breaking, it allows for more efficient 
pre-aggregations.


### Classes that I suggest not to declare public

  - `Key` (I would suggest to entirely remove that class, separate Pull 
Requests)
  
  - `Nothing` (May even remove that and replace it by `Void`)
  
  - `InputSplitAssigner` : I think we need to change that one a bit to 
support more fine grained fault tolerance for data sources.
  
  - `ConfigConstants` : Not sure if we should encourage people to use this 
class.
 
  - `AccumulatorHelper` : No need to expose this, and might change.
  
  - `IntCounter`, `LongCounter`, `DoubleCounter` : These are actually 
"sums", not couners. If we declare them stable, we should at least fix their 
wrong name ;-)
  
  - Anything under `org.apache.flink.api.common.aggregators`. Would be nice 
to have aggregators unified with accumulators in the long run.
  
  - For the same reason, we need to mark the Aggregator releated methods in 
`IterationRuntimeContext` as experimental.
  
  - `StrictlyLocalAssignment` : This is still experimental, we have not yet 
had users of that and a chance to see it validated.
  
  - `SemanticProperties` could be declared public experimental.
  
  - `TypeSerializer` and `TypeComparator` : These are not touched by users 
usually, and we may want to change them a lot depending on future designs for 
serialization improvements and intermediate data layout.


### The following classes should be declared as stable, in my opinion:

  - `GenericInputSplit` : Part of the publis stable `GenericInputFormat`
  
  - `SplittableIterator` : This is an exposed class for parallel collection 
sources.
  
  - `TraversableOnceException` : Simple (and virtually empty) exception 
class thrown into user code by the runtime.
 

### Classes that are good candidates for public API:

  - `NumberSequenceIterator` : This is a good and stable utility class.
  
  - `XORShiftRandom` : This is a good and stable utility class.
  
  - `MemorySegment` : If in the future, we allow UDFs to reserve and use 
managed memory (as suggested by Chengxliang Li's pull request, this will need 
to be declared stable later anyways)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[aljoscha]

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/1427#discussion_r46833253
  
--- Diff: 
flink-annotations/src/main/java/org/apache/flink/annotation/PublicExperimental.java
 ---
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.flink.annotation;
+
+import java.lang.annotation.Documented;
+
+/**
+ * Interface to mark methods within stable, public APIs as experimental.
--- End diff --

It is also used to mark classes/interfaces, not just methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1427#issuecomment-162559507
  
Are inner classes of a class tagged as `PublicInterface` also marked as 
public/stable?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-162557821
  
Should we also change names of stuff or open separate Jiras, PRs. I'm 
asking, for example, because of `CheckpointNotifier` which should be called 
`CheckpointListener` or something like it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1428#issuecomment-162578546
  
I created a patch of changes in the Stream API that I would propose:


[stream-api-annotations.patch.txt](https://github.com/apache/flink/files/54231/stream-api-annotations.patch.txt)

(you have to remove the .txt extension, github would not let me upload 
without it...)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on a diff in the pull request:

https://github.com/apache/flink/pull/1427#discussion_r46756747
  
--- Diff: flink-core/src/main/java/org/apache/flink/types/Pair.java ---
@@ -33,6 +34,7 @@
  * @param  Type of the pair's first element.
  * @param  Type of the pair's second element.
  */
+@PublicInterface
 public abstract class Pair, V extends Key> implements 
Key> {
--- End diff --

Most likely, yes. I've asked Stephan in the JIRA just to make sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1426#issuecomment-161045370
  
Hi @rmetzger, thanks for taking the initiative for this issue!
I tried to review the PR, but the diff set it too large for the Github 
viewer. I know, I could checkout the code and do the diffs on my machine, but I 
hope I am not the only one reviewing the changes.

Would you mind splitting up the PR into a let's say four smaller ones for 
Core, DataSet, DataStream, and Connectors (any other reasonable split is fine 
with me as well)?
That would really ease reviewing the changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger closed the pull request at:

https://github.com/apache/flink/pull/1426


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1427#discussion_r46323631
  
--- Diff: flink-core/src/main/java/org/apache/flink/types/Pair.java ---
@@ -33,6 +34,7 @@
  * @param  Type of the pair's first element.
  * @param  Type of the pair's second element.
  */
+@PublicInterface
 public abstract class Pair, V extends Key> implements 
Key> {
--- End diff --

is this the class mentioned in FLINK-2344? 
https://issues.apache.org/jira/browse/FLINK-2344


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1427

[FLINK-3093] Introduce annotations for interface stability in flink-core

Please see https://github.com/apache/flink/pull/1426

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink 
interface_stability_no_maven-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1427


commit a08ee41a81c9417975c789b7dcccda63840bcb93
Author: Robert Metzger 
Date:   2015-12-01T18:32:00Z

[FLINK-3093] Introduce annotations for interface stability in flink-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1428

[FLINK-3093] Introduce annotations for interface stability in remaining 
modules

Please see https://github.com/apache/flink/pull/1426

This pull request depends on https://github.com/apache/flink/pull/1427

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink 
interface_stability_no_maven-rest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1428.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1428


commit a088b1e62cebf0d044cf2a276a475dd7519ca01b
Author: Robert Metzger 
Date:   2015-12-01T18:32:30Z

[FLINK-3093] Introduce annotations for interface stability in remaining 
modules




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1426#issuecomment-161056518
  
Okay, I'll reopen two new PRs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3093] Introduce annotations for interfa...

[rmetzger]

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1426

[FLINK-3093] Introduce annotations for interface stability

With this pull request, I'm introducing annotations for marking the 
stability of certain Flink interfaces.

The annotations are in the `flink-annotations` module. I needed to create a 
new module because the maven plugin I'm intending to use needs to have access 
to the annotations at runtime.
My first idea was to add them to `flink-core`, but that lead to circular 
dependencies.

Speaking of maven plugins: I did not yet include the infrastructure to 
enforce interface stability. I'm currently trying to find a maven plugin which 
will allow us to automatically verify the stability.
The problem is that the plugin needs some additional features until we can 
use it. But I hope we get it in place with the 1.0.0 release.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink interface_stability_no_maven

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1426


commit 7c23d762aabafdb2ed46aaafd306d474751624ff
Author: Robert Metzger 
Date:   2015-11-10T19:46:07Z

[FLINK-3093] Introduce annotations for interface stability




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---