[jira] [Created] (FLINK-32025) Make job cancellation button on UI configurable

[Ted Yu (Jira)]

Ted Yu created FLINK-32025:
--

 Summary: Make job cancellation button on UI configurable
 Key: FLINK-32025
 URL: https://issues.apache.org/jira/browse/FLINK-32025
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


On the flink job UI, there is `Cancel Job` button.

When the job UI is shown to users, it is desirable to hide the button so that 
normal user doesn't mistakenly cancel a long running flink job.

This issue adds configuration for hiding the `Cancel Job` button.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor

[Ted Yu (JIRA)]

Ted Yu created FLINK-10468:
--

 Summary: Potential missing break for PARTITION_CUSTOM in 
OutputEmitter ctor
 Key: FLINK-10468
 URL: https://issues.apache.org/jira/browse/FLINK-10468
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
switch (strategy) {
case PARTITION_CUSTOM:
  extractedKeys = new Object[1];
case FORWARD:
{code}
It seems a 'break' is missing prior to FORWARD case.
{code}
if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) {
  throw new NullPointerException("Partitioner must not be null when the 
ship strategy is set to custom partitioning.");
}
{code}
Since the above check is for PARTITION_CUSTOM, it seems we can place the check 
in the switch statement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10467) Upgrade commons-compress to 1.18

[Ted Yu (JIRA)]

Ted Yu created FLINK-10467:
--

 Summary: Upgrade commons-compress to 1.18
 Key: FLINK-10467
 URL: https://issues.apache.org/jira/browse/FLINK-10467
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


org.apache.commons:commons-compress defines an API for working with compression 
and archive formats.

Affected versions of this package are vulnerable to Directory Traversal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API

[Ted Yu (JIRA)]

Ted Yu created FLINK-10446:
--

 Summary: Use the "guava beta checker" plugin to keep off of @Beta 
API
 Key: FLINK-10446
 URL: https://issues.apache.org/jira/browse/FLINK-10446
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


The Guava people publish an Error Prone plugin to detect when stuff that's 
annotated with @Beta gets used. Those things shouldn't be used because the 
project gives no promises about deprecating before removal.

plugin:

https://github.com/google/guava-beta-checker



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10389) TaskManagerServicesConfiguration ctor contains self assignment

[Ted Yu (JIRA)]

Ted Yu created FLINK-10389:
--

 Summary: TaskManagerServicesConfiguration ctor contains self 
assignment
 Key: FLINK-10389
 URL: https://issues.apache.org/jira/browse/FLINK-10389
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


TaskManagerServicesConfiguration has:
{code}
this.systemResourceMetricsEnabled = systemResourceMetricsEnabled;
{code}
There is no systemResourceMetricsEnabled parameter to the ctor.

This was reported by findbugs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10388) RestClientTest sometimes fails with AssertionError

[Ted Yu (JIRA)]

Ted Yu created FLINK-10388:
--

 Summary: RestClientTest sometimes fails with AssertionError
 Key: FLINK-10388
 URL: https://issues.apache.org/jira/browse/FLINK-10388
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu


Running the test on Linux I got:
{code}
testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest)  Time 
elapsed: 1.918 sec  <<< FAILURE!
java.lang.AssertionError:
Expected: an instance of 
org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
 but: 
 is a 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at 
org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10228) Add metrics for netty direct memory consumption

[Ted Yu (JIRA)]

Ted Yu created FLINK-10228:
--

 Summary: Add metrics for netty direct memory consumption
 Key: FLINK-10228
 URL: https://issues.apache.org/jira/browse/FLINK-10228
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


netty direct memory usage can be exposed via metrics so that operator can keep 
track of memory consumption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10125) Unclosed ByteArrayDataOutputView in RocksDBMapState

[Ted Yu (JIRA)]

Ted Yu created FLINK-10125:
--

 Summary: Unclosed ByteArrayDataOutputView in RocksDBMapState
 Key: FLINK-10125
 URL: https://issues.apache.org/jira/browse/FLINK-10125
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
  ByteArrayDataOutputView dov = new ByteArrayDataOutputView(1);
{code}
dov is used in a try block but it is not closed in case of Exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9924) Upgrade zookeeper to 3.4.13

[Ted Yu (JIRA)]

Ted Yu created FLINK-9924:
-

 Summary: Upgrade zookeeper to 3.4.13
 Key: FLINK-9924
 URL: https://issues.apache.org/jira/browse/FLINK-9924
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


zookeeper 3.4.13 is being released.

ZOOKEEPER-2959 fixes data loss when observer is used
ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / 
cloud)
environment



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9880) Incorrect argument order calling BucketerContext#update

[Ted Yu (JIRA)]

Ted Yu created FLINK-9880:
-

 Summary: Incorrect argument order calling BucketerContext#update
 Key: FLINK-9880
 URL: https://issues.apache.org/jira/browse/FLINK-9880
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


In StreamingFileSink.java :
{code}
bucketerContext.update(context.timestamp(), currentProcessingTime, 
context.currentWatermark());
{code}
However, the method update is declared as :
{code}
void update(@Nullable Long elementTimestamp, long currentWatermark, long 
currentProcessingTime) {
{code}
The second and third parameters seem to be swapped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9849) Upgrade hbase version to 2.0.1 for hbase connector

[Ted Yu (JIRA)]

Ted Yu created FLINK-9849:
-

 Summary: Upgrade hbase version to 2.0.1 for hbase connector
 Key: FLINK-9849
 URL: https://issues.apache.org/jira/browse/FLINK-9849
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Currently hbase 1.4.3 is used for hbase connector.

We should upgrade to 2.0.1 which was recently released.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9825) Upgrade checkstyle version to 8.6

[Ted Yu (JIRA)]

Ted Yu created FLINK-9825:
-

 Summary: Upgrade checkstyle version to 8.6
 Key: FLINK-9825
 URL: https://issues.apache.org/jira/browse/FLINK-9825
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


We should upgrade checkstyle version to 8.6+ so that we can use the "match 
violation message to this regex" feature for suppression. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9824) Support IPv6 literal

[Ted Yu (JIRA)]

Ted Yu created FLINK-9824:
-

 Summary: Support IPv6 literal
 Key: FLINK-9824
 URL: https://issues.apache.org/jira/browse/FLINK-9824
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Currently we use colon as separator when parsing host and port.

We should support the usage of IPv6 literals in parsing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.5.1, release candidate #2

[Ted Yu]

+1

Checked signatures of artifacts

Ran test suite

On Fri, Jul 6, 2018 at 11:42 AM Yaz Sh  wrote:

> +1
>
> Tests have been done on OS X
>
> - Ran in cluster mode ./bin/start-cluster.sh
> - Checked that *.out files are empty
> - Stopped cluster ./bin/stop-cluster.sh
> - Checked for No Exceptions on log output
>
> - Test Examples via WebUI
> - Test Example via CLI with different flags (-d, -c, -q, -p)
>
> - Added more task Managers via flink-config.yml and re-ran the Examples
> - Added more task Manager via ./bin/taskmanager.sh and re-ran the Examples
>
> - Checked “checksum” for all packages
> - Checked GPG signature for all packages
> - Checked the README.md
>
>
> Cheers,
> Yazdan
>
>
> > On Jul 6, 2018, at 11:09 AM, Chesnay Schepler 
> wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #2 for the version
> 1.5.1, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 11D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.5.1-rc2" [5],
> > * website pull request listing the new release and adding announcement
> blog post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12343053
> > [2] https://dist.apache.org/repos/dist/dev/flink/1.5.1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1170
> > [5]
> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=refs/tags/release-1.5.1-rc2
> > [6] https://github.com/apache/flink-web/pull/112
> >
> >
> >
>
>

Re: Flink Kafka TimeoutException

[Ted Yu]

Have you tried increasing the request.timeout.ms parameter (Kafka) ?

Which Flink / Kafka release are you using ?

Cheers

On Thu, Jul 5, 2018 at 5:39 AM Amol S - iProgrammer 
wrote:

> Hello,
>
> I am using flink with kafka and getting below exception.
>
> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for
> helloworld.t-7: 30525 ms has passed since last append
>
> ---
> *Amol Suryawanshi*
> Java Developer
> am...@iprogrammer.com
>
>
> *iProgrammer Solutions Pvt. Ltd.*
>
>
>
> *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society,
> Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016,
> MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer*
> www.iprogrammer.com 
> 
>

[jira] [Created] (FLINK-9736) Potential null reference in KeyGroupPartitionedPriorityQueue#poll()

[Ted Yu (JIRA)]

Ted Yu created FLINK-9736:
-

 Summary: Potential null reference in 
KeyGroupPartitionedPriorityQueue#poll()
 Key: FLINK-9736
 URL: https://issues.apache.org/jira/browse/FLINK-9736
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
final PQ headList = heapOfkeyGroupedHeaps.peek();
final T head = headList.poll();
{code}
{{peek}} call may return null.
The return value should be checked before de-referencing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9735) Potential resource leak in RocksDBStateBackend#getDbOptions

[Ted Yu (JIRA)]

Ted Yu created FLINK-9735:
-

 Summary: Potential resource leak in 
RocksDBStateBackend#getDbOptions
 Key: FLINK-9735
 URL: https://issues.apache.org/jira/browse/FLINK-9735
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
if (optionsFactory != null) {
  opt = optionsFactory.createDBOptions(opt);
}
{code}
opt, an DBOptions instance, should be closed before being rewritten.

getColumnOptions has similar issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Job jumbo jar can't be extracted on Windows

[Ted Yu]

Looking at javadoc of separatorChar :

 * The system-dependent default name-separator character.  This field is

 * initialized to contain the first character of the value of the system

 * property file.separator.  On UNIX systems the value of
this

 * field is '/'; on Microsoft Windows systems it is 
'\\'.

Looks like the code you quoted should get the correct separator.

On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI  wrote:

> When I try to "flink run" a job jar that includes dependent jar's - it
> fails on Windows with the following Exception:
>
> org.apache.flink.client.program.ProgramInvocationException: Unknown I/O
> error while extracting contained jar files.
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:752)
> at org.apache.flink.client.program.PackagedProgram.
> (PackagedProgram.java:194)
> at org.apache.flink.client.cli.CliFrontend.buildProgram(
> CliFrontend.java:833)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
> java:201)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(
> CliFrontend.java:1020)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
> CliFrontend.java:1096)
> at org.apache.flink.runtime.security.NoOpSecurityContext.
> runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
> java:1096)
> Caused by: org.apache.flink.client.program.ProgramInvocationException: An
> I/O error occurred while creating temporary file to extract nested library
> 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:708)
> ... 7 more
> Caused by: java.io.IOException: Unable to create temporary file,
> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\flink-
> connector-kafka-0.10_2.11-1.4.1.jar
> at java.io.File$TempDirectory.generateFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:702)
> ... 7 more
>
> I think we have a problem in PackagedProgram.java - it fails to extract
> contained jar's on Windows, because on Windows File.separatorChar is '\',
> but JarEntry.getName() returns '/' as file separator.
> Line 699 of https://github.com/apache/flink/blob/master/flink-
> clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
> is no-op on Windows.
>
> I think we need to replace line 699 with
> name = name.replace('/', '_');
> as I don't see a way to have JarEntry use platform-specific file
> separator, and ZIP spec dictates the use of forward-slashes anyway.
> Line 680 already uses hard-coded '/'.
>
>
> Alex Nekrassov
>

[jira] [Created] (FLINK-9697) Provide connector for Kafka 2.0.0

[Ted Yu (JIRA)]

Ted Yu created FLINK-9697:
-

 Summary: Provide connector for Kafka 2.0.0
 Key: FLINK-9697
 URL: https://issues.apache.org/jira/browse/FLINK-9697
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Kafka 2.0.0 would be released soon.

Here is vote thread:

http://search-hadoop.com/m/Kafka/uyzND1vxnEd23QLxb?subj=+VOTE+2+0+0+RC1

We should provide connector for Kafka 2.0.0 once it is released.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9675) Avoid FileInputStream/FileOutputStream

[Ted Yu (JIRA)]

Ted Yu created FLINK-9675:
-

 Summary: Avoid FileInputStream/FileOutputStream
 Key: FLINK-9675
 URL: https://issues.apache.org/jira/browse/FLINK-9675
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


They rely on finalizers (before Java 11), which create unnecessary GC load. The 
alternatives, Files.newInputStream, are as easy to use and don't have this 
issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Flink-9407] Question about proposed ORC Sink !

[Ted Yu]

For #1, the word exist should be exit, right ?Thanks
 Original message From: zhangminglei <18717838...@163.com> 
Date: 6/23/18  10:12 AM  (GMT+08:00) To: sagar loke  Cc: 
dev , user  Subject: Re: 
[Flink-9407] Question about proposed ORC Sink ! 
Hi, Sagar.
1. It solves the issue partially meaning files which have finished 
checkpointing don't show .pending status but the files which were in progress   
  when the program exists are still in .pending state.
Ans: 
Yea, Make the program exists and in that time if a checkpoint does not finished 
will lead the status keeps in .pending state then. Under the normal 
circumstances, the programs that running in the production env will never be 
stoped or existed if everything is fine.
2. Ideally, writer should work with default settings correct ? Meaning we don't 
have to explicitly set these parameters to make it work.     Is this assumption 
correct ?
Ans: 
Yes. Writer should work with default settings correct.Yes. We do not have to 
explicitly set these parameters to make it work.Yes. Assumption correct indeed.
However, you know, flink is a real time streaming framework, so under normal 
circumstances,you don't really go to use the default settings when it comes to 
a specific business. Especially together work with offline end(Like hadoop 
mapreduce). In this case, you need to tell the offline end when time a bucket 
is close and when time the data for the specify bucket is ready. So, you can 
take a look on https://issues.apache.org/jira/browse/FLINK-9609.
CheersZhangminglei

在 2018年6月23日，上午8:23，sagar loke  写道：
Hi Zhangminglei,
Thanks for the reply.
1. It solves the issue partially meaning files which have finished 
checkpointing don't show .pending status but the files which were in progress   
  when the program exists are still in .pending state.
2. Ideally, writer should work with default settings correct ? Meaning we don't 
have to explicitly set these parameters to make it work.     Is this assumption 
correct ?
Thanks,Sagar
On Fri, Jun 22, 2018 at 3:25 AM, zhangminglei <18717838...@163.com> wrote:
Hi, Sagar. Please use the below code and you will find the part files status 
from _part-0-107.in-progress   to _part-0-107.pending and finally to 
part-0-107. [For example], you need to run the program for a while. However, we 
need set some parameters, like the following. Moreover, enableCheckpointing IS 
also needed. I know why you always see the .pending file since the below 
parameters default value is 60 seconds even though you set the 
enableCheckpoint. So, that is why you can not see the finished file status 
until 60 seconds passed.
Attached is the ending on my end, and you will see what you want! 
Please let me know if you still have the problem.
CheersZhangminglei
setInactiveBucketCheckInterval(2000)
.setInactiveBucketThreshold(2000);
public class TestOrc {
   public static void main(String[] args) throws Exception {
  StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
  env.setParallelism(1);
  env.enableCheckpointing(1000);
  env.setStateBackend(new MemoryStateBackend());

  String orcSchemaString = "struct";
  String path = "hdfs://10.199.196.0:9000/data/hive/man";

  BucketingSink bucketingSink = new BucketingSink<>(path);

  bucketingSink
 .setWriter(new OrcFileWriter<>(orcSchemaString))
 .setInactiveBucketCheckInterval(2000)
 .setInactiveBucketThreshold(2000);

  DataStream dataStream = env.addSource(new ManGenerator());

  dataStream.addSink(bucketingSink);

  env.execute();
   }

   public static class ManGenerator implements SourceFunction {

  @Override
  public void run(SourceContext ctx) throws Exception {
 for (int i = 0; i < 2147483000; i++) {
Row row = new Row(3);
row.setField(0, "Sagar");
row.setField(1, 26 + i);
row.setField(2, false);
ctx.collect(row);
 }
  }

  @Override
  public void cancel() {

  }
   }
}


在 2018年6月22日，上午11:14，sagar loke  写道：
Sure, we can solve it together :)
Are you able to reproduce it ?
Thanks,Sagar
On Thu, Jun 21, 2018 at 7:28 PM zhangminglei <18717838...@163.com> wrote:
Sagar, flush will be called when do a checkpoint. Please see 
bucketState.currentFileValidLength = bucketState.writer.flush();

@Override
public void snapshotState(FunctionSnapshotContext context) throws Exception {
   Preconditions.checkNotNull(restoredBucketStates, "The operator has not been 
properly initialized.");

   restoredBucketStates.clear();

   synchronized (state.bucketStates) {
  int subtaskIdx = getRuntimeContext().getIndexOfThisSubtask();

  for (Map.Entry> bucketStateEntry : 
state.bucketStates.entrySet()) {
 BucketState bucketState = bucketStateEntry.getValue();

 if (bucketState.isWriterOpen) {
bucketState.currentFileValidLength = bucketState.writer.flush();
 }

Re: [ANNOUNCE] New committer Piotr Nowojski

[Ted Yu]

Congratulations, Piotr!

On Fri, Jun 22, 2018 at 12:26 PM, Till Rohrmann 
wrote:

> Hi everybody,
>
> On behalf of the PMC I am delighted to announce Piotr Nowojski as a new
> Flink
> committer!
>
> Piotr has been an active member of our community for more than a year.
> Among other things, he contributed the TwoPhaseCommitSink, worked
> extensively on improving Flink's network stack and is now contributing to
> stream SQL. He is also helping the community by reviewing PRs, answering
> questions and driving discussions on the mailing list.
>
> Please join me in congratulating Piotr for becoming a Flink committer!
>
> Cheers,
> Till
>

Re: [ANNOUNCE] New committer: Sihua Zhou

[Ted Yu]

Congratulations Sihua!

On Fri, Jun 22, 2018 at 6:42 AM, zhangminglei <18717838...@163.com> wrote:

> Congrats! Sihua
>
> Cheers
> Minglei.
>
> > 在 2018年6月22日，下午9:17，Till Rohrmann  写道：
> >
> > Hi everybody,
> >
> > On behalf of the PMC I am delighted to announce Sihua Zhou as a new Flink
> > committer!
> >
> > Sihua has been an active member of our community for several months.
> Among
> > other things, he helped developing Flip-6, improved Flink's state
> backends
> > and fixed a lot of major and minor issues. Moreover, he is helping the
> > Flink community reviewing PRs, answering users on the mailing list and
> > proposing new features.
> >
> > Please join me in congratulating Sihua for becoming a Flink committer!
> >
> > Cheers,
> > Till
>
>
>

Re: Please review FLINK-9610 Add Kafka partitioner that uses the key to partition by

[Ted Yu]

Interesting enhancement.

I left a minor comment on the PR.

Cheers

On Tue, Jun 19, 2018 at 12:26 AM, Niels Basjes  wrote:

> Hi,
>
> Yesterday we ran into problems regarding the distribution of records across
> Kafka where Flink was used as the producer. So we fixed this and realized
> that the code to do this would be useful to others.
>
> I put up a Jira ticket and pull request yesterday and it passes all
> automated tests.
> Please review.
>
> https://issues.apache.org/jira/browse/FLINK-9610
> https://github.com/apache/flink/pull/6181
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

[jira] [Created] (FLINK-9580) Potentially unclosed ByteBufInputStream in RestClient#readRawResponse

[Ted Yu (JIRA)]

Ted Yu created FLINK-9580:
-

 Summary: Potentially unclosed ByteBufInputStream in 
RestClient#readRawResponse
 Key: FLINK-9580
 URL: https://issues.apache.org/jira/browse/FLINK-9580
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
  ByteBufInputStream in = new ByteBufInputStream(content);
  byte[] data = new byte[in.available()];
  in.readFully(data);
{code}
In the catch block, ByteBufInputStream is not closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: how to build the connectors and examples from the source

[Ted Yu]

Which connector from the following list are you trying to build ?

https://flink.apache.org/ecosystem.html#connectors

The available connectors from 1.5.0 are quite recent. Is there any
functionality missing in the 1.5.0 release ?

Thanks

On Tue, Jun 12, 2018 at 5:17 PM, Chris Kellogg  wrote:

> How can one build a connectors jar from the source?
>
> Also, is there a quick way to build the examples from the source without
> having to do a mvn clean package -DskipTests?
>
>
> Thanks.
> Chris
>

Re: Static code analysis for Flink project

[Ted Yu]

I took a look at some of the blocker defects.
e.g.
https://sonarcloud.io/project/issues?id=org.apache.flink%3Aflink-parent=AWPxETxA3e-qcckj1Sl1=false=BLOCKER=BUG

For 
./flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java
, the closing of DBOptions using try-with-resources is categorized as
blocker by the analysis.

I don't think that categorization is proper.

We can locate the high priority defects, according to consensus, and fix
those.

Cheers

On Tue, Jun 12, 2018 at 2:01 PM,  wrote:

> Hello Flink community.
>
> I am new in Flink project and probably don't understand it a lot. Could
> you please clarify one question to me?
>
> I download Flink sources and build it from scratch. I found checkstyle
> guidelines that every Flink developer should follow which is very useful.
> However, I didn't find anything about static analysis tools like Sonarcube.
> I have looked through mailing lists archive but without success. That
> seemed very strange to me.
>
> I have setup Sonarcube and run analysis on whole Flink project. After a
> while I have got 442 bugs, 511 vulnerabilities and more than 13K Code
> Smells issues. You can see them all here: https://sonarcloud.io/
> dashboard?id=org.apache.flink%3Aflink-parent
>
> I looked through some of bugs and vulnerabilities and there are many
> important ones (in my opinions) like these:
> - 'other' is dereferenced. A "NullPointerException" could be thrown;
> "other" is nullable here.
> - Either re-interrupt this method or rethrow the "InterruptedException".
> - Move this call to "wait()" into a synchronized block to be sure the
> monitor on "Object" is held.
> - Refactor this code so that the Iterator supports multiple traversal
> - Use try-with-resources or close this "JsonGenerator" in a "finally"
> clause. Use try-with-resources or close this "JsonGenerator" in a "finally"
> clause.
> - Cast one of the operands of this subtraction operation to a "long".
> - Make "ZERO_CALENDAR" an instance variable.
> - Add a "NoSuchElementException" for iteration beyond the end of the
> collection.
> - Replace the call to "Thread.sleep(...)" with a call to "wait(...)".
> - Call "Optional#isPresent()" before accessing the value.
> - Change this condition so that it does not always evaluate to "false".
> Expression is always false.
> - This class overrides "equals()" and should therefore also override
> "hashCode()".
> - "equals(Object obj)" should test argument type
> - Not enough arguments in LOG.debug function. Not enough arguments.
> - Remove this return statement from this finally block.
> - "notify" may not wake up the appropriate thread.
> - Remove the boxing to "Double".
> - Classes should not be compared by name
> - "buffers" is a method parameter, and should not be used for
> synchronization.
>
> Are there any plans to work on static analysis support for Flink project
> or it was intentionally agreed do not use static analysis as time consuming
> and worthless?
>
> Thank you in advance for you replies.
>
> Best Regards,
> ---
> Alex Arkhipov
>
>

Re: [WEBSITE] Proposal to rework the Flink website

[Ted Yu]

+1

On Tue, Jun 5, 2018 at 9:46 AM, Stephan Ewen  wrote:

> Big +1 to this!
>
> I would like to contribute to this effort and help strengthen the way Flink
> presents itself.
>
>
> On Tue, Jun 5, 2018 at 11:56 AM, Fabian Hueske  wrote:
>
> > Hi everybody,
> >
> > I've opened a PR [1] that reworks parts of the Flink website (
> > flink.apache.org).
> >
> > My goal is to improve the structure of the website and provide more
> > valuable information about the project and the community.
> >
> > A visitor (who doesn't know Flink yet) should be able to easily find
> > answers to the following questions:
> > * What is Apache Flink?
> > * Does it address my use case?
> > * Is it credible? / Who is using it?
> >
> > To achieve that, I have:
> > * Rework menu structure into three sections to address different
> audiences:
> >   - Potential users (see above)
> >   - Users
> >   - Contributors
> > * Reworked start page: updated the figure, added a feature grid, moved
> > "Powered By" section up
> > * Replaced Features page by more detailed "What is Flink?" pages
> > * Reworked "Use Cases" page
> >
> > The PR should also improve the page for users who have questions about
> > Flink or need help.
> > For that, I have:
> > * Added a "Getting Help" page (less content than the detailed community
> > page)
> > * Removed IRC channel info
> >
> > Please give feedback, suggest improvements, and proof read the new texts.
> >
> > Thanks, Fabian
> >
> > [1] https://github.com/apache/flink-web/pull/109
> >
>

Re: [PROPOSAL] Improving Flink’s timer management for large state

[Ted Yu]

+1
 Original message From: Bowen Li  Date: 
5/27/18  12:31 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: [PROPOSAL] 
Improving Flink’s timer management for large state 
+1 LGTM. RocksDB timer service is one of the most highly anticipated
features from Flink users, and it's finally coming, officially. I also
would love to see bringing timer more closely to state backend, for the
sake of easier development and maintenance of code.

On Fri, May 25, 2018 at 7:13 AM, Stefan Richter  wrote:

> Hi,
>
> I am currently planning how to improve Flink’s timer management for large
> state. In particular, I would like to introduce timer state that is managed
> in RocksDB and also to improve the capabilities of the heap-based timer
> service, e.g. support for asynchronous checkpoints. You can find a short
> outline of my planned approach in this document:
>
> https://docs.google.com/document/d/1XbhJRbig5c5Ftd77d0mKND1bePyTC
> 26Pz04EvxdA7Jc/edit?usp=sharing
>
> As always, your questions, feedback, and comments are highly appreciated.
>
> Best,
> Stefan

Re: [VOTE] Release 1.5.0, release candidate #5

[Ted Yu]

+1

Checked signatures
Ran test suite

Due to FLINK-9340 and FLINK-9091, I had to run tests in multiple rounds.

Cheers

On Wed, May 23, 2018 at 7:39 AM, Fabian Hueske  wrote:

> +1 (binding)
>
> - checked hashes and signatures
> - checked source archive and didn't find unexpected binary files
> - built from source archive skipping the tests (mvn -DskipTests clean
> install), started a local cluster, and ran an example program.
>
> Thanks,
> Fabian
>
>
>
> 2018-05-23 15:39 GMT+02:00 Till Rohrmann :
>
> > Fabian pointed me to the updated ASF release policy [1] and the changes
> it
> > implies for the checksum files. New releases should no longer provide a
> MD5
> > checksum file and the sha checksum file should have a proper file name
> > extension `sha512` instead of `sha`. I've updated the release artifacts
> [2]
> > accordingly.
> >
> > [1] https://www.apache.org/dev/release-distribution.html
> > [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc5/
> >
> > Cheers,
> > Till
> >
> > On Wed, May 23, 2018 at 11:18 AM, Piotr Nowojski <
> pi...@data-artisans.com>
> > wrote:
> >
> > > +1 from me.
> > >
> > > Additionally for this RC5 I did some manual tests to double check
> > backward
> > > compatibility of the bug fix:
> > > https://issues.apache.org/jira/browse/FLINK-9295 <
> > > https://issues.apache.org/jira/browse/FLINK-9295>
> > >
> > > The issue with this bug fix was that it was merged after 1.5.0 RC4 but
> > > just before 1.5.0 RC5, so it missed release testing.
> > >
> > > Piotrek
> > >
> > > > On 23 May 2018, at 09:08, Till Rohrmann 
> wrote:
> > > >
> > > > Thanks for the pointer Sihua, I've properly closed FLINK-9070.
> > > >
> > > > On Wed, May 23, 2018 at 4:49 AM, sihua zhou 
> > wrote:
> > > >
> > > >>
> > > >> Hi,
> > > >> just one minor thing, I found the JIRA release notes seem a bit
> > > >> inconsistent with the this RC. For example,
> > https://issues.apache.org/
> > > >> jira/browse/FLINK-9058 hasn't been merged yet but included in the
> > > release
> > > >> notes, and https://issues.apache.org/jira/browse/FLINK-9070 has
> been
> > > >> merged but not included in the relase notes.
> > > >>
> > > >> Best, Sihua
> > > >>
> > > >>
> > > >> On 05/23/2018 09:18，Till Rohrmann
> > > >>  wrote：
> > > >>
> > > >> Hi everyone,
> > > >>
> > > >> Please review and vote on the release candidate #5 for the version
> > > 1.5.0,
> > > >> as follows:
> > > >> [ ] +1, Approve the release
> > > >> [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >>
> > > >>
> > > >> The complete staging area is available for your review, which
> > includes:
> > > >> * JIRA release notes [1],
> > > >> * the official Apache source release and binary convenience releases
> > to
> > > be
> > > >> deployed to dist.apache.org [2], which are signed with the key with
> > > >> fingerprint 1F302569A96CFFD5 [3],
> > > >> * all artifacts to be deployed to the Maven Central Repository [4],
> > > >> * source code tag "release-1.5.0-rc5" [5],
> > > >>
> > > >> Please use this document for coordinating testing efforts: [6]
> > > >>
> > > >> Since the newly included fixes affect only individual components and
> > are
> > > >> covered by tests, I will shorten the voting period until tomorrow
> > 5:30pm
> > > >> CET. It is adopted by majority approval, with at least 3 PMC
> > affirmative
> > > >> votes.
> > > >>
> > > >> Thanks,
> > > >> Your friendly Release Manager
> > > >>
> > > >> [1]
> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > >> projectId=12315522=12341764
> > > >> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc5/
> > > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >> [4] https://repository.apache.org/content/repositories/
> > > >> orgapacheflink-1160/
> > > >> [5]
> > > >> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
> > > >> 841bfe4cceecc9cd6ad3d568173fdc0149a5dc9b
> > > >> [6]
> > > >> https://docs.google.com/document/d/1C1WwUphQj597jExWAXFUVkLH9Bi7-
> > > >> ir6drW9BgB8Ezo/edit?usp=sharing
> > > >>
> > > >> Pro-tip: you can create a settings.xml file with these contents:
> > > >>
> > > >> 
> > > >> 
> > > >> flink-1.5.0
> > > >> 
> > > >> 
> > > >> 
> > > >> flink-1.5.0
> > > >> 
> > > >> 
> > > >> flink-1.5.0
> > > >> 
> > > >>
> > > >> https://repository.apache.org/content/repositories/
> > orgapacheflink-1160/
> > > >> 
> > > >> 
> > > >> 
> > > >> archetype
> > > >> 
> > > >>
> > > >> https://repository.apache.org/content/repositories/
> > orgapacheflink-1160/
> > > >> 
> > > >> 
> > > >> 
> > > >> 
> > > >> 
> > > >> 
> > > >>
> > > >> And reference that in you maven commands via --settings
> > > >> path/to/settings.xml. This is useful for creating a quickstart based
> > on
> > > the
> > > >> staged release and for building against the staged jars.
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [VOTE] Enable GitBox integration (#2)

[Ted Yu]

+1
 Original message From: Chesnay Schepler  
Date: 5/22/18  1:12 AM  (GMT-08:00) To: dev@flink.apache.org Subject: [VOTE] 
Enable GitBox integration (#2) 
Hello,

since no concerns have been raised in the discussion about enabling 
GitBox [1] I'm opening this vote to make things official.

Please vote on enabling GitBox integration for the flink and flink-web 
repositories as follows:

[ ] +1, Approve GitBox
[ ] -1, Do not approve GitBox (please provide specific comments)

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.

This is the second attempt for this vote. The previous vote was 
cancelled due to the flink-web repository settings. Changes to the 
previous vote are highlighted in bold.

If accepted I will file a ticket with INFRA to enable GitBox with the 
following settings:

flink:

   * no wiki
   * no issues
   * no projects
   * no merge commit button (rebase/squash merge button still enabled)
   * protected branches [2]: master, release-1.[0-5] (the latter we will
 have to update with each release)

flink-web

   * no wiki
   **no issues*
   * no projects
   * no merge commit button (rebase/squash merge button still enabled)
   * protected branch: asf-site
   * default branch: asf-site (while we're at it...)

[1] 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-GitBox-td22328.html
[2] https://help.github.com/articles/about-protected-branches/

Re: [VOTE] Release 1.5.0, release candidate #4

[Ted Yu]

bq. We can merge the fix to the master and see if the problem goes away.

+1

On Fri, May 18, 2018 at 9:53 AM, Fabian Hueske <fhue...@gmail.com> wrote:

> FLINK-9091 is hard to reproduce and only affects the build but no
> functionality of Flink or its correctness.
> IMO, we should not block the release it.
>
> Since it's hard to reproduce, we cannot be sure that Timo's fix actually
> solves the problem.
> We can merge the fix to the master and see if the problem goes away.
>
> Best, Fabian
>
> 2018-05-18 18:31 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
>
> > When I ran tests, I hit FLINK-9091 .
> >
> > It seems Timo already has a fix.
> >
> > It would be nice to include the fix since flink-json module is new in 1.5
> > release.
> >
> > On Fri, May 18, 2018 at 9:26 AM, Till Rohrmann <trohrm...@apache.org>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > since the effective voting period for the RC4 was quite short and it
> > would
> > > be better to give the current RC a bit more exposure, I will extend the
> > > voting period for another 72 hours until Monday 6:30pm CET.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, May 18, 2018 at 4:22 PM, Till Rohrmann <trohrm...@apache.org>
> > > wrote:
> > >
> > > > - Checked checksums and GPG files
> > > > - Verified that source archives do not contain any binaries
> > > > - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
> > > > - Verified LICENSE and NOTICE file: The LICENSE file contains
> > unnecessary
> > > > entries for jline-reader and jline-terminal
> > > > - Checked licenses of newly added dependencies
> > > > - Checked pom files
> > > > - Read README.md
> > > > - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
> > > > because org.apache.flink.queryablestate.messages.
> > > KvStateRequest.serialize()
> > > > uses getBytes without charset. This is non-blocking because it was
> > > > introduced with 1.4 or before
> > > > - Checked builds with SBT and the SBT quickstarts
> > > > - Executed the run-nightly-tests.sh for 10 hours without failures
> > > > - Executed Jepsen tests without failures
> > > >
> > > > +1
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, May 18, 2018 at 12:58 PM, Piotr Nowojski <
> > > pi...@data-artisans.com>
> > > > wrote:
> > > >
> > > >> +1 From me. I have tested Scala and Java quickstarts and they work
> > fine
> > > >> except of sbt versions. But sbt quickstarts are also failing on my
> > > >> environment for 1.4.x. So it’s either pre-existing issue, or me
> doing
> > > >> something wrong.
> > > >>
> > > >> I have also run some quick tests on EMR cluster, without problems.
> > > >>
> > > >> Piotrek
> > > >>
> > > >> > On 18 May 2018, at 09:48, Till Rohrmann <trohrm...@apache.org>
> > wrote:
> > > >> >
> > > >> > Just a quick correction, I accidentally wrote that the voting
> period
> > > >> ends
> > > >> > tomorrow 5:30pm CET but what I actually meant was today 5:30pm
> CET.
> > > The
> > > >> > problem was that I compiled the mail before midnight.
> > > >> >
> > > >> > Cheers,
> > > >> > Till
> > > >> >
> > > >> > On Fri, May 18, 2018 at 2:50 AM, Till Rohrmann <
> > trohrm...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> >> Hi everyone,
> > > >> >>
> > > >> >> Please review and vote on the release candidate #4 for the
> version
> > > >> 1.5.0,
> > > >> >> as follows:
> > > >> >> [ ] +1, Approve the release
> > > >> >> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >> >>
> > > >> >>
> > > >> >> The complete staging area is available for your review, which
> > > includes:
> > > >> >> * JIRA release notes [1],
> > > >> >> * the official Apache source release and binary convenience
> > releases
> > 

Re: [VOTE] Release 1.5.0, release candidate #4

[Ted Yu]

When I ran tests, I hit FLINK-9091 .

It seems Timo already has a fix.

It would be nice to include the fix since flink-json module is new in 1.5
release.

On Fri, May 18, 2018 at 9:26 AM, Till Rohrmann  wrote:

> Hi everyone,
>
> since the effective voting period for the RC4 was quite short and it would
> be better to give the current RC a bit more exposure, I will extend the
> voting period for another 72 hours until Monday 6:30pm CET.
>
> Cheers,
> Till
>
> On Fri, May 18, 2018 at 4:22 PM, Till Rohrmann 
> wrote:
>
> > - Checked checksums and GPG files
> > - Verified that source archives do not contain any binaries
> > - Built Flink with Hadoop 2.8.1 and Scala 2.11.7 from source release
> > - Verified LICENSE and NOTICE file: The LICENSE file contains unnecessary
> > entries for jline-reader and jline-terminal
> > - Checked licenses of newly added dependencies
> > - Checked pom files
> > - Read README.md
> > - Run manual tests in flink-tests: CheckForbiddenMethodsUsage fails
> > because org.apache.flink.queryablestate.messages.
> KvStateRequest.serialize()
> > uses getBytes without charset. This is non-blocking because it was
> > introduced with 1.4 or before
> > - Checked builds with SBT and the SBT quickstarts
> > - Executed the run-nightly-tests.sh for 10 hours without failures
> > - Executed Jepsen tests without failures
> >
> > +1
> >
> > Cheers,
> > Till
> >
> > On Fri, May 18, 2018 at 12:58 PM, Piotr Nowojski <
> pi...@data-artisans.com>
> > wrote:
> >
> >> +1 From me. I have tested Scala and Java quickstarts and they work fine
> >> except of sbt versions. But sbt quickstarts are also failing on my
> >> environment for 1.4.x. So it’s either pre-existing issue, or me doing
> >> something wrong.
> >>
> >> I have also run some quick tests on EMR cluster, without problems.
> >>
> >> Piotrek
> >>
> >> > On 18 May 2018, at 09:48, Till Rohrmann  wrote:
> >> >
> >> > Just a quick correction, I accidentally wrote that the voting period
> >> ends
> >> > tomorrow 5:30pm CET but what I actually meant was today 5:30pm CET.
> The
> >> > problem was that I compiled the mail before midnight.
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Fri, May 18, 2018 at 2:50 AM, Till Rohrmann 
> >> wrote:
> >> >
> >> >> Hi everyone,
> >> >>
> >> >> Please review and vote on the release candidate #4 for the version
> >> 1.5.0,
> >> >> as follows:
> >> >> [ ] +1, Approve the release
> >> >> [ ] -1, Do not approve the release (please provide specific comments)
> >> >>
> >> >>
> >> >> The complete staging area is available for your review, which
> includes:
> >> >> * JIRA release notes [1],
> >> >> * the official Apache source release and binary convenience releases
> >> to be
> >> >> deployed to dist.apache.org [2], which are signed with the key with
> >> >> fingerprint 1F302569A96CFFD5 [3],
> >> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> >> * source code tag "release-1.5.0-rc3" [5],
> >> >>
> >> >> Please use this document for coordinating testing efforts: [6]
> >> >>
> >> >> Since the newly included fixes affect only individual components and
> >> are
> >> >> covered by tests, I will keep the voting period of the RC3 which goes
> >> at
> >> >> least until tomorrow 5:30pm CET. It is adopted by majority approval,
> >> with
> >> >> at least 3 PMC affirmative votes.
> >> >>
> >> >> Thanks,
> >> >> Your friendly Release Manager
> >> >>
> >> >> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> >> projectId=12315522=12341764
> >> >> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc4/
> >> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> >> [4] https://repository.apache.org/content/repositories/
> >> >> orgapacheflink-1159/
> >> >> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
> >> >> 953f45316b09072e76f3cf3d37d704a7cb7b32c6
> >> >> [6] https://docs.google.com/document/d/13evetGNVeGvSufeqC0bcfiHGS_
> >> >> v8n06I0p6nEtaQzS4/edit?usp=sharing
> >> >>
> >> >> Pro-tip: you can create a settings.xml file with these contents:
> >> >>
> >> >> 
> >> >> 
> >> >>  flink-1.5.0
> >> >> 
> >> >> 
> >> >>  
> >> >>flink-1.5.0
> >> >>
> >> >>  
> >> >>flink-1.5.0
> >> >>
> >> >>https://repository.apache.org/content/repositories/
> >> >> orgapacheflink-1159/
> >> >>
> >> >>  
> >> >>  
> >> >>archetype
> >> >>
> >> >>https://repository.apache.org/content/repositories/
> >> >> orgapacheflink-1159/
> >> >>
> >> >>  
> >> >>
> >> >>  
> >> >> 
> >> >> 
> >> >>
> >> >> And reference that in you maven commands via --settings
> >> >> path/to/settings.xml. This is useful for creating a quickstart based
> >> on the
> >> >> staged release and for building against the staged jars.
> >> >>
> >>
> >>
> >
>

[jira] [Created] (FLINK-9393) LocatableInputSplit#hashCode should take hostnames into account

[Ted Yu (JIRA)]

Ted Yu created FLINK-9393:
-

 Summary: LocatableInputSplit#hashCode should take hostnames into 
account
 Key: FLINK-9393
 URL: https://issues.apache.org/jira/browse/FLINK-9393
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Currently:
{code}
  public int hashCode() {
return this.splitNumber;
{code}
This is not symmetrical with {{equals}} method where hostnames are compared.

LocatableInputSplit#hashCode should take hostnames into account.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] GitBox

[Ted Yu]

+1
 Original message From: Shuyi Chen  Date: 
5/16/18  1:12 PM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: [DISCUSS] 
GitBox 
+1 :) A lot of projects  are already
using it.

On Wed, May 16, 2018 at 3:40 AM, Chesnay Schepler 
wrote:

> Hello,
>
> during the discussion about how to better manage pull requests [1] the
> topic of GitBox integration came up again.
>
> This seems like a good opportunity to restart this discussion that we had
> about a year ago [2].
>
> * What is GitBox
>
>    Essentially, GitBox allow us to use GitHub features.
>    We can decide for ourselves which features we want enabled.
>
>    We could merge PRs directly on GitHub at the button of a click.
>    That said the merge functionality is fairly limited and would
>    require picture-perfect commits in the pull requests.
>    Commits can be squashed, but you cannot amend commits in any way, be
>    it fixing typos or changing the commit message. Realistically this
>    limits how much we can use this feature, and it may lead to a
>    decline in the quality of commit messages.
>
>    Labels can be useful for the management of PRs as (ready for review,
>    delayed for next release, waiting for changes). This is really what
>    I'm personally most interested in.
>
>    We've been using GitBox for flink-shaded for a while now and i
>    didn't run into any issue. AFAIK GitBox is also the default for new
>    projects.
>
> * What this means for committers:
>
>    The apache git remote URL will change, which will require all
>    committers to update their git setup.
>    This also implies that we may have to update the website build scripts.
>    The new URL would (probably) be
>    /https://gitbox.apache.org/repos/asf/flink.git/.
>
>    To make use of GitHub features you have to link your GitHub and
>    Apache accounts. [3]
>    This also requires setting up two-factor authentication on GitHub.
>
>    Update the scm entry in the parent pom.xml.
>
> * What this means for contributors:
>
>    Nothing should change for contributors. Small changes (like typos)
>    may be merged more quickly, if the commit message is appropriate, as
>    it could be done directly through GitHub.
>
> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.
> com/Closing-automatically-inactive-pull-requests-tt22248.html
> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.
> com/DISCUSS-GitBox-td18027.html
> [3] https://gitbox.apache.org/setup/
>



-- 
"So you have to trust that the dots will somehow connect in your future."

Re: [DISCUSS] Drop "canEqual" from TypeInformation, TypeSerializer, etc.

[Ted Yu]

+1 from me as well.

I checked a few serializer classes. The `equals` method on serializers
contains the logic of `canEqual` method whose existence seems redundant.

On Wed, May 16, 2018 at 1:49 AM, Tzu-Li (Gordon) Tai 
wrote:

> +1.
>
> Looking at the implementations of the `canEqual` method in several
> serializers, it seems like all that is done is a check whether the object
> is of the same serializer class.
> We’ll have to be careful and double check all `equals` method on
> serializers that may have relied on the `canEqual` method to perform the
> preliminary type check.
> Otherwise, this sounds good.
>
> On 16 May 2018 at 4:35:47 PM, Stephan Ewen (se...@apache.org) wrote:
>
> Hi all!
>
> As part of an attempt to simplify some code in the TypeInfo and
> TypeSerializer area, I would like to drop the "canEqual" methods for the
> following reason:
>
> "canEqual()" is necessary to make proper equality checks across
> hierarchies
> of types. This is for example useful in a collection API, stating for
> example whether a List can be equal to a Collection if they have the same
> contents. We don't have that here.
>
> A certain type information (and serializer) is equal to another one if
> they
> describe the same type, strictly. There is no necessity for cross
> hierarchy
> checks.
>
> This has also let to the situation that most type infos and serializers
> implement just a dummy/default version of "canEqual". Many "equals()"
> methods do not even call the other object's "canEqual", etc.
>
> As a first step, we could simply deprecate the method and implement an
> empty default, and remove all calls to that method.
>
> Best,
> Stephan
>

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

[Ted Yu]

bq. In a design document, Timo mentioned that we can ship multiple JAR files

Mind telling us where the design doc can be retrieved ?

Thanks

On Wed, May 16, 2018 at 1:29 AM, Fabian Hueske  wrote:

> Hi,
>
> I'm not sure if we need to modify the existing method.
> What we need is a bit different from what registerCachedFile() provides.
> The method ensures that a file is copied to each TaskManager and can be
> locally accessed from a function's RuntimeContext.
> In our case, we don't need to access the file but would like to make sure
> that it is loaded into the class loader.
> So, we could also just add a method like registerUserJarFile().
>
> In a design document, Timo mentioned that we can ship multiple JAR files
> with a job.
> So, we could also implement the UDF shipping logic by loading the Jar
> file(s) to the client and distribute them from there.
> In that case, we would not need to add new method to the execution
> environment.
>
> Best,
> Fabian
>
> 2018-05-15 3:50 GMT+02:00 Rong Rong :
>
> > +1. This could be very useful for "dynamic" UDF.
> >
> > Just to clarify, if I understand correctly, we are tying to use an ENUM
> > indicator to
> > (1) Replace the current Boolean isExecutable flag.
> > (2) Provide additional information used by ExecutionEnvironment to decide
> > when/where to use the DistributedCached file.
> >
> > In this case, DistributedCache.CacheType or DistributedCache.FileType
> > sounds more intuitive, what do you think?
> >
> > Also, I was wondering is there any other useful information for the
> cached
> > file to be passed to runtime.
> > If we are just talking about including the library to the classloader,
> can
> > we directly extend the interface with
> >
> > public void registerCachedFile(
> > String filePath,
> > String name,
> > boolean executable,
> > boolean includeInClassLoader)
> >
> >
> > Thanks,
> > Rong
> >
> > On Sun, May 13, 2018 at 11:14 PM, Shuyi Chen  wrote:
> >
> > > Hi Flink devs,
> > >
> > > In an effort to support loading external libraries and creating UDFs
> from
> > > external libraries using DDL in Flink SQL, we want to use Flink’s Blob
> > > Server to distribute the external libraries in runtime and load those
> > > libraries into the user code classloader automatically.
> > >
> > > However, the current [Stream]ExecutionEnvironment.registerCachedFile
> > > interface limits only to registering executable or non-executable
> blobs.
> > > It’s not possible to tell in runtime if the blob files are libraries
> and
> > > should be loaded into the user code classloader in RuntimeContext.
> > > Therefore, I want to propose to add an enum called *BlobType*
> explicitly
> > to
> > > indicate the type of the Blob file being distributed, and the following
> > > interface in [Stream]ExecutionEnvironment to support it. In general, I
> > > think the new BlobType information can be used by Flink runtime to
> > > preprocess the Blob files if needed.
> > >
> > > */***
> > > ** Registers a file at the distributed cache under the given name. The
> > file
> > > will be accessible*
> > > ** from any user-defined function in the (distributed) runtime under a
> > > local path. Files*
> > > ** may be local files (as long as all relevant workers have access to
> > it),
> > > or files in a distributed file system.*
> > > ** The runtime will copy the files temporarily to a local cache, if
> > > needed.*
> > > ***
> > > ** The {@link org.apache.flink.api.common.functions.RuntimeContext}
> > can
> > > be obtained inside UDFs via*
> > > ** {@link
> > > org.apache.flink.api.common.functions.RichFunction#
> getRuntimeContext()}
> > > and
> > > provides access*
> > > ** {@link org.apache.flink.api.common.ca
> > > che.DistributedCache} via*
> > > ** {@link
> > > org.apache.flink.api.common.functions.RuntimeContext#
> > > getDistributedCache()}.*
> > > ***
> > > ** @param filePath The path of the file, as a URI (e.g.
> > "file:///some/path"
> > > or "hdfs://host:port/and/path")*
> > > ** @param name The name under which the file is registered.*
> > > ** @param blobType indicating the type of the Blob file*
> > > **/*
> > >
> > > *public void registerCachedFile(String filePath, String name,
> > > DistributedCache.BlobType blobType) {...}*
> > >
> > > Optionally, we can add another interface to register UDF Jars which
> will
> > > use the interface above to implement.
> > >
> > > *public void registerJarFile(String filePath, String name) {...}*
> > >
> > > The existing interface in the following will be marked deprecated:
> > >
> > > *public void registerCachedFile(String filePath, String name, boolean
> > > executable) {...}*
> > >
> > > And the following interface will be implemented using the new interface
> > > proposed above with a EXECUTABLE BlobType:
> > >
> > > *public void registerCachedFile(String filePath, String name) { ... }*
> > >
> > > Thanks a lot.
> > > 

Re: Closing (automatically?) inactive pull requests

[Ted Yu]

> >> is not the end of the world. It always can be reopened.
> >>
> >> Regarding only 30% blocked on contributor. I wonder what would be this
> >> number if we tried to ask in the rest of old PRs “Hey, are you still
> >> interested in reviewing/merging this?”. If old PR is waiting for a
> reviewer
> >> for X months, it doesn’t mean that’s not abandoned. Even if it doesn’t,
> >> reducing our overhead by those 30% of the PRs is something.
> >>
> >> Piotrek
> >>
> >> On 15 May 2018, at 10:10, Fabian Hueske <fhue...@gmail.com> wrote:
> >>>
> >>> I'm with Chesnay on this issue.
> >>> Stale PRs, i.e., a PR where a contributor becomes inactive, are one of
> >>> our
> >>> smallest issues, IMO.
> >>>
> >>> There are more reasons for the high number of PRs.
> >>> * Lack of timely reviews.
> >>> * Not eagerly closing PRs that have no or very little chance of being
> >>> merged. Common reasons are
> >>>   1) lack of interest in the feature by committers,
> >>>   2) too extensive changes and hence time consuming reviews, or
> >>>   3) bad quality.
> >>>
> >>> For 1), there are lots of older JIRA issues, that have low priority but
> >>> which are picked up by contributors. In the contribution guidelines we
> >>> ask
> >>> contributors to let us know when they want to work on an issue. So far
> >>> our
> >>> attitude has been, yes sure go ahead. I've seen very little attempts of
> >>> warning somebody to work on issues that won't be easily merged.
> >>> Once a PR has been opened, we should also be honest and let
> contributors
> >>> know that it has no chance or might take a while to get reviewed.
> >>> For 2) this is typically not so much of an issue if the feature is
> >>> interesting. However, if 1) and 2) meet, chances to get a change in
> drop
> >>> even more.
> >>>
> >>> A common "strategy" to deal with PRs that fall into 1), 2), or 3) is to
> >>> not
> >>> look at them or giving a shallow review.
> >>> Of course, contributors become unresponsive if we don't look at their
> PRs
> >>> for weeks or months. But that's not their fault.
> >>> Instead, I think we should be honest and communicate the chances of a
> PR
> >>> more clearly.
> >>>
> >>> Browsing over the list of open PRs, I feel that most older PRs fall
> into
> >>> the category of low-priority (or even unwanted) features.
> >>> Bug fixes or features that the committers care about usually make it
> into
> >>> the code base.
> >>> In case a contributor becomes inactive, committers often take over an
> >>> push
> >>> a contribution over the line.
> >>>
> >>> So, I do not support an auto-close mechanism.
> >>> I think a better way to address the issue is better communication, more
> >>> eagerly closing PRs with no chance, and putting more reviewing effort.
> >>> IMO, we should only eagerly close PRs that have no chance of being
> >>> merged.
> >>> PRs with low-prio features might be picked up later (for Flink 1.5, I
> >>> merged a contribution from PR #1990 after it was requested a few times
> by
> >>> users).
> >>>
> >>> However, I think we could do a pass over the oldest PRs and check if we
> >>> can
> >>> close a bunch.
> >>> There are quite a few contributions (many for flink-ml) that I don't
> see
> >>> a
> >>> chance for getting merged.
> >>>
> >>> Best, Fabian
> >>>
> >>>
> >>> -
> >>>
> >>> 2018-05-15 9:13 GMT+02:00 Chesnay Schepler <ches...@apache.org>:
> >>>
> >>> -1
> >>>>
> >>>> For clarification (since the original mail indicates otherwise), the
> >>>> number of pull requests that this would affect is fairly small.
> >>>> Only about 25-30% of all open PRs are blocked on the contributor, the
> >>>> remaining ones are actually blocked on the review.
> >>>> Thus is reject the premise that one has to search through that many
> PRs
> >>>> to
> >>>> find something to review, there is plenty.
> >>>>
> >>>> I believe it to be highly unfair for us to close PRs due to
> inac

Re: Closing (automatically?) inactive pull requests

[Ted Yu]

bq. this pull request requires a review, please simply write any comment.

Shouldn't the wording of such comment be known before hand ?

Otherwise pull request waiting for committers' review may be mis-classified.

Cheers

On Mon, May 14, 2018 at 7:59 PM, blues zheng  wrote:

> +1 for the proposal.
>
>
> Best,
> blues
> On 05/14/2018 20:58, Ufuk Celebi wrote:
> Hey Piotr,
>
> thanks for bringing this up. I really like this proposal and also saw
> it work successfully at other projects. So +1 from my side.
>
> - I like the approach with a notification one week before
> automatically closing the PR
> - I think a bot will the best option as these kinds of things are
> usually followed enthusiastically in the beginning but eventually
> loose traction
>
> We can enable better integration with GitHub by using ASF GitBox
> (https://gitbox.apache.org/setup/) but we should discuss that in a
> separate thread.
>
> – Ufuk
>
> On Mon, May 14, 2018 at 12:04 PM, Piotr Nowojski
>  wrote:
> > Hey,
> >
> > We have lots of open pull requests and quite some of them are
> stale/abandoned/inactive. Often such old PRs are impossible to merge due to
> conflicts and it’s easier to just abandon and rewrite them. Especially
> there are some PRs which original contributor created long time ago,
> someone else wrote some comments/review and… that’s about it. Original
> contributor never shown up again to respond to the comments. Regardless of
> the reason such PRs are clogging the GitHub, making it difficult to keep
> track of things and making it almost impossible to find a little bit old
> (for example 3+ months) PRs that are still valid and waiting for reviews.
> To do something like that, one would have to dig through tens or hundreds
> of abandoned PRs.
> >
> > What I would like to propose is to agree on some inactivity dead line,
> lets say 3 months. After crossing such deadline, PRs should be
> marked/commented as “stale”, with information like:
> >
> > “This pull request has been marked as stale due to 3 months of
> inactivity. It will be closed in 1 week if no further activity occurs. If
> you think that’s incorrect or this pull request requires a review, please
> simply write any comment.”
> >
> > Either we could just agree on such policy and enforce it manually (maybe
> with some simple tooling, like a simple script to list inactive PRs - seems
> like couple of lines in python by using PyGithub) or we could think about
> automating this action. There are some bots that do exactly this (like this
> one: https://github.com/probot/stale  ),
> but probably they would need to be adopted to limitations of our Apache
> repository (we can not add labels and we can not close the PRs via GitHub).
> >
> > What do you think about it?
> >
> > Piotrek
>

[jira] [Created] (FLINK-9363) Bump up the Jackson version

[Ted Yu (JIRA)]

Ted Yu created FLINK-9363:
-

 Summary: Bump up the Jackson version
 Key: FLINK-9363
 URL: https://issues.apache.org/jira/browse/FLINK-9363
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


CVE's for Jackson:

CVE-2017-17485
CVE-2018-5968
CVE-2018-7489

We can upgrade to 2.9.5



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.5.0, release candidate #2

[Ted Yu]

Can you try out mvn 3.5.2 ?

I don't get the error when running the command line you gave.

BTW 2.7.3.2.6.2.0-205 was quite old release.

Cheers

On Mon, May 14, 2018 at 7:15 AM, shashank734  wrote:

> While building from source failing with following error :
>
> Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce
> (dependency-convergence) on project flink-bucketing-sink-test
>
>
> MVN Version : 3.0.5
>
> Command : mvn clean install -DskipTests -Dscala.version=2.11.7
> -Pvendor-repos -Dhadoop.version=2.7.3.2.6.2.0-205 > build-output_2.log
>
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

Re: Snapshots and RC-candidate maven artifacts

[Ted Yu]

Flavio:
Can you use the snapshot for 1.5 RC ?
https://repository.apache.org/content/repositories/orgapacheflink-1154/

It was uploaded on Apr 2nd.

FYI

On Mon, May 14, 2018 at 7:54 AM, Fabian Hueske  wrote:

> Hi,
>
> I'd assume that we stopped updating 1.5-SNAPSHOT jars when we forked off
> the release-1.5 branch and updated the version on master to 1.6-SNAPSHOT.
>
> Best, Fabian
>
> 2018-05-14 15:51 GMT+02:00 Flavio Pompermaier :
>
> > Hi to all.
> > we were trying to run a 1.5 Flink job and we set the version to
> > 1.5-SNAPSHOT.
> > Unfortunately the 1.5-SNAPSHOT version uploaded on the apache snapshot
> repo
> > is very old (february 2018). Shouldn't be this version be updated as
> well?
> >
> > Best,
> > Flavio
> >
>

[jira] [Created] (FLINK-9340) ScheduleOrUpdateConsumersTest may fail with Address already in use

[Ted Yu (JIRA)]

Ted Yu created FLINK-9340:
-

 Summary: ScheduleOrUpdateConsumersTest may fail with Address 
already in use
 Key: FLINK-9340
 URL: https://issues.apache.org/jira/browse/FLINK-9340
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu


When ScheduleOrUpdateConsumersTest is run in the test suite, I saw:
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.034 sec <<< 
FAILURE! - in 
org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest
org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest  
Time elapsed: 8.034 sec  <<< ERROR!
java.net.BindException: Address already in use
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:433)
  at sun.nio.ch.Net.bind(Net.java:425)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1081)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:502)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:487)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:904)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
  at 
org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
  at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
  at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
  at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
{code}
Seems there was address / port conflict.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.5.0, release candidate #2

[Ted Yu]

I ran the test suite twice and both failed with:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.784 sec
<<< FAILURE! - in
org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest
org.apache.flink.runtime.jobmanager.scheduler.ScheduleOrUpdateConsumersTest
Time elapsed: 9.784 sec  <<< ERROR!
java.net.BindException: Address already in use
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:433)
  at sun.nio.ch.Net.bind(Net.java:425)
  at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  at
org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
  at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
  at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1081)
  at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:502)
  at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:487)
  at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:904)
  at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
  at
org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
  at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
  at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
  at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
  at java.lang.Thread.run(Thread.java:748)

The test passes when run alone.

On Thu, May 10, 2018 at 9:37 AM, Till Rohrmann  wrote:

> Hi everyone,
>
> it took some time to compile the next release candidate but here we are:
> Please review and vote on the release candidate #2 for the version 1.5.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1F302569A96CFFD5 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.5.0-rc2" [5],
>
> Please use this document for coordinating testing efforts: [6]
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Your friendly Release Manager
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341764
> [2] http://people.apache.org/~trohrmann/flink-1.5.0-rc2/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1155
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=commit;h=
> 37af4d7e7072958a6d8bdfc49de2ed3a5f66c889
> [6] https://docs.google.com/document/d/1rJe_6yDPBurnhipmcSeCnpYFnr1SAuHyOQ
> N2-08mJYc/edit?usp=sharing
>
> Pro-tip: you can create a settings.xml file with these contents:
>
> 
> 
>   flink-1.5.0
> 
> 
>   
> flink-1.5.0
> 
>   
> flink-1.5.0
> 
>
> https://repository.apache.org/content/repositories/orgapacheflink-1155/
> 
>   
>   
> archetype
> 
>
> https://repository.apache.org/content/repositories/orgapacheflink-1155/
> 
>   
> 
>   
> 
> 
>
> And reference that in you maven commands via --settings
> path/to/settings.xml. This is useful for creating a quickstart based on the
> staged release and for building against the staged jars.
>

Re: [ANNOUNCE] Two new committers: Xingcan Cui and Nico Kruber

[Ted Yu]

Congratulations, Xingcan and Nico !

On Tue, May 8, 2018 at 11:52 AM, Fabian Hueske  wrote:

> Hi everyone,
>
> I'm happy to announce that two members of the Flink community accepted the
> offer of the PMC to become committers.
>
> * Xingcan Cui has been contributing to Flink for about a year, focusing on
> Flink's relational APIs (SQL & Table API). In the past year, Xingcan has
> started design discussions, helped reviewing several pull requests, and
> replied to questions on the user mailing list.
>
> * Nico Kruber is an active contributor since 1.5 years and worked mostly on
> internal features, such as the blob manager and a new network stack. Nico
> answers many questions on the user mailing list, reports lots of bugs and
> is a very active PR reviewer.
>
> Please join me in congratulating Xingcan and Nico.
>
> Cheers,
> Fabian
>

[jira] [Created] (FLINK-9265) Upgrade Prometheus version

[Ted Yu (JIRA)]

Ted Yu created FLINK-9265:
-

 Summary: Upgrade Prometheus version
 Key: FLINK-9265
 URL: https://issues.apache.org/jira/browse/FLINK-9265
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


We're using 0.0.26

Latest release is 2.2.1

This issue is for upgrading the Prometheus version



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9236) Use Apache Parent POM 19

[Ted Yu (JIRA)]

Ted Yu created FLINK-9236:
-

 Summary: Use Apache Parent POM 19
 Key: FLINK-9236
 URL: https://issues.apache.org/jira/browse/FLINK-9236
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Flink is still using Apache Parent POM 18. Apache Parent POM 19 is out. This 
will also fix Javadoc generation with JDK 10+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9231) Enable SO_REUSEADDR on listen sockets

[Ted Yu (JIRA)]

Ted Yu created FLINK-9231:
-

 Summary: Enable SO_REUSEADDR on listen sockets
 Key: FLINK-9231
 URL: https://issues.apache.org/jira/browse/FLINK-9231
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


This allows sockets to be bound even if there are sockets
from a previous application that are still pending closure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: TaskManager deadlock on NetworkBufferPool

[Ted Yu]

Amit:
Execution plan attachment didn't come through.

Please consider using third party website for storing the plan.

FYI

On Thu, Apr 19, 2018 at 10:04 AM, Amit Jain <aj201...@gmail.com> wrote:

> @Ufuk Please find execution plan in the attachment.
>
> @Nico Job is not making progress at all. This issue is happening
> randomly. Few of our jobs are working with only few MB of data and still,
> they are getting stuck even TM have 22G with 2 slots per TM.
>
> I've started using 1.5 and facing few issues which I'm communicating with
> community these days. However, this issue seems to be solved there :-) Do
> you guys have a timeline for 1.5 release?
>
> --
> Thanks,
> Amit
>
>
>
>
>
> On Fri, Apr 6, 2018 at 10:40 PM, Ufuk Celebi <u...@apache.org> wrote:
>
>> Hey Amit!
>>
>> Thanks for posting this here. I don't think it's an issue of the
>> buffer pool per se. Instead I think there are two potential causes
>> here:
>>
>> 1. The generated flow doesn't use blocking intermediate results for a
>> branching-joining flow.
>> => I think we can check it if you run and post the output of
>> `StreamExecutionEnvironment#getExecutionPlan()` here. Can you please
>> post the result of this here?
>>
>> 2. The blocking intermediate result is used but there is an issue with
>> the implementation of them.
>> => Depending on the output of 1, we can investigate this option.
>>
>> As Fabian mentioned, running this with a newer version of Flink will
>> be very helpful. If the problem still persists, it will also make it
>> more likely that the issue will be fixed faster. ;-)
>>
>> – Ufuk
>>
>>
>> On Fri, Apr 6, 2018 at 5:43 AM, Nico Kruber <n...@data-artisans.com>
>> wrote:
>> > I'm not aware of any changes regarding the blocking buffer pools though.
>> >
>> > Is it really stuck or just making progress slowly? (You can check with
>> > the number or records sent/received in the Web UI)
>> >
>> > Anyway, this may also simply mean that the task is back-pressured
>> > depending on how the operators are wired together. In that case, all
>> > available buffers for that ResultPartition have been used (records were
>> > serialized into them) and are now waiting on Netty to send or a
>> > SpillingSubpartition to spill data to disk.
>> > Please also check for warnings or errors in the affected TaskManager's
>> > log files.
>> >
>> > If you can reproduce the problem, could you try reducing your program to
>> > a minimal working example and provide it here for further debugging?
>> >
>> >
>> > Thanks
>> > Nico
>> >
>> > On 04/04/18 23:00, Fabian Hueske wrote:
>> >> Hi Amit,
>> >>
>> >> The network stack has been redesigned for the upcoming Flink 1.5
>> release.
>> >> The issue might have been fixed by that.
>> >>
>> >> There's already a first release candidate for Flink 1.5.0 available
>> [1].
>> >> It would be great if you would have the chance to check if the bug is
>> still
>> >> present.
>> >>
>> >> Best, Fabian
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/a6b6fb1a42a975608fa8641
>> c86df30b47f022985ade845f1f1ec542a@%3Cdev.flink.apache.org%3E
>> >>
>> >> 2018-04-04 20:23 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
>> >>
>> >>> I searched for 0x0005e28fe218 in the two files you attached
>> >>> to FLINK-2685 but didn't find any hit.
>> >>>
>> >>> Was this the same instance as the attachment to FLINK-2685 ?
>> >>>
>> >>> Thanks
>> >>>
>> >>> On Wed, Apr 4, 2018 at 10:21 AM, Amit Jain <aj201...@gmail.com>
>> wrote:
>> >>>
>> >>>> +u...@flink.apache.org
>> >>>>
>> >>>> On Wed, Apr 4, 2018 at 11:33 AM, Amit Jain <aj201...@gmail.com>
>> wrote:
>> >>>>> Hi,
>> >>>>>
>> >>>>> We are hitting TaskManager deadlock on NetworkBufferPool bug in
>> Flink
>> >>>> 1.3.2.
>> >>>>> We have set of ETL's merge jobs for a number of tables and stuck
>> with
>> >>>> above
>> >>>>> issue randomly daily.
>> >>>>>
>> >>>>> I'm attaching the thread dump of JobManager and one of the Task
>> Manager
>> 

[jira] [Created] (FLINK-9223) bufferConsumers should be closed in SpillableSubpartitionTest#testConsumeSpilledPartitionSpilledBeforeAdd

[Ted Yu (JIRA)]

Ted Yu created FLINK-9223:
-

 Summary: bufferConsumers should be closed in 
SpillableSubpartitionTest#testConsumeSpilledPartitionSpilledBeforeAdd
 Key: FLINK-9223
 URL: https://issues.apache.org/jira/browse/FLINK-9223
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu


{code}
BufferConsumer[] bufferConsumers = Arrays.stream(bufferBuilders).map(
  BufferBuilder::createBufferConsumer
).toArray(BufferConsumer[]::new);
{code}
After operation on bufferConsumers is done, the BufferConsumer's in the array 
should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9214) YarnClient should be stopped in YARNSessionCapacitySchedulerITCase#testDetachedPerJobYarnClusterInternal

[Ted Yu (JIRA)]

Ted Yu created FLINK-9214:
-

 Summary: YarnClient should be stopped in 
YARNSessionCapacitySchedulerITCase#testDetachedPerJobYarnClusterInternal
 Key: FLINK-9214
 URL: https://issues.apache.org/jira/browse/FLINK-9214
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu


YARNSessionCapacitySchedulerITCase#testDetachedPerJobYarnClusterInternal 
creates YarnClient without stopping it at the end of the test.

YarnClient yc should be stopped before returning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9185) Potential null dereference in PrioritizedOperatorSubtaskState#resolvePrioritizedAlternatives

[Ted Yu (JIRA)]

Ted Yu created FLINK-9185:
-

 Summary: Potential null dereference in 
PrioritizedOperatorSubtaskState#resolvePrioritizedAlternatives
 Key: FLINK-9185
 URL: https://issues.apache.org/jira/browse/FLINK-9185
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
if (alternative != null
  && alternative.hasState()
  && alternative.size() == 1
  && approveFun.apply(reference, alternative.iterator().next())) {
{code}
The return value from approveFun.apply would be unboxed.
We should check that the return value is not null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Documentation glitch w/AsyncFunction?

[Ted Yu]

Sounds good to me.

On Sat, Apr 14, 2018 at 9:55 AM, Ken Krugler <kkrugler_li...@transpac.com>
wrote:

> Hi Ted,
>
> Thanks - yes regarding renaming the variable, and changing the type.
>
> The other issue is that most clients return a Future, not a
> CompletableFuture.
>
> So should I do the bit of extra code to show using a CompletableFuture
> with a Future?
>
> — Ken
>
> > On Apr 14, 2018, at 9:13 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > bq.  resultFuture.thenAccept( (String result) -> {
> >
> > I think the type of variable for the above call should be
> CompletableFuture.
> > Meaning, the variable currently named resultFuture should be renamed so
> > that the intention is clearer.
> >
> > bq.  resultFuture.complete(Collections.singleton(new
> > Tuple2<>(str, result)));
> >
> > Looking at existing code in unit tests, the complete() call is on the
> > parameter.
> >
> > Cheers
> >
> > On Sat, Apr 14, 2018 at 8:34 AM, Ken Krugler <
> kkrugler_li...@transpac.com>
> > wrote:
> >
> >> Hi devs,
> >>
> >> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/
> >> operators/asyncio.html <https://ci.apache.org/
> projects/flink/flink-docs-
> >> release-1.4/dev/stream/operators/asyncio.html>
> >>
> >> Has this example of asyncInvoke:
> >>
> >>> @Override
> >>>public void asyncInvoke(final String str, final
> >> ResultFuture<Tuple2<String, String>> resultFuture) throws Exception {
> >>>
> >>>// issue the asynchronous request, receive a future for result
> >>>Future resultFuture = client.query(str);
> >>>
> >>>// set the callback to be executed once the request by the
> >> client is complete
> >>>// the callback simply forwards the result to the result future
> >>>resultFuture.thenAccept( (String result) -> {
> >>>
> >>>resultFuture.complete(Collections.singleton(new
> >> Tuple2<>(str, result)));
> >>>
> >>>});
> >>>}
> >>
> >> 1. there’s a resultFuture parameter, and a resultFuture variable.
> >>
> >> 2. resultFuture.thenAccept() is a method available for
> CompletableFuture <
> >> https://docs.oracle.com/javase/8/docs/api/java/util/
> >> concurrent/CompletableFuture.html>, not Future.
> >>
> >> I can fix this up, but I’m wondering what you think the code should do,
> >> assuming there’s a typical client that returns a Future vs. a
> >> CompletableFuture.
> >>
> >> e.g. I could use CompletableFuture.supplyAsync(new Supplier()
> { }
> >> with a get that calls the Future’s get().
> >>
> >> Thanks,
> >>
> >> — Ken
> >>
> >> 
> >> http://about.me/kkrugler
> >> +1 530-210-6378
> >>
> >>
>
> 
> http://about.me/kkrugler
> +1 530-210-6378
>
>

Re: Documentation glitch w/AsyncFunction?

[Ted Yu]

bq.  resultFuture.thenAccept( (String result) -> {

I think the type of variable for the above call should be CompletableFuture.
Meaning, the variable currently named resultFuture should be renamed so
that the intention is clearer.

bq.  resultFuture.complete(Collections.singleton(new
Tuple2<>(str, result)));

Looking at existing code in unit tests, the complete() call is on the
parameter.

Cheers

On Sat, Apr 14, 2018 at 8:34 AM, Ken Krugler 
wrote:

> Hi devs,
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/
> operators/asyncio.html  release-1.4/dev/stream/operators/asyncio.html>
>
> Has this example of asyncInvoke:
>
> > @Override
> > public void asyncInvoke(final String str, final
> ResultFuture> resultFuture) throws Exception {
> >
> > // issue the asynchronous request, receive a future for result
> > Future resultFuture = client.query(str);
> >
> > // set the callback to be executed once the request by the
> client is complete
> > // the callback simply forwards the result to the result future
> > resultFuture.thenAccept( (String result) -> {
> >
> > resultFuture.complete(Collections.singleton(new
> Tuple2<>(str, result)));
> >
> > });
> > }
>
> 1. there’s a resultFuture parameter, and a resultFuture variable.
>
> 2. resultFuture.thenAccept() is a method available for CompletableFuture <
> https://docs.oracle.com/javase/8/docs/api/java/util/
> concurrent/CompletableFuture.html>, not Future.
>
> I can fix this up, but I’m wondering what you think the code should do,
> assuming there’s a typical client that returns a Future vs. a
> CompletableFuture.
>
> e.g. I could use CompletableFuture.supplyAsync(new Supplier() { }
> with a get that calls the Future’s get().
>
> Thanks,
>
> — Ken
>
> 
> http://about.me/kkrugler
> +1 530-210-6378
>
>

[jira] [Created] (FLINK-9150) Prepare for Java 10

[Ted Yu (JIRA)]

Ted Yu created FLINK-9150:
-

 Summary: Prepare for Java 10
 Key: FLINK-9150
 URL: https://issues.apache.org/jira/browse/FLINK-9150
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


When compiling with Java 10, I see the following compilation error:
{code}
[ERROR] Failed to execute goal on project flink-shaded-hadoop2: Could not 
resolve dependencies for project 
org.apache.flink:flink-shaded-hadoop2:jar:1.6-SNAPSHOT: Could not find artifact 
jdk.tools:jdk.tools:jar:1.6 at specified path /a/jdk-10/../lib/tools.jar -> 
[Help 1]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: TaskManager deadlock on NetworkBufferPool

[Ted Yu]

I searched for 0x0005e28fe218 in the two files you attached
to FLINK-2685 but didn't find any hit.

Was this the same instance as the attachment to FLINK-2685 ?

Thanks

On Wed, Apr 4, 2018 at 10:21 AM, Amit Jain  wrote:

> +u...@flink.apache.org
>
> On Wed, Apr 4, 2018 at 11:33 AM, Amit Jain  wrote:
> > Hi,
> >
> > We are hitting TaskManager deadlock on NetworkBufferPool bug in Flink
> 1.3.2.
> > We have set of ETL's merge jobs for a number of tables and stuck with
> above
> > issue randomly daily.
> >
> > I'm attaching the thread dump of JobManager and one of the Task Manager
> (T1)
> > running stuck job.
> > We also observed, sometimes new job scheduled on T1 progresses even
> another
> > job is stuck there.
> >
> > "CHAIN DataSource (at createInput(ExecutionEnvironment.java:553)
> > (org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormat)) -> Map
> (Map
> > at main(MergeTableSecond.java:175)) -> Map (Key Extractor) (6/9)" #1501
> > daemon prio=5 os_prio=0 tid=0x7f9ea84d2fb0 nid=0x22fe in
> Object.wait()
> > [0x7f9ebf102000]
> >java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at
> > org.apache.flink.runtime.io.network.buffer.
> LocalBufferPool.requestBuffer(LocalBufferPool.java:224)
> > - locked <0x0005e28fe218> (a java.util.ArrayDeque)
> > at
> > org.apache.flink.runtime.io.network.buffer.LocalBufferPool.
> requestBufferBlocking(LocalBufferPool.java:193)
> > at
> > org.apache.flink.runtime.io.network.api.writer.
> RecordWriter.sendToTarget(RecordWriter.java:132)
> > - locked <0x0005e29125f0> (a
> > org.apache.flink.runtime.io.network.api.serialization.
> SpanningRecordSerializer)
> > at
> > org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(
> RecordWriter.java:89)
> > at
> > org.apache.flink.runtime.operators.shipping.OutputCollector.collect(
> OutputCollector.java:65)
> > at
> > org.apache.flink.runtime.operators.util.metrics.
> CountingCollector.collect(CountingCollector.java:35)
> > at
> > org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(
> ChainedMapDriver.java:79)
> > at
> > org.apache.flink.runtime.operators.util.metrics.
> CountingCollector.collect(CountingCollector.java:35)
> > at
> > org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(
> ChainedMapDriver.java:79)
> > at
> > org.apache.flink.runtime.operators.util.metrics.
> CountingCollector.collect(CountingCollector.java:35)
> > at
> > org.apache.flink.runtime.operators.DataSourceTask.
> invoke(DataSourceTask.java:168)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > --
> > Thanks,
> > Amit
>

Re: Using Slack for online discussions

[Ted Yu]

bq. A bot could archive the messages to a web site, or forward to the email
list

I think some formatting / condensing may be needed if communication on
Slack is forwarded to mailing list - since the sentences on Slack may not
be as polished as on the emails.
There is also the mapping between a person's identity on Slack versus on
email.

FYI

On Tue, Apr 3, 2018 at 7:56 AM, TechnoMage <mla...@technomage.com> wrote:

> We use Slack in many contexts, company, community, etc.  It has many
> advantages over email.  For one being a separate channel from general email
> it stands out when there are new messages.  Notifications can be configured
> separately for each channel, and can arrive on multiple mobile devices with
> synchronization between them.  A bot could archive the messages to a web
> site, or forward to the email list.  It also allows upload of code snippets
> with formatting and voice/screen sharing where appropriate.  I would love
> to see it a supported platform.
>
> Michael
>
> > On Apr 3, 2018, at 7:52 AM, Thomas Weise <t...@apache.org> wrote:
> >
> > The invite link is self service. Everyone can signup.
> >
> > As for the searchable record, it may be possible to archive what's posted
> > on the slack channel by subscribing the mailing list.
> >
> > I think a communication platform like Slack or IRC complements email, the
> > type of messages there would typically be different from email threads.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, Apr 3, 2018 at 7:37 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> >> It is the lack of searchable public record (of Slack) that we should be
> >> concerned with.
> >>
> >> Also, requiring invitation would be bottleneck for the growth of
> >> participants.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 3, 2018 at 6:11 AM, Till Rohrmann <trohrm...@apache.org>
> >> wrote:
> >>
> >>> I'm a bit torn here. On the one hand I think Slack would be nice
> because
> >> it
> >>> allows a more direct interaction. Similar to the IRC channel we once
> had.
> >>>
> >>> On the other hand, I fear that some information/discussions might get
> >> lost
> >>> in the depths of Slack and at least after the 1 message limit has
> >> been
> >>> reached. Posting these things on the ML allows to persist the
> information
> >>> in the ML archives. Moreover, it would discourage to some extent the
> >> usage
> >>> of the ML in general which is not in the sense of the ASF.
> >>>
> >>> The problem that only ASF committers have access to the ASF slack
> channel
> >>> can be solved by an explicit invite for everyone [1].
> >>>
> >>> [1] https://s.apache.org/slack-invite
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>>
> >>> On Tue, Apr 3, 2018 at 3:09 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>>
> >>>> Thanks for the background information.
> >>>> I withdraw previous +1
> >>>>  Original message From: Chesnay Schepler <
> >>>> ches...@apache.org> Date: 4/3/18  4:50 AM  (GMT-08:00) To:
> >>>> dev@flink.apache.org Subject: Re: Using Slack for online discussions
> >>>> -1
> >>>>
> >>>> 1. According to INFRA-14292
> >>>><https://issues.apache.org/jira/browse/INFRA-14292> the ASF Slack
> >>>>isn't run by the ASF. This alone puts this service into rather
> >>>>questionable territory as it /looks/ like an official ASF service.
> >>>>If anyone can provide information to the contrary, please do so.
> >>>> 2. We already discuss things on the mailing lists, JIRA and GitHub.
> >> All
> >>>>of these are available to the public, whereas the slack channel
> >>>>requires an @apache mail address, i.e. you have to be a committer.
> >>>>This minimizes the target audience rather significantly. I would
> >>>>much rather prefer something that is also available to
> >> contributors.
> >>>>
> >>>> On 30.03.2018 10:17, Ted Yu wrote:
> >>>>> +1
> >>>>>  Original message From: Jean-Baptiste Onofré <
> >>>> j...@nanthrax.net> Date: 3/30/18  12:43 AM  (GMT-08:00) To:
> >>>> dev@flink.apache.org Subject: Re: Using Slack for online discussions
> >>>>> +1
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>> On 03/29/2018 07:22 PM, Thomas Weise wrote:
> >>>>>> How does the community think about using ASF Slack for online
> >>>> discussions?
> >>>>>>
> >>>>>> The Beam project started to use it recently and it seems to work
> >> quite
> >>>> well
> >>>>>> [1] [2].
> >>>>>>
> >>>>>> [1]
> >>>>>> https://lists.apache.org/thread.html/1bee60193823a8411dcfad7a6de695
> >>>> ac1d779ae5d14f7b954ffa5961@%3Cdev.beam.apache.org%3E
> >>>>>> [2]
> >>>>>> https://lists.apache.org/thread.html/b82871dc16b63a4e74cd6373405361
> >>>> dfbd75810073f98542ba82ed41@%3Cdev.beam.apache.org%3E
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Thomas
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Using Slack for online discussions

[Ted Yu]

It is the lack of searchable public record (of Slack) that we should be
concerned with.

Also, requiring invitation would be bottleneck for the growth of
participants.

Cheers

On Tue, Apr 3, 2018 at 6:11 AM, Till Rohrmann <trohrm...@apache.org> wrote:

> I'm a bit torn here. On the one hand I think Slack would be nice because it
> allows a more direct interaction. Similar to the IRC channel we once had.
>
> On the other hand, I fear that some information/discussions might get lost
> in the depths of Slack and at least after the 1 message limit has been
> reached. Posting these things on the ML allows to persist the information
> in the ML archives. Moreover, it would discourage to some extent the usage
> of the ML in general which is not in the sense of the ASF.
>
> The problem that only ASF committers have access to the ASF slack channel
> can be solved by an explicit invite for everyone [1].
>
> [1] https://s.apache.org/slack-invite
>
> Cheers,
> Till
>
>
> On Tue, Apr 3, 2018 at 3:09 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Thanks for the background information.
> > I withdraw previous +1
> >  Original message From: Chesnay Schepler <
> > ches...@apache.org> Date: 4/3/18  4:50 AM  (GMT-08:00) To:
> > dev@flink.apache.org Subject: Re: Using Slack for online discussions
> > -1
> >
> >  1. According to INFRA-14292
> > <https://issues.apache.org/jira/browse/INFRA-14292> the ASF Slack
> > isn't run by the ASF. This alone puts this service into rather
> > questionable territory as it /looks/ like an official ASF service.
> > If anyone can provide information to the contrary, please do so.
> >  2. We already discuss things on the mailing lists, JIRA and GitHub. All
> > of these are available to the public, whereas the slack channel
> > requires an @apache mail address, i.e. you have to be a committer.
> >     This minimizes the target audience rather significantly. I would
> > much rather prefer something that is also available to contributors.
> >
> > On 30.03.2018 10:17, Ted Yu wrote:
> > > +1
> > >  Original message From: Jean-Baptiste Onofré <
> > j...@nanthrax.net> Date: 3/30/18  12:43 AM  (GMT-08:00) To:
> > dev@flink.apache.org Subject: Re: Using Slack for online discussions
> > > +1
> > >
> > > Regards
> > > JB
> > >
> > > On 03/29/2018 07:22 PM, Thomas Weise wrote:
> > >> How does the community think about using ASF Slack for online
> > discussions?
> > >>
> > >> The Beam project started to use it recently and it seems to work quite
> > well
> > >> [1] [2].
> > >>
> > >> [1]
> > >> https://lists.apache.org/thread.html/1bee60193823a8411dcfad7a6de695
> > ac1d779ae5d14f7b954ffa5961@%3Cdev.beam.apache.org%3E
> > >> [2]
> > >> https://lists.apache.org/thread.html/b82871dc16b63a4e74cd6373405361
> > dfbd75810073f98542ba82ed41@%3Cdev.beam.apache.org%3E
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> >
> >
>

Re: Using Slack for online discussions

[Ted Yu]

Thanks for the background information. 
I withdraw previous +1
 Original message From: Chesnay Schepler <ches...@apache.org> 
Date: 4/3/18  4:50 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: Using 
Slack for online discussions 
-1

 1. According to INFRA-14292
    <https://issues.apache.org/jira/browse/INFRA-14292> the ASF Slack
    isn't run by the ASF. This alone puts this service into rather
    questionable territory as it /looks/ like an official ASF service.
    If anyone can provide information to the contrary, please do so.
 2. We already discuss things on the mailing lists, JIRA and GitHub. All
    of these are available to the public, whereas the slack channel
    requires an @apache mail address, i.e. you have to be a committer.
    This minimizes the target audience rather significantly. I would
    much rather prefer something that is also available to contributors.

On 30.03.2018 10:17, Ted Yu wrote:
> +1
>  Original message From: Jean-Baptiste Onofré 
> <j...@nanthrax.net> Date: 3/30/18  12:43 AM  (GMT-08:00) To: 
> dev@flink.apache.org Subject: Re: Using Slack for online discussions
> +1
>
> Regards
> JB
>
> On 03/29/2018 07:22 PM, Thomas Weise wrote:
>> How does the community think about using ASF Slack for online discussions?
>>
>> The Beam project started to use it recently and it seems to work quite well
>> [1] [2].
>>
>> [1]
>> https://lists.apache.org/thread.html/1bee60193823a8411dcfad7a6de695ac1d779ae5d14f7b954ffa5961@%3Cdev.beam.apache.org%3E
>> [2]
>> https://lists.apache.org/thread.html/b82871dc16b63a4e74cd6373405361dfbd75810073f98542ba82ed41@%3Cdev.beam.apache.org%3E
>>
>> Thanks,
>> Thomas
>>

Re: Using Slack for online discussions

[Ted Yu]

+1
 Original message From: Jean-Baptiste Onofré 
 Date: 3/30/18  12:43 AM  (GMT-08:00) To: 
dev@flink.apache.org Subject: Re: Using Slack for online discussions 
+1

Regards
JB

On 03/29/2018 07:22 PM, Thomas Weise wrote:
> How does the community think about using ASF Slack for online discussions?
> 
> The Beam project started to use it recently and it seems to work quite well
> [1] [2].
> 
> [1]
> https://lists.apache.org/thread.html/1bee60193823a8411dcfad7a6de695ac1d779ae5d14f7b954ffa5961@%3Cdev.beam.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/b82871dc16b63a4e74cd6373405361dfbd75810073f98542ba82ed41@%3Cdev.beam.apache.org%3E
> 
> Thanks,
> Thomas
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[jira] [Created] (FLINK-9091) Failure while enforcing releasability in building flink-json module

[Ted Yu (JIRA)]

Ted Yu created FLINK-9091:
-

 Summary: Failure while enforcing releasability in building 
flink-json module
 Key: FLINK-9091
 URL: https://issues.apache.org/jira/browse/FLINK-9091
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
 Attachments: f-json.out

Got the following when building flink-json module:
{code}
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
failed with message:
Failed while enforcing releasability. See above detailed error message.
...
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce 
(dependency-convergence) on project flink-json: Some Enforcer rules have 
failed.   Look above for specific messages explaining why the rule failed. -> 
[Help 1]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9090) Replace UUID.randomUUID with deterministic PRNG

[Ted Yu (JIRA)]

Ted Yu created FLINK-9090:
-

 Summary: Replace UUID.randomUUID with deterministic PRNG
 Key: FLINK-9090
 URL: https://issues.apache.org/jira/browse/FLINK-9090
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Currently UUID.randomUUID is called in various places in the code base.

* It is non-deterministic.
* It uses a single secure random for UUID generation. This uses a single JVM 
wide lock, and this can lead to lock contention and other performance problems.

We should move to something that is deterministic by using seeded PRNGs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9089) Upgrade Orc dependency to 1.4.3

[Ted Yu (JIRA)]

Ted Yu created FLINK-9089:
-

 Summary: Upgrade Orc dependency to 1.4.3
 Key: FLINK-9089
 URL: https://issues.apache.org/jira/browse/FLINK-9089
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Currently flink-orc uses Orc 1.4.1 release.

This issue upgrades to Orc 1.4.3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9088) Upgrade Nifi connector dependency to 1.6.0

[Ted Yu (JIRA)]

Ted Yu created FLINK-9088:
-

 Summary: Upgrade Nifi connector dependency to 1.6.0
 Key: FLINK-9088
 URL: https://issues.apache.org/jira/browse/FLINK-9088
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Currently dependency of Nifi is 0.6.1

We should upgrade to 1.6.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9087) Return value of broadcastEvent should be closed in StreamTask#performCheckpoint

[Ted Yu (JIRA)]

Ted Yu created FLINK-9087:
-

 Summary: Return value of broadcastEvent should be closed in 
StreamTask#performCheckpoint
 Key: FLINK-9087
 URL: https://issues.apache.org/jira/browse/FLINK-9087
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
for (StreamRecordWriter<SerializationDelegate<StreamRecord>> 
streamRecordWriter : streamRecordWriters) {
  try {
streamRecordWriter.broadcastEvent(message);
{code}
The BufferConsumer returned by broadcastEvent() should be closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Flip 6 mesos support

[Ted Yu]

Renjie:The log didn't go through.
Consider logging a JIRA and attach the log there. 
Thanks
 Original message From: Renjie Liu  
Date: 3/23/18  1:38 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: Flip 
6 mesos support 
Hi, Till:Attached is my log.
I'm also looking into this, could you please assign this bug to me? I'm also 
trying to contribute to flink.

On Fri, Mar 23, 2018 at 4:11 PM Till Rohrmann  wrote:
HI Renjie, could you share the logs with us? This sounds like a bug we

should fix.



Cheers,

Till



On Fri, Mar 23, 2018 at 4:42 AM, Renjie Liu  wrote:



> Hi, Till:

> Has anybody succeeded to deploy flip 6 mode on mesos?

>

> I'm testing flip 6 using the master branch and I just can't run jobs. The

> following are my configurations:

>

> *jobmanager.rpc.address: qt9ss.prod.mediav.com

> *

> *jobmanager.rpc.port: 6123*

> *jobmanager.heap.mb: 1024*

> *taskmanager.heap.mb: 1024*

> *taskmanager.numberOfTaskSlots: 5*

> *parallelism.default: 1*

> *web.port: 8081*

> *mesos.master: zk://dk71ss.jx.shbt2.qihoo.net:2191

> ,dk72ss.jx.shbt2.qihoo.net:2191

> ,dk5ss.jx.shbt2.

> qihoo.net:2191/mesos

> *

> *mesos.resourcemanager.tasks.container.type: docker*

> *mesos.resourcemanager.tasks.container.image.name

> :

> dk1ss.prod.mediav.com:5000/adq/flink:1.6.0-SNAPSHOT

> *

> *mesos.resourcemanager.framework.user: mediav*

> *mesos.resourcemanager.tasks.cpus: 5*

> *mesos.resourcemanager.tasks.mem: 10240*

> *mesos.resourcemanager.framework.name

> : Flink*

> *mesos.failover-timeout: 60*

>

> From the mesos side, I can see that when I submit a job, flink master will

> request a contianer with 5 cores. But the job submission still fails the

> following error:

> *org.apache.flink.runtime.jobmanager.scheduler.

> NoResourceAvailableException:

> Could not allocate all requires slots within timeout of 30 ms. Slots

> required: 1, slots allocated: 0*

>

> My job only requires 1 slot but job manager keeps reporting that no slots

> avaiable.

>

> On Wed, Mar 21, 2018 at 10:42 PM Till Rohrmann 

> wrote:

>

> > The resources consumed by the JobMaster can be specified by

> > `jobmanager.heap.mb`.

> >

> > Cheers,

> > Till

> >

> > On Wed, Mar 21, 2018 at 3:20 PM, Renjie Liu 

> > wrote:

> >

> > > Hi, Till:

> > >

> > > In fact, I want to ask the resources consume by job manager

> > >

> > > Till Rohrmann  于 2018年3月21日周三 下午8:17写道：

> > >

> > > > As many as the application needs to run. If you start a job with

> > > > parallelism 10 then it will ask for 10 slots (assuming slot sharing).

> > > >

> > > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu <

> liurenjie2...@gmail.com>

> > > > wrote:

> > > >

> > > > > So how many slots a job manager may consume?

> > > > >

> > > > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann <

> trohrm...@apache.org>

> > > > > wrote:

> > > > >

> > > > > > At the moment this is not possible. In order to do this, you will

> > > have

> > > > to

> > > > > > use the per-job mode and run each job on a dedicated Flink

> cluster.

> > > > > >

> > > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <

> > > liurenjie2...@gmail.com>

> > > > > > wrote:

> > > > > >

> > > > > > > For example, we have 2 jobs.

> > > > > > > For job 1, I want to start job manger with 1 CPU and 100M

> memory.

> > > > Job 1

> > > > > > > need s10 slots, and I want to deploy these 10 slots in 2 task

> > > > managers,

> > > > > > > each with 5 cores and 1G memory.

> > > > > > >

> > > > > > > For job 2, I want to start job manager with 2 CPU and 200M

> > memory.

> > > > Job

> > > > > 2

> > > > > > > needs 100 slots and I want to deploy these 100 slot in 10 task

> > > > > managers,

> > > > > > > each with 10 cores and 2G memory.

> > > > > > >

> > > > > > > Is this possible?

> > > > > > >

> > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <

> > > trohrm...@apache.org>

> > > > > > > wrote:

> > > > > > >

> > > > > > > > Hi Renjie,

> > > > > > > >

> > > > > > > > what do you mean with specifying different JM and TM

> resources

> > > for

> > > > > > > > different jobs exactly?

> > > > > > > >

> > > > > > > > Cheers,

> > > > > > > > Till

> > > > > > > >

> > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu <

> > > > > liurenjie2...@gmail.com>

> > > > > > > > wrote:

> > > > > > > >

> > > > > > > > > Hi, Till:

> > > > > > > > >

> > > > > > > > > How to specify job manager and task manager resources for

> > > > different

> > > > > > > jobs

> > > > > > > > in

> 

[jira] [Created] (FLINK-9048) LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers sometimes fails

[Ted Yu (JIRA)]

Ted Yu created FLINK-9048:
-

 Summary: 
LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers 
sometimes fails
 Key: FLINK-9048
 URL: https://issues.apache.org/jira/browse/FLINK-9048
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu


As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
  Time elapsed: 41.681 sec  <<< FAILURE!
java.lang.AssertionError: Thread 
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini 
cluster, but not shut down
  at org.junit.Assert.fail(Assert.java:88)
  at 
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Too many open files

[Ted Yu]

Not sure if you have looked at FLINK-8707

FYI

On Tue, Mar 20, 2018 at 2:13 PM, Govindarajan Srinivasaraghavan <
govindragh...@gmail.com> wrote:

> Hi,
>
> We have a streaming job that runs on flink in docker and checkpointing
> happens every 10 seconds. After several starts and cancellations we are
> facing this issue with file handles.
>
> The job reads data from kafka, processes it and writes it back to kafka and
> we are using RocksDB state backend. For now we have increased the number
> file handles to resolve the problem but would like to know if this is
> expected or is it an issue. Thanks.
>
> java.io.FileNotFoundException:
> /tmp/flink-io-b3043cd6-50c8-446a-8c25-fade1b1862c0/
> cb317fc2578db72b3046468948fa00f2f17039b6104e72fb8c58938e5869cfbc.0.buffer
> (Too many open files)
>
> at java.io.RandomAccessFile.open0(Native Method)
>
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
>
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
>
> at
> org.apache.flink.streaming.runtime.io.BufferSpiller.createSpillingChannel(
> BufferSpiller.java:259)
>
> at
> org.apache.flink.streaming.runtime.io.BufferSpiller.<
> init>(BufferSpiller.java:120)
>
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.<
> init>(BarrierBuffer.java:149)
>
> at
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.
> (StreamTwoInputProcessor.java:147)
>
> at
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.init(
> TwoInputStreamTask.java:79)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:235)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Regards,
> Govind
>

[jira] [Created] (FLINK-9019) Unclosed closeableRegistry in StreamTaskStateInitializerImpl#rawOperatorStateInputs

[Ted Yu (JIRA)]

Ted Yu created FLINK-9019:
-

 Summary: Unclosed closeableRegistry in 
StreamTaskStateInitializerImpl#rawOperatorStateInputs
 Key: FLINK-9019
 URL: https://issues.apache.org/jira/browse/FLINK-9019
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
 final CloseableRegistry closeableRegistry = new CloseableRegistry();
...
 if (rawOperatorState != null) {
...
  }
}

return CloseableIterable.empty();
{code}
If rawOperatorState is null, closeableRegistry would be left unclosed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-9018) Unclosed snapshotCloseableRegistry in RocksDBKeyedStateBackend#FullSnapshotStrategy#performSnapshot

[Ted Yu (JIRA)]

Ted Yu created FLINK-9018:
-

 Summary: Unclosed snapshotCloseableRegistry in 
RocksDBKeyedStateBackend#FullSnapshotStrategy#performSnapshot
 Key: FLINK-9018
 URL: https://issues.apache.org/jira/browse/FLINK-9018
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
  final CloseableRegistry snapshotCloseableRegistry = new 
CloseableRegistry();

  if (kvStateInformation.isEmpty()) {
if (LOG.isDebugEnabled()) {
  LOG.debug("Asynchronous RocksDB snapshot performed on empty keyed 
state at {}. Returning null.",
timestamp);
}

return DoneFuture.of(SnapshotResult.empty());
  }
{code}
If the method returns in the above if block, snapshotCloseableRegistry is not 
closed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0

[Ted Yu]

+1 on marking bugs as fixed for 1.5.0 only.
 Original message From: Piotr Nowojski 
 Date: 3/15/18  12:48 AM  (GMT-08:00) To: 
dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as resolved 
in 1.5.0 as resolved in 1.6.0 
Same as Chesnay

+1 for marking bugs as fixed 1.5.0 only

> On 15 Mar 2018, at 07:57, Chesnay Schepler  wrote:
> 
> +1 to mark bugs as fixed in 1.5.0 only.
> 
> On 15.03.2018 01:40, Aljoscha Krettek wrote:
>> Hi,
>> 
>> We currently have some issues that are marked as resolved for both 1.5.0 and 
>> 1.6.0 [1]. The reason is that we have the release-1.5 branch and the master 
>> branch, which will eventually become the branch for 1.6.0.
>> 
>> I think this can lead to confusion because the release notes are created 
>> based on that data. Say, we fix a bug "foo" after we created the release-1.5 
>> branch. Now we will have "[FLINK-] Fixed foo" in the release notes for 
>> 1.5.0 and 1.6.0. We basically start our Flink 1.6.0 release notes with 
>> around 50 issues that were never bugs in 1.6.0 because they were fixed in 
>> 1.5.0. Plus, having "[FLINK-] Fixed foo" in the 1.6.0 release notes 
>> indicates that "foo" was actually a bug in 1.5.0 (because we now had to fix 
>> it), but it wasn't.
>> 
>> I would propose to remove fixVersion 1.6.0 from all issues that have 1.5.0 
>> as fixVersion. What do you think?
>> 
>> On a side note: a bug that is fixed in 1.5.1 should be marked as fixed for 
>> 1.6.0 separately, because 1.6.0 is not a direct successor to 1.5.1.
>> 
>> Best,
>> Aljoscha
>> 
>> [1] 
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20and%20fixVersion%20%3D%201.6.0%20and%20resolution%20!%3D%20unresolved
> 
> 

Re: [VOTE] Release 1.3.3, release candidate #2

[Ted Yu]

+1

Ran the following command - passed:

mvn clean package -Pjdk8

On Wed, Mar 14, 2018 at 3:26 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on release candidate #2 for Flink 1.3.3, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code branch “release-1.3.3-rc2” [5],
> * website pull request listing the new release [6].
> * A complete list of all new commits in release-1.3.3-rc2, since
> release-1.3.2 [7]
>
> This release candidate contains fixes for only the following issues:
> [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase
> [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.
> checkpointsMatch()
> [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in
> infinite loop
> [FLINK-7783] Don’t always remove checkpoints in
> ZooKeeperCompletedCheckpointStore#recover()
>
> Since the last candidate was cancelled only due to incorrect binaries in
> the source artifacts, I think we can also have a shorter voting period for
> RC2.
>
> Please test the release and vote for the release candidate before Thursday
> (March 15th), 7pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Please let me know if you disagree with the shortened voting time.
>
> Thanks,
> Gordon
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341142
> [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc2/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1151
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=
> shortlog;h=refs/heads/release-1.3.3-rc2
> [6] https://github.com/apache/flink-web/pull/104
> [7]
> - 90559b5413455d9d0f2b61c389a60e26e5c87800 [hotfix] Properly delete temp
> flink dir in create_source_release.sh
> - 99c0353a34c09af5bedb73f525f691dd7e78fcdd [hotfix] Ensure pristine
> release in tools/releasing/create_source_release.sh
> - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper
> checkpoint store behaviour with ITCase
> - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare
> checkpoints with order in CompletedCheckpoint.checkpointsMatch()
> - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix
> ZookeeperCompleted checkpoint store can get stuck in infinite loop
> - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always
> remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
>
>

[jira] [Created] (FLINK-8933) Avoid calling Class#newInstance

[Ted Yu (JIRA)]

Ted Yu created FLINK-8933:
-

 Summary: Avoid calling Class#newInstance
 Key: FLINK-8933
 URL: https://issues.apache.org/jira/browse/FLINK-8933
 Project: Flink
  Issue Type: Task
Reporter: Ted Yu


Class#newInstance is deprecated starting in Java 9 - 
https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
undeclared checked exceptions.

The suggested replacement is getDeclaredConstructor().newInstance(), which 
wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.3.3, release candidate #1

[Ted Yu]

+1

Ran the following command which passed:

mvn clean package -Pjdk8

On Mon, Mar 12, 2018 at 9:09 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on release candidate #1 for Flink 1.3.3, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code branch “release-1.3.3-rc1” [5],
> * website pull request listing the new release [6].
> * A complete list of all new commits in release-1.3.3-rc1, since
> release-1.3.2 [7]
>
> This release candidate contains fixes for only the following issues:
> [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase
> [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.
> checkpointsMatch()
> [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in
> infinite loop
> [FLINK-7783] Don’t always remove checkpoints in
> ZooKeeperCompletedCheckpointStore#recover()
>
> Since we’re aiming for a minimal release candidate for 1.3.3 with only
> fixes for the proposed issues. I propose that we have a shortened voting
> time of 2 days. Please let me know if you disagree with this.
>
> The vote will be open for at least 48 hours.
> Please test the release and vote for the release candidate before
> Wednesday (March 14th), 7pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341142
> [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1150
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=
> shortlog;h=refs/heads/release-1.3.3-rc1
> [6] https://github.com/apache/flink-web/pull/104
> [7]
> - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper
> checkpoint store behaviour with ITCase
> - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare
> checkpoints with order in CompletedCheckpoint.checkpointsMatch()
> - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix
> ZookeeperCompleted checkpoint store can get stuck in infinite loop
> - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always
> remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
>
>

[jira] [Created] (FLINK-8931) TASK_KILLING is not covered by match in TaskMonitor#whenUnhandled

[Ted Yu (JIRA)]

Ted Yu created FLINK-8931:
-

 Summary: TASK_KILLING is not covered by match in 
TaskMonitor#whenUnhandled
 Key: FLINK-8931
 URL: https://issues.apache.org/jira/browse/FLINK-8931
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Noticed the following :
{code}
[WARNING] 
/a/flink-1.3.3/flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/TaskMonitor.scala:157:
 warning: match may not be exhaustive.
[WARNING] It would fail on the following input: TASK_KILLING
[WARNING]   msg.status().getState match {
[WARNING]^
[WARNING] 
/a/flink-1.3.3/flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/TaskMonitor.scala:170:
 warning: match may not be exhaustive.
[WARNING] It would fail on the following input: TASK_KILLING
[WARNING]   msg.status().getState match {
[WARNING]^
{code}
It seems that TASK_KILLING should be covered by the last case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release 1.3.3, release candidate #1

[Ted Yu]

When I used the following command:

mvn package -Pscala-2.11,jdk8

I got:

[ERROR] Failed to execute goal on project flink-scala_2.10: Could not
resolve dependencies for project
org.apache.flink:flink-scala_2.10:jar:1.3.3: Could not find artifact
org.scalamacros:quasiquotes_2.11:jar:2.0.1 in central (
https://repo.maven.apache.org/maven2) -> [Help 1]

Has anyone else seen the above ?

Cheers

On Mon, Mar 12, 2018 at 9:09 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on release candidate #1 for Flink 1.3.3, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code branch “release-1.3.3-rc1” [5],
> * website pull request listing the new release [6].
> * A complete list of all new commits in release-1.3.3-rc1, since
> release-1.3.2 [7]
>
> This release candidate contains fixes for only the following issues:
> [FLINK-8487] Verify ZooKeeper checkpoint store behaviour with ITCase
> [FLINK-8890] Compare checkpoints with order in CompletedCheckpoint.
> checkpointsMatch()
> [FLINK-8807] Fix ZookeeperCompleted checkpoint store can get stuck in
> infinite loop
> [FLINK-7783] Don’t always remove checkpoints in
> ZooKeeperCompletedCheckpointStore#recover()
>
> Since we’re aiming for a minimal release candidate for 1.3.3 with only
> fixes for the proposed issues. I propose that we have a shortened voting
> time of 2 days. Please let me know if you disagree with this.
>
> The vote will be open for at least 48 hours.
> Please test the release and vote for the release candidate before
> Wednesday (March 14th), 7pm CET.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12315522=12341142
> [2] http://people.apache.org/~tzulitai/flink-1.3.3-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1150
> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=
> shortlog;h=refs/heads/release-1.3.3-rc1
> [6] https://github.com/apache/flink-web/pull/104
> [7]
> - b2437f87e361a822adbad6f1c3e6eb14eeeb09fa [FLINK-8487] Verify ZooKeeper
> checkpoint store behaviour with ITCase
> - 1432092f29c548c55af562ff7b4a7973fedd2e22 [FLINK-8890] Compare
> checkpoints with order in CompletedCheckpoint.checkpointsMatch()
> - df37d7acfea10a5ca3186f3c53294f2050758627 [FLINK-8807] Fix
> ZookeeperCompleted checkpoint store can get stuck in infinite loop
> - f69bdf207b92ca47a5ce3e29f6ec7193ed17ec72 [FLINK-7783] Don’t always
> remove checkpoints in ZooKeeperCompletedCheckpointStore#recover()
>
>

Re: [DISCUSS] Releasing Flink 1.3.3 with only critical fixes

[Ted Yu]

+1 to Aljoscha's proposal.

[jira] [Created] (FLINK-8771) Upgrade scalastyle to 1.0.0

[Ted Yu (JIRA)]

Ted Yu created FLINK-8771:
-

 Summary: Upgrade scalastyle to 1.0.0
 Key: FLINK-8771
 URL: https://issues.apache.org/jira/browse/FLINK-8771
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: Ted Yu


scalastyle 1.0.0 fixes issue with import order, explicit type for public 
methods, line length limitation and comment validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8708) Unintended integer division in StandaloneThreadedGenerator

[Ted Yu (JIRA)]

Ted Yu created FLINK-8708:
-

 Summary: Unintended integer division in StandaloneThreadedGenerator
 Key: FLINK-8708
 URL: https://issues.apache.org/jira/browse/FLINK-8708
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


In 
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/statemachine/generator/StandaloneThreadedGenerator.java
 :
{code}
double factor = (ts - lastTimeStamp) / 1000;
{code}
Proper casting should be done before the integer division



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8693) initPythonInterpreter is called twice with same arguments in InterpreterUtils#initAndExecPythonScript

[Ted Yu (JIRA)]

Ted Yu created FLINK-8693:
-

 Summary: initPythonInterpreter is called twice with same arguments 
in InterpreterUtils#initAndExecPythonScript
 Key: FLINK-8693
 URL: https://issues.apache.org/jira/browse/FLINK-8693
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
initPythonInterpreter(fullArgs, scriptDirectory.toUri().getPath(), 
scriptName);

PythonInterpreter pythonInterpreter = initPythonInterpreter(fullArgs, 
scriptDirectory.toUri().getPath(), scriptName);
{code}
The two calls to initPythonInterpreter() have the same parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8687) MaterializedCollectStreamResult#retrievePage should take resultLock

[Ted Yu (JIRA)]

Ted Yu created FLINK-8687:
-

 Summary: MaterializedCollectStreamResult#retrievePage should take 
resultLock
 Key: FLINK-8687
 URL: https://issues.apache.org/jira/browse/FLINK-8687
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Currently MaterializedCollectStreamResult#retrievePage checks page range and 
calls snapshot.subList() without holding resultLock.

resultLock should be taken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8554) Upgrade AWS SDK

[Ted Yu (JIRA)]

Ted Yu created FLINK-8554:
-

 Summary: Upgrade AWS SDK
 Key: FLINK-8554
 URL: https://issues.apache.org/jira/browse/FLINK-8554
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


AWS SDK 1.11.271 fixes a lot of bugs.

One of which would exhibit the following:
{code}
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8464) TestRaftReconfigurationWithSimulatedRpc fails with assertion error

[Ted Yu (JIRA)]

Ted Yu created FLINK-8464:
-

 Summary: TestRaftReconfigurationWithSimulatedRpc fails with 
assertion error
 Key: FLINK-8464
 URL: https://issues.apache.org/jira/browse/FLINK-8464
 Project: Flink
  Issue Type: Test
Reporter: Ted Yu
 Attachments: ratis-8464.tst

As of commit 7b3a9a6f5f8e8075727d84e3ddeae7b594eda89c, I observed the following 
:
{code}
testRevertConfigurationChange(org.apache.ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc)
  Time elapsed: 2.119 sec  <<< FAILURE!
java.lang.AssertionError: 1 0 expected: but was:
{code}
1 was confIndex and 0 was log.getLastCommittedIndex()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8415) Unprotected access to recordsToSend in LongRecordWriterThread#shutdown()

[Ted Yu (JIRA)]

Ted Yu created FLINK-8415:
-

 Summary: Unprotected access to recordsToSend in 
LongRecordWriterThread#shutdown()
 Key: FLINK-8415
 URL: https://issues.apache.org/jira/browse/FLINK-8415
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public void shutdown() {
running = false;
recordsToSend.complete(0L);
{code}
In other methods, access to recordsToSend is protected by synchronized keyword.

shutdown() should do the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8394) Lack of synchronization accessing expectedRecord in ReceiverThread#shutdown

[Ted Yu (JIRA)]

Ted Yu created FLINK-8394:
-

 Summary: Lack of synchronization accessing expectedRecord in 
ReceiverThread#shutdown
 Key: FLINK-8394
 URL: https://issues.apache.org/jira/browse/FLINK-8394
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public void shutdown() {
running = false;
interrupt();
expectedRecord.complete(0L);
{code}
Access to expectedRecord should be protected by synchronization, as done on 
other methods.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8335) Upgrade hbase connector dependency to 1.4.0

[Ted Yu (JIRA)]

Ted Yu created FLINK-8335:
-

 Summary: Upgrade hbase connector dependency to 1.4.0
 Key: FLINK-8335
 URL: https://issues.apache.org/jira/browse/FLINK-8335
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


hbase 1.4.0 has been released.

1.4.0 shows speed improvement over previous 1.x releases.

This issue is to upgrade the dependency to 1.4.0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8198) Useless check against -1 in LimitedConnectionsFileSystem#fromConfig

[Ted Yu (JIRA)]

Ted Yu created FLINK-8198:
-

 Summary: Useless check against -1 in 
LimitedConnectionsFileSystem#fromConfig
 Key: FLINK-8198
 URL: https://issues.apache.org/jira/browse/FLINK-8198
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
return new ConnectionLimitingSettings(
totalLimit == -1 ? 0 : totalLimit,
limitIn == -1 ? 0 : limitIn,
limitOut == -1 ? 0 : limitOut,
openTimeout,
{code}
If any of the 3 variables is negative, control would have returned in the if 
block above.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release 1.4.0, release candidate #1

[Ted Yu]

Long weekend should end this Sunday. 
Closing vote Wednesday would be great. 
Thanks
 Original message From: Aljoscha Krettek <aljos...@apache.org> 
Date: 11/24/17  5:34 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: 
[VOTE] Release 1.4.0, release candidate #1 
How long will the long weekend be? I thought about closing the vote on 
Wednesday, i.e. not count the weekend. Would that work?

Best,
Aljoscha

> On 24. Nov 2017, at 12:18, Ted Yu <yuzhih...@gmail.com> wrote:
> 
> Aljoscha:
> Thanks for spinning RC.
> 
> bq. The vote will be open for at least 72 hours
> 
> As you are aware, it is long weekend in US.
> 
> Is it possible to extend by 24 hours so that developers in US can
> participate in validation ?
> 
> Cheers
> 
> On Fri, Nov 24, 2017 at 2:57 AM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
> 
>> Hi everyone,
>> 
>> Please review and vote on release candidate #1 for the version 1.4.0, as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.4.0-rc1" [5],
>> * website pull request listing the new release [6].
>> 
>> Please have a careful look at the website PR because I changed some
>> wording and we're now also releasing a binary without Hadoop dependencies.
>> 
>> Please use this document for coordinating testing efforts: [7]
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Your friendly Release Manager
>> 
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12315522=12340533
>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1139
>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=
>> a0b322cf77851d3b8589812a0c8e443e9e320e67
>> [6] https://github.com/apache/flink-web/pull/95
>> [7] https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj
>> 4_CEmMTpCkY81s/edit?usp=sharing
>> 
>> Pro-tip: you can create a settings.xml file with these contents:
>> 
>> 
>> 
>>  flink-1.4.0
>> 
>> 
>>  
>>    flink-1.4.0
>>    
>>  
>>    flink-1.4.0
>>    
>>    https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>    
>>  
>>  
>>    archetype
>>    
>>    https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>    
>>  
>>    
>>  
>> 
>> 
>> 
>> And reference that in you maven commands via --settings
>> path/to/settings.xml. This is useful for creating a quickstart based on the
>> staged release and for building against the staged jars.

[jira] [Created] (FLINK-8146) Potential resource leak in PythonPlanBinder#unzipPythonLibrary

[Ted Yu (JIRA)]

Ted Yu created FLINK-8146:
-

 Summary: Potential resource leak in 
PythonPlanBinder#unzipPythonLibrary
 Key: FLINK-8146
 URL: https://issues.apache.org/jira/browse/FLINK-8146
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
while (entry != null) {
...
}
zis.closeEntry();
{code}
Looking at the catch block inside the loop, it seems the intention is to close 
zis upon getting exception.
zis.close() should be called outside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Adding a blog post about the 1.4/1.5 timeline and features

[Ted Yu]

+1

Looking forward to the new releases.

[jira] [Created] (FLINK-8100) Consider introducing log4j-extras

[Ted Yu (JIRA)]

Ted Yu created FLINK-8100:
-

 Summary: Consider introducing log4j-extras 
 Key: FLINK-8100
 URL: https://issues.apache.org/jira/browse/FLINK-8100
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


log4j-extras allows log rotation as well as compression.

https://logging.apache.org/log4j/extras/download.html

We should consider using log4j-extras.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8075) Redundant clone() in Configuration ctor

[Ted Yu (JIRA)]

Ted Yu created FLINK-8075:
-

 Summary: Redundant clone() in Configuration ctor
 Key: FLINK-8075
 URL: https://issues.apache.org/jira/browse/FLINK-8075
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


In  
flink-filesystems/flink-s3-fs-presto/src/main/java/org/apache/hadoop/conf/Configuration.java
 , at line 703:
{code}
  public Configuration(Configuration other) {
this.resources = (ArrayList) other.resources.clone();
synchronized(other) {
  if (other.properties != null) {
this.properties = (Properties)other.properties.clone();
  }
{code}
The first clone() call is without synchronization and without null check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8048) RAT check complaint

[Ted Yu (JIRA)]

Ted Yu created FLINK-8048:
-

 Summary: RAT check complaint
 Key: FLINK-8048
 URL: https://issues.apache.org/jira/browse/FLINK-8048
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


Running {{mvn rat:check}} gives warning about the following files:

test-infra/end-to-end-test/test-data/words
.editorconfig
.github/PULL_REQUEST_TEMPLATE.md
.github/CONTRIBUTING.md



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort

[Ted Yu (JIRA)]

Ted Yu created FLINK-8037:
-

 Summary: Missing cast in integer arithmetic in 
TransactionalIdsGenerator#generateIdsToAbort
 Key: FLINK-8037
 URL: https://issues.apache.org/jira/browse/FLINK-8037
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public Set generateIdsToAbort() {
Set idsToAbort = new HashSet<>();
for (int i = 0; i < safeScaleDownFactor; i++) {
  idsToAbort.addAll(generateIdsToUse(i * poolSize * totalNumberOfSubtasks));
{code}
The operands are integers where generateIdsToUse() expects long parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8036) Consider using gradle to build Flink

[Ted Yu (JIRA)]

Ted Yu created FLINK-8036:
-

 Summary: Consider using gradle to build Flink
 Key: FLINK-8036
 URL: https://issues.apache.org/jira/browse/FLINK-8036
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


Here is summary from Lukasz over this thread 
(http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool)
 w.r.t. performance boost from using gradle:

Maven performs parallelization at the module level, an entire module needs
to complete before any dependent modules can start, this means running all
the checks like findbugs, checkstyle, tests need to finish. Gradle has task
level parallelism between subprojects which means that as soon as the
compile and shade steps are done for a project, and dependent subprojects
can typically start. This means that we get increased parallelism due to
not needing to wait for findbugs, checkstyle, tests to run. I typically see
~20 tasks (at peak) running on my desktop in parallel.

Flink should consider using gradle - on Linux with SSD, a clean build takes an 
hour.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7966) Create flink-connector-kafka-1.0 submodule

[Ted Yu (JIRA)]

Ted Yu created FLINK-7966:
-

 Summary: Create flink-connector-kafka-1.0 submodule
 Key: FLINK-7966
 URL: https://issues.apache.org/jira/browse/FLINK-7966
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Major


Kafka 1.0 has been released.
When compiling flink-connector-kafka-0.11 against 1.0.0 release, I got:
{code}
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[468,30]
 cannot find symbol
[ERROR]   symbol: method setTransactionTimeout(long)
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[469,30]
 cannot find symbol
[ERROR]   symbol: method enableTransactionTimeoutWarnings(double)
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[509,9]
 method does not override or implement a method  from a supertype
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[511,22]
 cannot find symbol
[ERROR]   symbol: method ignoreFailuresAfterTransactionTimeout()
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[586,66]
 cannot find symbol
[ERROR]   symbol:   method currentTransaction()
[ERROR]   location: class 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[619,82]
 cannot find symbol
[ERROR]   symbol:   method currentTransaction()
[ERROR]   location: class 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
[ERROR] 
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[895,66]
 cannot find symbol
[ERROR]   symbol:   method currentTransaction()
[ERROR]   location: class 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
{code}
We should create flink-connector-kafka-1.0 submodule to accommodate the new 
Kafka release



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7936) Lack of synchronization w.r.t. taskManagers in MetricStore#add()

[Ted Yu (JIRA)]

Ted Yu created FLINK-7936:
-

 Summary: Lack of synchronization w.r.t. taskManagers in 
MetricStore#add()
 Key: FLINK-7936
 URL: https://issues.apache.org/jira/browse/FLINK-7936
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) 
info).taskManagerID;
  tm = taskManagers.computeIfAbsent(tmID, k -> new 
TaskManagerMetricStore());
{code}
In other places, access to taskManagers is protected by lock on MetricStore.this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7897) Consider using nio.Files for file deletion in TransientBlobCleanupTask

[Ted Yu (JIRA)]

Ted Yu created FLINK-7897:
-

 Summary: Consider using nio.Files for file deletion in 
TransientBlobCleanupTask
 Key: FLINK-7897
 URL: https://issues.apache.org/jira/browse/FLINK-7897
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


nio.Files#delete() provides better clue as to why the deletion may fail:

https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#delete(java.nio.file.Path)

Depending on the potential exception, the call to localFile.exists() may be 
skipped.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7796) RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation should close snapshotCloseableRegistry

[Ted Yu (JIRA)]

Ted Yu created FLINK-7796:
-

 Summary: RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation 
should close snapshotCloseableRegistry
 Key: FLINK-7796
 URL: https://issues.apache.org/jira/browse/FLINK-7796
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


snapshotCloseableRegistry, being CloseableRegistry, depends on invocation of 
close() method to release certain resource.

It seems close() can be called from releaseSnapshotResources()



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7795) Utilize error-prone to discover common coding mistakes

[Ted Yu (JIRA)]

Ted Yu created FLINK-7795:
-

 Summary: Utilize error-prone to discover common coding mistakes
 Key: FLINK-7795
 URL: https://issues.apache.org/jira/browse/FLINK-7795
 Project: Flink
  Issue Type: Improvement
Reporter: Ted Yu


http://errorprone.info/ is a tool which detects common coding mistakes.
We should incorporate into Flink build.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)