[jira] [Created] (FLINK-8150) WebUI in FLIP-6 mode exposes TaskManagerConnection IDs as TaskManager IDs

2017-11-24 Thread Gary Yao (JIRA) 
Gary Yao created FLINK-8150:
---

 Summary: WebUI in FLIP-6 mode exposes TaskManagerConnection IDs as 
TaskManager IDs
 Key: FLINK-8150
 URL: https://issues.apache.org/jira/browse/FLINK-8150
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination, REST
Affects Versions: 1.5.0
Reporter: Gary Yao
Priority: Blocker
 Fix For: 1.5.0


TaskManager IDs exposed by 
{{org.apache.flink.runtime.rest.handler.taskmanager.TaskManagersHandler}} 
cannot be used as input to query TaskManager metrics with method 
{{MetricStore#getTaskManagerMetricStore(String)}}.

*Reason*
{{ResourceManager#requestTaskManagerInfo(Time)}} returns {{TaskManagerInfo}} s 
where the instance IDs are set to the IDs of the {{TaskExecutorConnection}} s. 
While {{ResourceManager#requestTaskManagerMetricQueryServicePaths(Time)}} 
returns the Taskmanager resource IDs.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8149) Replace usages of deprecated SerializationSchema

2017-11-24 Thread Hai Zhou UTC+8 (JIRA) 
Hai Zhou UTC+8 created FLINK-8149:
-

 Summary: Replace usages of deprecated SerializationSchema
 Key: FLINK-8149
 URL: https://issues.apache.org/jira/browse/FLINK-8149
 Project: Flink
  Issue Type: Improvement
  Components: Kinesis Connector
Affects Versions: 1.4.0
Reporter: Hai Zhou UTC+8
Assignee: Hai Zhou UTC+8
 Fix For: 1.5.0


The deprecated {{SerializationSchema}}  in {{flink-streaming-java}}, has been 
moved to {{flink-core}}.
But, the deprecate {{SerializationSchema}} is still used in 
{{flink-connector-kinesis}}.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8148) Test instability in YarnFileStageTest

2017-11-24 Thread Nico Kruber (JIRA) 
Nico Kruber created FLINK-8148:
--

 Summary: Test instability in YarnFileStageTest
 Key: FLINK-8148
 URL: https://issues.apache.org/jira/browse/FLINK-8148
 Project: Flink
  Issue Type: Bug
  Components: Tests, YARN
Affects Versions: 1.4.0
Reporter: Nico Kruber
Assignee: Nico Kruber
Priority: Critical


{code}
Running org.apache.flink.yarn.YarnFileStageTestS3ITCase
Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 13.152 sec <<< 
FAILURE! - in org.apache.flink.yarn.YarnFileStageTestS3ITCase
testRecursiveUploadForYarnS3(org.apache.flink.yarn.YarnFileStageTestS3ITCase)  
Time elapsed: 8.515 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at 
org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:171)
at 
org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3(YarnFileStageTestS3ITCase.java:192)
{code}

from https://travis-ci.org/apache/flink/jobs/305861539

{{YarnFileStageTestS3ITCase.testRecursiveUploadForYarn}} verifies that the test 
directory used is cleaned up by 
{{YarnFileStageTest.testCopyFromLocalRecursive}} which should clean up the 
directory (in a {{finally}} block). However, for S3, we may not always see our 
own deletes.

Quoting from https://aws.amazon.com/s3/faqs/ here:
{quote}Q: What data consistency model does Amazon S3 employ?
Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS 
of new objects and eventual consistency for overwrite PUTS and DELETES.{quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release 1.4.0, release candidate #1

2017-11-24 Thread Ted Yu 
Long weekend should end this Sunday. 
Closing vote Wednesday would be great. 
Thanks
 Original message From: Aljoscha Krettek  
Date: 11/24/17  5:34 AM  (GMT-08:00) To: dev@flink.apache.org Subject: Re: 
[VOTE] Release 1.4.0, release candidate #1 
How long will the long weekend be? I thought about closing the vote on 
Wednesday, i.e. not count the weekend. Would that work?

Best,
Aljoscha

> On 24. Nov 2017, at 12:18, Ted Yu  wrote:
> 
> Aljoscha:
> Thanks for spinning RC.
> 
> bq. The vote will be open for at least 72 hours
> 
> As you are aware, it is long weekend in US.
> 
> Is it possible to extend by 24 hours so that developers in US can
> participate in validation ?
> 
> Cheers
> 
> On Fri, Nov 24, 2017 at 2:57 AM, Aljoscha Krettek 
> wrote:
> 
>> Hi everyone,
>> 
>> Please review and vote on release candidate #1 for the version 1.4.0, as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.4.0-rc1" [5],
>> * website pull request listing the new release [6].
>> 
>> Please have a careful look at the website PR because I changed some
>> wording and we're now also releasing a binary without Hadoop dependencies.
>> 
>> Please use this document for coordinating testing efforts: [7]
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Your friendly Release Manager
>> 
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12315522=12340533
>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1139
>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=
>> a0b322cf77851d3b8589812a0c8e443e9e320e67
>> [6] https://github.com/apache/flink-web/pull/95
>> [7] https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj
>> 4_CEmMTpCkY81s/edit?usp=sharing
>> 
>> Pro-tip: you can create a settings.xml file with these contents:
>> 
>> 
>> 
>>  flink-1.4.0
>> 
>> 
>>  
>>    flink-1.4.0
>>    
>>  
>>    flink-1.4.0
>>    
>>    https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>    
>>  
>>  
>>    archetype
>>    
>>    https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>    
>>  
>>    
>>  
>> 
>> 
>> 
>> And reference that in you maven commands via --settings
>> path/to/settings.xml. This is useful for creating a quickstart based on the
>> staged release and for building against the staged jars.

Re: [VOTE] Release 1.4.0, release candidate #1

2017-11-24 Thread Aljoscha Krettek 
How long will the long weekend be? I thought about closing the vote on 
Wednesday, i.e. not count the weekend. Would that work?

Best,
Aljoscha

> On 24. Nov 2017, at 12:18, Ted Yu  wrote:
> 
> Aljoscha:
> Thanks for spinning RC.
> 
> bq. The vote will be open for at least 72 hours
> 
> As you are aware, it is long weekend in US.
> 
> Is it possible to extend by 24 hours so that developers in US can
> participate in validation ?
> 
> Cheers
> 
> On Fri, Nov 24, 2017 at 2:57 AM, Aljoscha Krettek 
> wrote:
> 
>> Hi everyone,
>> 
>> Please review and vote on release candidate #1 for the version 1.4.0, as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> 
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release and binary convenience releases to be
>> deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "release-1.4.0-rc1" [5],
>> * website pull request listing the new release [6].
>> 
>> Please have a careful look at the website PR because I changed some
>> wording and we're now also releasing a binary without Hadoop dependencies.
>> 
>> Please use this document for coordinating testing efforts: [7]
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Your friendly Release Manager
>> 
>> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12315522=12340533
>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc1/
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapacheflink-1139
>> [5] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=
>> a0b322cf77851d3b8589812a0c8e443e9e320e67
>> [6] https://github.com/apache/flink-web/pull/95
>> [7] https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj
>> 4_CEmMTpCkY81s/edit?usp=sharing
>> 
>> Pro-tip: you can create a settings.xml file with these contents:
>> 
>> 
>> 
>>  flink-1.4.0
>> 
>> 
>>  
>>flink-1.4.0
>>
>>  
>>flink-1.4.0
>>
>>https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>
>>  
>>  
>>archetype
>>
>>https://repository.apache.org/content/repositories/
>> orgapacheflink-1139/
>>
>>  
>>
>>  
>> 
>> 
>> 
>> And reference that in you maven commands via --settings
>> path/to/settings.xml. This is useful for creating a quickstart based on the
>> staged release and for building against the staged jars.

[jira] [Created] (FLINK-8147) Support and to CEP's pattern API

2017-11-24 Thread Dian Fu (JIRA) 
Dian Fu created FLINK-8147:
--

 Summary: Support and to CEP's pattern API
 Key: FLINK-8147
 URL: https://issues.apache.org/jira/browse/FLINK-8147
 Project: Flink
  Issue Type: Bug
  Components: CEP
Reporter: Dian Fu


Adding API such as {{and}} in CEP's pattern API will let us define patterns 
like {{(A -> B) and (C -> D)}}.
Its usage looks like
{noformat}
Pattern left = Pattern.begin("A").followedBy("B");
Pattern right = Pattern.begin("C").followedBy("D");
Pattern pattern = left.and(right)
{noformat}
If this makes sense, we should also support APIs such as {{or}} and {{not}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8146) Potential resource leak in PythonPlanBinder#unzipPythonLibrary

2017-11-24 Thread Ted Yu (JIRA) 
Ted Yu created FLINK-8146:
-

 Summary: Potential resource leak in 
PythonPlanBinder#unzipPythonLibrary
 Key: FLINK-8146
 URL: https://issues.apache.org/jira/browse/FLINK-8146
 Project: Flink
  Issue Type: Bug
Reporter: Ted Yu


{code}
while (entry != null) {
...
}
zis.closeEntry();
{code}
Looking at the catch block inside the loop, it seems the intention is to close 
zis upon getting exception.
zis.close() should be called outside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] Release 1.4.0, release candidate #1

2017-11-24 Thread Aljoscha Krettek 
Hi everyone,

Please review and vote on release candidate #1 for the version 1.4.0, as 
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
 
 
The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.4.0-rc1" [5],
* website pull request listing the new release [6].

Please have a careful look at the website PR because I changed some wording and 
we're now also releasing a binary without Hadoop dependencies.

Please use this document for coordinating testing efforts: [7]
 
The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.
 
Thanks,
Your friendly Release Manager
 
[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12340533
[2] http://people.apache.org/~aljoscha/flink-1.4.0-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1139
[5] 
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h=a0b322cf77851d3b8589812a0c8e443e9e320e67
[6] https://github.com/apache/flink-web/pull/95
[7] 
https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj4_CEmMTpCkY81s/edit?usp=sharing

Pro-tip: you can create a settings.xml file with these contents:



  flink-1.4.0


  
flink-1.4.0

  
flink-1.4.0


https://repository.apache.org/content/repositories/orgapacheflink-1139/

  
  
archetype


https://repository.apache.org/content/repositories/orgapacheflink-1139/

  

  



And reference that in you maven commands via --settings path/to/settings.xml. 
This is useful for creating a quickstart based on the staged release and for 
building against the staged jars.

[jira] [Created] (FLINK-8145) IOManagerAsync not properly shut down in various tests

2017-11-24 Thread Nico Kruber (JIRA) 
Nico Kruber created FLINK-8145:
--

 Summary: IOManagerAsync not properly shut down in various tests
 Key: FLINK-8145
 URL: https://issues.apache.org/jira/browse/FLINK-8145
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.4.0
Reporter: Nico Kruber
Assignee: Nico Kruber


In various tests, e.g. {{AsynchronousBufferFileWriterTest}}, 
{{BufferFileWriterReaderTest}}, or {{HashTableITCase}}, {{IOManagerAsync}} 
instances are used which are not shut down at all. The only thing cleaning them 
up seems to be a shutdown handler attached to the JVM. Since each 
{{IOManagerAsync}} is spawning a number of reader and writer threads, this may 
put a significant burden on the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-8144) Optimize the timer logic in RowTimeUnboundedOver

2017-11-24 Thread Dian Fu (JIRA) 
Dian Fu created FLINK-8144:
--

 Summary: Optimize the timer logic in RowTimeUnboundedOver
 Key: FLINK-8144
 URL: https://issues.apache.org/jira/browse/FLINK-8144
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: Dian Fu
Assignee: Dian Fu
 Fix For: 1.5.0


Currently the logic of {{RowTimeUnboundedOver}} is as follows:
1) When element comes, buffer it in MapState and and register a timer at 
{{current watermark + 1}}
2) When event timer triggered, scan the MapState and find the elements below 
the current watermark and process it. If there are remaining elements to 
process, register a new timer at {{current watermark + 1}}.

Let's assume that watermark comes about 5 seconds later than the event on 
average, then we will scan about 5000 times the MapState before actually 
processing the events.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)