[jira] [Closed] (FLINK-5452) Make table unit tests pass under cluster mode

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske closed FLINK-5452.

   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0

Fixed for 1.2.0 with 7f7692619b0fcf66a560fb8a20e9425f4fe02747
Fixed for 1.3.0 with 0ea996a64ca0ff9589ffcd5b89967f51aee1ffe6

> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
> Fix For: 1.2.0, 1.3.0
>
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (FLINK-5452) Make table unit tests pass under cluster mode

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske reopened FLINK-5452:
--

> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5386) Refactoring Window Clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831354#comment-15831354
 ] 

ASF GitHub Bot commented on FLINK-5386:
---

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3046
  
Hi, @fhueske thanks a lot for the review. I have updated the PR according 
to your comments. Let me know if I miss something，thanks again!!


> Refactoring Window Clause
> -
>
> Key: FLINK-5386
> URL: https://issues.apache.org/jira/browse/FLINK-5386
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Similar to the SQL, window clause is defined "as" a symbol which is 
> explicitly used in groupby/over. We are proposing to refactor the way to 
> write groupby+window tableAPI as follows: 
> {code}
> tab //Table('a,'b,'c)
> .window( Slide over 10.milli every 5.milli  as 'w1as 'w1) // WindowedTable
> .groupBy('w1,'a,'b) // WindowGroupedTable
> .select('a,'b,c.count as 'mycount) // Table
> .window(Tumble over 5.milli  on 'b as 'w2)
> .groupBy('w2)
> .select('a.count, 'w2.start, 'w2.end)
> {code}
> In this way, for row-window, we anyway need to define window clause as a 
> symbol. This change will make the API of window and row-window consistent, 
> example for row-window:
> {code}
>   .window(RowXXXWindow as ‘x, RowYYYWindow as ‘y) // WindowedTable
>   .select(‘a, ‘b.count over ‘x as ‘xcnt, ‘c.count over ‘y as ‘ycnt, ‘x.start, 
> ‘x.end)
> {code}
> What do you think? [~fhueske] [~twalthr]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5520) Disable outer joins with non-equality predicates

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske closed FLINK-5520.

   Resolution: Done
Fix Version/s: 1.3.0

Fixed for 1.2.0 with 9073c53f902df7b3dbbdec2a30d86030e49fe27e
Fixed for 1.3.0 with d1301c82b85c00284d90e8f5bdac4fd86dc5b173

> Disable outer joins with non-equality predicates
> 
>
> Key: FLINK-5520
> URL: https://issues.apache.org/jira/browse/FLINK-5520
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Assignee: lincoln.lee
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
>
> Outer joins with non-equality predicates (and at least one equality 
> predicate) compute incorrect results. 
> Since this is not a very common requirement, I propose to disable this 
> feature for the 1.2.0 release and correctly implement it for a later version.
> The fix should add checks in the Table API validation phase (to get a good 
> error message) and in the DataSetJoinRule to prevent translation of SQL 
> queries with non-equality predicates on outer joins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3046: [FLINK-5386][Table API & SQL] refactoring Window Clause

[sunjincheng121]

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3046
  
Hi, @fhueske thanks a lot for the review. I have updated the PR according 
to your comments. Let me know if I miss something，thanks again!!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5580) Kerberos keytabs not working for YARN deployment mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831322#comment-15831322
 ] 

ASF GitHub Bot commented on FLINK-5580:
---

GitHub user tzulitai opened a pull request:

https://github.com/apache/flink/pull/3177

[FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in 
YARN mode

Previously, the local path of the shipped keytab was set _after_ 
`SecurityConfiguration` was created, causing the picked up keytab path to be 
invalid and validation of the security configuration not passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tzulitai/flink FLINK-5580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3177


commit 871c6a6601af33f57c3794d6f627a874ac4c8731
Author: Tzu-Li (Gordon) Tai 
Date:   2017-01-20T00:41:05Z

[FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in 
YARN mode




> Kerberos keytabs not working for YARN deployment mode
> -
>
> Key: FLINK-5580
> URL: https://issues.apache.org/jira/browse/FLINK-5580
> Project: Flink
>  Issue Type: Bug
>  Components: Security, YARN
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Fix For: 1.2.0
>
>
> Setup: Kerberos security using keytabs, Flink session on YARN deployment (in 
> standalone, it works fine without problems).
> I’m getting these error messages in the YARN node managers, causing the 
> TaskManager containers to fail to start properly:
> {{org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException 
> as:tzulitai (auth:SIMPLE) 
> cause:org.apache.hadoop.security.AccessControlException: Client cannot 
> authenticate via:[TOKEN, KERBEROS]}}
> The security configuration for Hadoop has been set to "kerberos", to the 
> "auto: SIMPLE" seems very strange. It also seems as if credential tokens has 
> not been properly set for the {{ContainerLaunchContext}} s, which may be an 
> issue causing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3177: [FLINK-5580] [security] Fix path setting of shippe...

[tzulitai]

GitHub user tzulitai opened a pull request:

https://github.com/apache/flink/pull/3177

[FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in 
YARN mode

Previously, the local path of the shipped keytab was set _after_ 
`SecurityConfiguration` was created, causing the picked up keytab path to be 
invalid and validation of the security configuration not passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tzulitai/flink FLINK-5580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3177


commit 871c6a6601af33f57c3794d6f627a874ac4c8731
Author: Tzu-Li (Gordon) Tai 
Date:   2017-01-20T00:41:05Z

[FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in 
YARN mode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5571) add open and close methods for UserDefinedFunction in TableAPI & SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831296#comment-15831296
 ] 

ASF GitHub Bot commented on FLINK-5571:
---

GitHub user godfreyhe opened a pull request:

https://github.com/apache/flink/pull/3176

[FLINK-5571] [table] add open and close methods for UserDefinedFunction

Currently, a User Defined Function (UDF) in table API & SQL works on zero, 
one, or multiple values in custom evaluation method. Many UDFs need more 
complex features, e.g. report metrics, get parameters from job configuration, 
or get extra data from distribute cache file, etc. Adding open and close 
methods in UserDefinedFunction class can solve this problem. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/godfreyhe/flink udf-open-close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3176


commit 4ac2be7b3ab4dc0e8a18e43c79a031d7a16ee1ea
Author: godfreyhe 
Date:   2017-01-20T06:42:12Z

add open and close methods for UserDefinedFunction




> add open and close methods for UserDefinedFunction in TableAPI & SQL
> 
>
> Key: FLINK-5571
> URL: https://issues.apache.org/jira/browse/FLINK-5571
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: godfrey he
>Assignee: godfrey he
>
> Currently, a User Defined Function (UDF) in table API & SQL works on zero, 
> one, or multiple values in custom evaluation method. Many UDFs need more 
> complex features, e.g. report metrics, get parameters from job configuration, 
> or get extra data from distribute cache file, etc. Adding open and close 
> methods in UserDefinedFunction class can solve this problem. The code cloud 
> look like:
> {code}
> trait UserDefinedFunction {
>   def open(context: UDFContext): Unit = {}
>   def close(): Unit = {}
> }
> {code}
> UDFContext contains the information about metric reporters, job parameters, 
> distribute cache, etc. The code cloud look like:
> {code}
> class UDFContext(context: RuntimeContext) {
>   def getMetricGroup: MetricGroup = ???
>   def getDistributedCacheFile(name: String): File = ???
>   def getJobParameter(key: String, default: String): String = ???
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3176: [FLINK-5571] [table] add open and close methods fo...

[godfreyhe]

GitHub user godfreyhe opened a pull request:

https://github.com/apache/flink/pull/3176

[FLINK-5571] [table] add open and close methods for UserDefinedFunction

Currently, a User Defined Function (UDF) in table API & SQL works on zero, 
one, or multiple values in custom evaluation method. Many UDFs need more 
complex features, e.g. report metrics, get parameters from job configuration, 
or get extra data from distribute cache file, etc. Adding open and close 
methods in UserDefinedFunction class can solve this problem. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/godfreyhe/flink udf-open-close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3176


commit 4ac2be7b3ab4dc0e8a18e43c79a031d7a16ee1ea
Author: godfreyhe 
Date:   2017-01-20T06:42:12Z

add open and close methods for UserDefinedFunction




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-5452) Make table unit tests pass under cluster mode

[Kurt Young (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young closed FLINK-5452.
-

> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-5452) Make table unit tests pass under cluster mode

[Kurt Young (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young resolved FLINK-5452.
---
Resolution: Fixed

> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5493) FlinkDistributionOverlay does not properly display missing environment variables

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5493.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

master: d830cac813254995e3cc04fbb7ab1b471bdd6c90
1.2: 82a6b83ec4c1660dfe97fd299ac3169e5c10088b

> FlinkDistributionOverlay does not properly display missing environment 
> variables
> 
>
> Key: FLINK-5493
> URL: https://issues.apache.org/jira/browse/FLINK-5493
> Project: Flink
>  Issue Type: Bug
>  Components: Cluster Management
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Till Rohrmann
>Assignee: Boris Osipov
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> The class {{FlinkDistributionOverlay}} does not properly log missing 
> environment variables in case of an error. This should be changed so that the 
> user knows which variables he has to set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5493) FlinkDistributionOverlay does not properly display missing environment variables

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5493:

Assignee: Boris Osipov

> FlinkDistributionOverlay does not properly display missing environment 
> variables
> 
>
> Key: FLINK-5493
> URL: https://issues.apache.org/jira/browse/FLINK-5493
> Project: Flink
>  Issue Type: Bug
>  Components: Cluster Management
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Till Rohrmann
>Assignee: Boris Osipov
>Priority: Minor
> Fix For: 1.3.0
>
>
> The class {{FlinkDistributionOverlay}} does not properly log missing 
> environment variables in case of an error. This should be changed so that the 
> user knows which variables he has to set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5113.
---
   Resolution: Fixed
Fix Version/s: 1.3.0

master: 525edf1e6925b55302d991ddf537a2f16caba21d
1.2: 8b069fde3adccdcd5143de90d3d4834f33b5acff

> Make all Testing Functions implement CheckpointedFunction Interface.
> 
>
> Key: FLINK-5113
> URL: https://issues.apache.org/jira/browse/FLINK-5113
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0, 1.3.0
>
>
> Currently stateful functions implement the (old) Checkpointed interface.
> This is issue aims at porting all these function to the new 
> CheckpointedFunction interface, so that they can leverage the new 
> capabilities by it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4920) Add a Scala Function Gauge

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4920.
---
   Resolution: Fixed
Fix Version/s: 1.3.0

master: 570dbc8d2597ee9688579f399b8743636e70f891

> Add a Scala Function Gauge
> --
>
> Key: FLINK-4920
> URL: https://issues.apache.org/jira/browse/FLINK-4920
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics, Scala API
>Reporter: Stephan Ewen
>Assignee: Pattarawat Chormai
>  Labels: easyfix, starter
> Fix For: 1.3.0
>
>
> A useful metrics utility for the Scala API would be to add a Gauge that 
> obtains its value by calling a Scala Function0.
> That way, one can add Gauges in Scala programs using Scala lambda notation or 
> function references.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5113:

Affects Version/s: 1.3.0

> Make all Testing Functions implement CheckpointedFunction Interface.
> 
>
> Key: FLINK-5113
> URL: https://issues.apache.org/jira/browse/FLINK-5113
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> Currently stateful functions implement the (old) Checkpointed interface.
> This is issue aims at porting all these function to the new 
> CheckpointedFunction interface, so that they can leverage the new 
> capabilities by it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5321.
---
   Resolution: Fixed
Fix Version/s: 1.3.0

master: b36b43b9ad3f4b04a2a701b9139bb3eaf17b9ff8
1.2: 2bbc92cf637b2caf612c43b0f35ba2282b5bc981

> FlinkMiniCluster does not start Jobmanager MetricQueryService
> -
>
> Key: FLINK-5321
> URL: https://issues.apache.org/jira/browse/FLINK-5321
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Metrics
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.3.0
>
>
> The JobManager MetricQueryService is never started when using the 
> LocalFlinkMiniCluster. It lacks the call to 
> MetricRegistry#startQueryService().
> As a result jobmanager metrics aren't reporter to the web frontend, and it 
> causes repeated logging of exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5417) Fix the wrong config file name

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5417.
---
   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0

master: 6c03491ce30ba3f31fb85ffb864731aa3f5aa482
1.2: 7f20bd0d65cdce9e6180c20777959e4c522c55ce

> Fix the wrong config file name 
> ---
>
> Key: FLINK-5417
> URL: https://issues.apache.org/jira/browse/FLINK-5417
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Tao Wang
>Priority: Trivial
> Fix For: 1.2.0, 1.3.0
>
>
> As the config file name is conf/flink-conf.yaml, the usage 
> "conf/flink-config.yaml" in document is wrong and easy to confuse user. We 
> should correct them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5321:

Affects Version/s: 1.3.0

> FlinkMiniCluster does not start Jobmanager MetricQueryService
> -
>
> Key: FLINK-5321
> URL: https://issues.apache.org/jira/browse/FLINK-5321
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Metrics
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.3.0
>
>
> The JobManager MetricQueryService is never started when using the 
> LocalFlinkMiniCluster. It lacks the call to 
> MetricRegistry#startQueryService().
> As a result jobmanager metrics aren't reporter to the web frontend, and it 
> causes repeated logging of exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5417) Fix the wrong config file name

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5417:

Affects Version/s: 1.3.0
   1.2.0

> Fix the wrong config file name 
> ---
>
> Key: FLINK-5417
> URL: https://issues.apache.org/jira/browse/FLINK-5417
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Tao Wang
>Priority: Trivial
>
> As the config file name is conf/flink-conf.yaml, the usage 
> "conf/flink-config.yaml" in document is wrong and easy to confuse user. We 
> should correct them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5424) Improve Restart Strategy Logging

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5424.
---
   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0

master: c210ff37a4c55c835f76c031f2d9cf18165812aa
1.2: fc4b45e4781a4ccd595ad31aefeba448fdea9700

> Improve Restart Strategy Logging
> 
>
> Key: FLINK-5424
> URL: https://issues.apache.org/jira/browse/FLINK-5424
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Reporter: Shannon Carey
>Assignee: Shannon Carey
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> I'll be submitting a PR which includes some minor improvements to logging 
> related to restart strategies.
> Specifically, I added a toString so that the log contains better info about 
> failure-rate restart strategy, and I added an explanation in the log when the 
> restart strategy is responsible for preventing job restart (currently, 
> there's no indication that the restart strategy had anything to do with it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5419) Taskmanager metrics not accessible via REST

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5419.
---
Resolution: Fixed

master: c93e04c16a7865dbbc90d89799b46f91831e0a55
1.2: f90edc483000db42caaff0de650c1f5bca11bf47

> Taskmanager metrics not accessible via REST
> ---
>
> Key: FLINK-5419
> URL: https://issues.apache.org/jira/browse/FLINK-5419
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics, Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.3.0
>
>
> There is currently a URL clash between the TaskManagersHandler and 
> TaskManagerMetricsHandler, with both being routed to
> {code}
> /taskmanagers/:taskmanagerid/metrics
> {code}
> As a result it is not possible to query the full set of metrics for a 
> taskmanager, but only the hard-coded subset that is displayed on the metrics 
> tab on the taskmanager page.
> This is a side-effect of 6d53bbc4b92e651786ecc8c2c6dfeb8e450a16a3 making the 
> URL's more consistent. The TaskManager page in the web-interface has 3 tabs: 
> Metrics, Log and Stdout.
> The URLs for these tabs are
> {code}
> /taskmanager//metrics
> /taskmanager//log
> /taskmanager//stdout
> {code}
> which correspond to the REST URL's used. Previously, the metrics tab used 
> {code}/taskmanager/{code}
> However, 70704de0c82cbb7b143dd696221e11999feb3600 then exposed the metrics 
> gathered by the metrics system through the REST API. The assumption was that 
> general information for the taskmanagers are retrieved via 
> /taskmanager/, similar to how the job-related URL's are structured, 
> which sadly isn't the case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5434.
---
   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0

master: a7437cca0eeeb636f839ecbc8e96ede64405ab1b
1.2: ebf4443a87ec0a46cd821adc85187d9cb58ae992

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Vasia Kalavri
>Assignee: GaoLun
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5434:

Affects Version/s: 1.3.0
   1.2.0

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Vasia Kalavri
>Assignee: GaoLun
> Fix For: 1.2.0, 1.3.0
>
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5434:

Priority: Minor  (was: Major)

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Vasia Kalavri
>Assignee: GaoLun
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5329) Metric list is being cut off in the WebFrontend

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5329:

Fix Version/s: 1.3.0

> Metric list is being cut off in the WebFrontend
> ---
>
> Key: FLINK-5329
> URL: https://issues.apache.org/jira/browse/FLINK-5329
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
> Environment: FireFox 50.0.2
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: ss_metrics.png
>
>
> The list for selecting metrics is being cut off, see the attached screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-5329) Metric list is being cut off in the WebFrontend

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-5329:
---

Assignee: Chesnay Schepler

> Metric list is being cut off in the WebFrontend
> ---
>
> Key: FLINK-5329
> URL: https://issues.apache.org/jira/browse/FLINK-5329
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
> Environment: FireFox 50.0.2
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: ss_metrics.png
>
>
> The list for selecting metrics is being cut off, see the attached screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5434:

Assignee: GaoLun

> Remove unsupported project() transformation from Scala DataStream docs
> --
>
> Key: FLINK-5434
> URL: https://issues.apache.org/jira/browse/FLINK-5434
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Vasia Kalavri
>Assignee: GaoLun
>
> The Scala DataStream does not have a project() transformation, yet the docs 
> include it as a supported operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5329) Metric list is being cut off in the WebFrontend

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5329.
---
Resolution: Fixed

master: b8c34d455029907a237230e2f8c49a8b6b651970
1.2: 8cdbe44aea1f7e544e4a2907e5274012b1481c57

> Metric list is being cut off in the WebFrontend
> ---
>
> Key: FLINK-5329
> URL: https://issues.apache.org/jira/browse/FLINK-5329
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
> Environment: FireFox 50.0.2
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: ss_metrics.png
>
>
> The list for selecting metrics is being cut off, see the attached screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5380) Number of outgoing records not reported in web interface

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5380.
---
Resolution: Fixed

master: cb05915759b1d5ea4dbfcdd3ff76dcfd9cebe601
1.2: 792f7e45216377fa1d6f29dfc767d83cf1a84f37

> Number of outgoing records not reported in web interface
> 
>
> Key: FLINK-5380
> URL: https://issues.apache.org/jira/browse/FLINK-5380
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics, Streaming, Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: outRecordsNotreported.png
>
>
> The web frontend does not report any outgoing records in the web frontend.
> The amount of data in MB is reported correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5329) Metric list is being cut off in the WebFrontend

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5329:

Affects Version/s: 1.3.0

> Metric list is being cut off in the WebFrontend
> ---
>
> Key: FLINK-5329
> URL: https://issues.apache.org/jira/browse/FLINK-5329
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
> Environment: FireFox 50.0.2
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: ss_metrics.png
>
>
> The list for selecting metrics is being cut off, see the attached screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5432:

Affects Version/s: 1.3.0

> ContinuousFileMonitoringFunction is not monitoring nested files
> ---
>
> Key: FLINK-5432
> URL: https://issues.apache.org/jira/browse/FLINK-5432
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Yassine Marzougui
>Assignee: Yassine Marzougui
> Fix For: 1.2.0, 1.3.0
>
>
> The {{ContinuousFileMonitoringFunction}} does not monitor nested files even 
> if the inputformat has NestedFileEnumeration set to true. This can be fixed 
> by enabling a recursive scan of the directories in the {{listEligibleFiles}} 
> method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5432.
---
Resolution: Fixed

master: 9945904e2251e7c0e218e2766bf07778d1307277
1.2: 28c18e22127a85f773e7504a0e9d188bad9334e2

> ContinuousFileMonitoringFunction is not monitoring nested files
> ---
>
> Key: FLINK-5432
> URL: https://issues.apache.org/jira/browse/FLINK-5432
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Yassine Marzougui
>Assignee: Yassine Marzougui
> Fix For: 1.2.0, 1.3.0
>
>
> The {{ContinuousFileMonitoringFunction}} does not monitor nested files even 
> if the inputformat has NestedFileEnumeration set to true. This can be fixed 
> by enabling a recursive scan of the directories in the {{listEligibleFiles}} 
> method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5150) WebUI metric-related resource leak

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-5150.
---
Resolution: Fixed

master: cbd933bd1def789d1978be5349005dd7d9de3082
1.2: 79b68263f52690697ceec0528023d35feed0bd89

> WebUI metric-related resource leak
> --
>
> Key: FLINK-5150
> URL: https://issues.apache.org/jira/browse/FLINK-5150
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.2.0, 1.3.0
>
>
> The WebUI maintains a list of selected metrics for all jobs and vertices. 
> When a metric is selected in the metric screen it is added to this list, and 
> removed if it is unselected.
> The contents of this list are stored in the browser's localStorage. This 
> allows a user to setup a metric screen, move to another page, and return to 
> the original screen completely intact.
> However, if the metrics are never *unselected* by the user they will remain 
> in this list. They will also still be in this list if the WebUI can't even 
> display the corresponding job page anymore, if for example the history size 
> limit was exceeded. They will even survive a browser restart, since they are 
> not stored in a session-based storage.
> Furthermore, the WebUI still tries to update these metricsd, adding 
> additional overhead to the WebBackend and potentially network.
> In other words, if you _ever_ checked out metrics tab for some job, chances 
> are that the next time you start the WebInterface it will still try to update 
> the metrics for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5150:

Fix Version/s: 1.3.0

> WebUI metric-related resource leak
> --
>
> Key: FLINK-5150
> URL: https://issues.apache.org/jira/browse/FLINK-5150
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.2.0, 1.3.0
>
>
> The WebUI maintains a list of selected metrics for all jobs and vertices. 
> When a metric is selected in the metric screen it is added to this list, and 
> removed if it is unselected.
> The contents of this list are stored in the browser's localStorage. This 
> allows a user to setup a metric screen, move to another page, and return to 
> the original screen completely intact.
> However, if the metrics are never *unselected* by the user they will remain 
> in this list. They will also still be in this list if the WebUI can't even 
> display the corresponding job page anymore, if for example the history size 
> limit was exceeded. They will even survive a browser restart, since they are 
> not stored in a session-based storage.
> Furthermore, the WebUI still tries to update these metricsd, adding 
> additional overhead to the WebBackend and potentially network.
> In other words, if you _ever_ checked out metrics tab for some job, chances 
> are that the next time you start the WebInterface it will still try to update 
> the metrics for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831232#comment-15831232
 ] 

ASF GitHub Bot commented on FLINK-5321:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2991


> FlinkMiniCluster does not start Jobmanager MetricQueryService
> -
>
> Key: FLINK-5321
> URL: https://issues.apache.org/jira/browse/FLINK-5321
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager, Metrics
>Affects Versions: 1.2.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.2.0
>
>
> The JobManager MetricQueryService is never started when using the 
> LocalFlinkMiniCluster. It lacks the call to 
> MetricRegistry#startQueryService().
> As a result jobmanager metrics aren't reporter to the web frontend, and it 
> causes repeated logging of exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4920) Add a Scala Function Gauge

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831234#comment-15831234
 ] 

ASF GitHub Bot commented on FLINK-4920:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3080


> Add a Scala Function Gauge
> --
>
> Key: FLINK-4920
> URL: https://issues.apache.org/jira/browse/FLINK-4920
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics, Scala API
>Reporter: Stephan Ewen
>Assignee: Pattarawat Chormai
>  Labels: easyfix, starter
>
> A useful metrics utility for the Scala API would be to add a Gauge that 
> obtains its value by calling a Scala Function0.
> That way, one can add Gauges in Scala programs using Scala lambda notation or 
> function references.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5329) Metric list is being cut off in the WebFrontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831227#comment-15831227
 ] 

ASF GitHub Bot commented on FLINK-5329:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3109


> Metric list is being cut off in the WebFrontend
> ---
>
> Key: FLINK-5329
> URL: https://issues.apache.org/jira/browse/FLINK-5329
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0
> Environment: FireFox 50.0.2
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: ss_metrics.png
>
>
> The list for selecting metrics is being cut off, see the attached screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5424) Improve Restart Strategy Logging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831230#comment-15831230
 ] 

ASF GitHub Bot commented on FLINK-5424:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3079


> Improve Restart Strategy Logging
> 
>
> Key: FLINK-5424
> URL: https://issues.apache.org/jira/browse/FLINK-5424
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Reporter: Shannon Carey
>Assignee: Shannon Carey
>Priority: Minor
>
> I'll be submitting a PR which includes some minor improvements to logging 
> related to restart strategies.
> Specifically, I added a toString so that the log contains better info about 
> failure-rate restart strategy, and I added an explanation in the log when the 
> restart strategy is responsible for preventing job restart (currently, 
> there's no indication that the restart strategy had anything to do with it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5150:

Affects Version/s: (was: 1.1.3)

> WebUI metric-related resource leak
> --
>
> Key: FLINK-5150
> URL: https://issues.apache.org/jira/browse/FLINK-5150
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.2.0
>
>
> The WebUI maintains a list of selected metrics for all jobs and vertices. 
> When a metric is selected in the metric screen it is added to this list, and 
> removed if it is unselected.
> The contents of this list are stored in the browser's localStorage. This 
> allows a user to setup a metric screen, move to another page, and return to 
> the original screen completely intact.
> However, if the metrics are never *unselected* by the user they will remain 
> in this list. They will also still be in this list if the WebUI can't even 
> display the corresponding job page anymore, if for example the history size 
> limit was exceeded. They will even survive a browser restart, since they are 
> not stored in a session-based storage.
> Furthermore, the WebUI still tries to update these metricsd, adding 
> additional overhead to the WebBackend and potentially network.
> In other words, if you _ever_ checked out metrics tab for some job, chances 
> are that the next time you start the WebInterface it will still try to update 
> the metrics for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831226#comment-15831226
 ] 

ASF GitHub Bot commented on FLINK-5432:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3090


> ContinuousFileMonitoringFunction is not monitoring nested files
> ---
>
> Key: FLINK-5432
> URL: https://issues.apache.org/jira/browse/FLINK-5432
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.2.0
>Reporter: Yassine Marzougui
>Assignee: Yassine Marzougui
> Fix For: 1.2.0, 1.3.0
>
>
> The {{ContinuousFileMonitoringFunction}} does not monitor nested files even 
> if the inputformat has NestedFileEnumeration set to true. This can be fixed 
> by enabling a recursive scan of the directories in the {{listEligibleFiles}} 
> method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5150) WebUI metric-related resource leak

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831225#comment-15831225
 ] 

ASF GitHub Bot commented on FLINK-5150:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3104


> WebUI metric-related resource leak
> --
>
> Key: FLINK-5150
> URL: https://issues.apache.org/jira/browse/FLINK-5150
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.2.0
>
>
> The WebUI maintains a list of selected metrics for all jobs and vertices. 
> When a metric is selected in the metric screen it is added to this list, and 
> removed if it is unselected.
> The contents of this list are stored in the browser's localStorage. This 
> allows a user to setup a metric screen, move to another page, and return to 
> the original screen completely intact.
> However, if the metrics are never *unselected* by the user they will remain 
> in this list. They will also still be in this list if the WebUI can't even 
> display the corresponding job page anymore, if for example the history size 
> limit was exceeded. They will even survive a browser restart, since they are 
> not stored in a session-based storage.
> Furthermore, the WebUI still tries to update these metricsd, adding 
> additional overhead to the WebBackend and potentially network.
> In other words, if you _ever_ checked out metrics tab for some job, chances 
> are that the next time you start the WebInterface it will still try to update 
> the metrics for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3068: [FLINK-5380] Fix task metrics reuse for single-ope...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3068


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3092: [FLINK-5419] Make full TM metrics available throug...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3092


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3071: [FLINK-5417][DOCUMENTATION]correct the wrong confi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3071


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2939: [FLINK-5113] Ports all functions in the tests to t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2939


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5417) Fix the wrong config file name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831231#comment-15831231
 ] 

ASF GitHub Bot commented on FLINK-5417:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3071


> Fix the wrong config file name 
> ---
>
> Key: FLINK-5417
> URL: https://issues.apache.org/jira/browse/FLINK-5417
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Tao Wang
>Priority: Trivial
>
> As the config file name is conf/flink-conf.yaml, the usage 
> "conf/flink-config.yaml" in document is wrong and easy to confuse user. We 
> should correct them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-5150:

Affects Version/s: 1.3.0
   1.2.0

> WebUI metric-related resource leak
> --
>
> Key: FLINK-5150
> URL: https://issues.apache.org/jira/browse/FLINK-5150
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Critical
> Fix For: 1.2.0
>
>
> The WebUI maintains a list of selected metrics for all jobs and vertices. 
> When a metric is selected in the metric screen it is added to this list, and 
> removed if it is unselected.
> The contents of this list are stored in the browser's localStorage. This 
> allows a user to setup a metric screen, move to another page, and return to 
> the original screen completely intact.
> However, if the metrics are never *unselected* by the user they will remain 
> in this list. They will also still be in this list if the WebUI can't even 
> display the corresponding job page anymore, if for example the history size 
> limit was exceeded. They will even survive a browser restart, since they are 
> not stored in a session-based storage.
> Furthermore, the WebUI still tries to update these metricsd, adding 
> additional overhead to the WebBackend and potentially network.
> In other words, if you _ever_ checked out metrics tab for some job, chances 
> are that the next time you start the WebInterface it will still try to update 
> the metrics for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3080


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831233#comment-15831233
 ] 

ASF GitHub Bot commented on FLINK-5113:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2939


> Make all Testing Functions implement CheckpointedFunction Interface.
> 
>
> Key: FLINK-5113
> URL: https://issues.apache.org/jira/browse/FLINK-5113
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> Currently stateful functions implement the (old) Checkpointed interface.
> This is issue aims at porting all these function to the new 
> CheckpointedFunction interface, so that they can leverage the new 
> capabilities by it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5380) Number of outgoing records not reported in web interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831228#comment-15831228
 ] 

ASF GitHub Bot commented on FLINK-5380:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3068


> Number of outgoing records not reported in web interface
> 
>
> Key: FLINK-5380
> URL: https://issues.apache.org/jira/browse/FLINK-5380
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics, Streaming, Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: outRecordsNotreported.png
>
>
> The web frontend does not report any outgoing records in the web frontend.
> The amount of data in MB is reported correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2991: [FLINK-5321] [metrics] LocalFlinkMiniCluster start...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2991


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5419) Taskmanager metrics not accessible via REST

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831229#comment-15831229
 ] 

ASF GitHub Bot commented on FLINK-5419:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3092


> Taskmanager metrics not accessible via REST
> ---
>
> Key: FLINK-5419
> URL: https://issues.apache.org/jira/browse/FLINK-5419
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics, Webfrontend
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.3.0
>
>
> There is currently a URL clash between the TaskManagersHandler and 
> TaskManagerMetricsHandler, with both being routed to
> {code}
> /taskmanagers/:taskmanagerid/metrics
> {code}
> As a result it is not possible to query the full set of metrics for a 
> taskmanager, but only the hard-coded subset that is displayed on the metrics 
> tab on the taskmanager page.
> This is a side-effect of 6d53bbc4b92e651786ecc8c2c6dfeb8e450a16a3 making the 
> URL's more consistent. The TaskManager page in the web-interface has 3 tabs: 
> Metrics, Log and Stdout.
> The URLs for these tabs are
> {code}
> /taskmanager//metrics
> /taskmanager//log
> /taskmanager//stdout
> {code}
> which correspond to the REST URL's used. Previously, the metrics tab used 
> {code}/taskmanager/{code}
> However, 70704de0c82cbb7b143dd696221e11999feb3600 then exposed the metrics 
> gathered by the metrics system through the REST API. The assumption was that 
> general information for the taskmanagers are retrieved via 
> /taskmanager/, similar to how the job-related URL's are structured, 
> which sadly isn't the case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3137: [hotfix] [doc] Fix several broken "Linking with Fl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3109: [FLINK-5329] Fix metric list being cut off

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3109


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3090: [FLINK-5432] Fix nested files enumeration in Conti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3090


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3079: [FLINK-5424] Improve Restart Strategy Logging

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3079


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3010: [minor] Fix String formats in FlinkDistributionOve...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3010


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3104: [FLINK-5150] [webui] Store metrics in sessionStora...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource

[ramkrish86]

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015379
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java
 ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.RichInputFormat;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.flink.types.Row;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.ClusterConnection;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.HRegionLocator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * {@link InputFormat} subclass that wraps the access for HTables. Returns 
the result as {@link Row}
+ */
+public class HBaseTableSourceInputFormat extends RichInputFormat implements ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HBaseTableSourceInputFormat.class);
+   private String tableName;
+   private TypeInformation[] fieldTypeInfos;
+   private String[] fieldNames;
+   private transient Table table;
+   private transient Scan scan;
+   private transient Connection conn;
+   private ResultScanner resultScanner = null;
+
+   private byte[] lastRow;
+   private int scannedRows;
+   private boolean endReached = false;
+   private org.apache.hadoop.conf.Configuration conf;
+   private static final String COLON = ":";
+
+   public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration 
conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) {
+   this.conf = conf;
+   this.tableName = tableName;
+   this.fieldNames = fieldNames;
+   this.fieldTypeInfos = fieldTypeInfos;
+   }
+
+   @Override
+   public void configure(Configuration parameters) {
+   LOG.info("Initializing HBaseConfiguration");
+   connectToTable();
+   if(table != null) {
+   scan = createScanner();
+   }
+   }
+
+   private Scan createScanner() {
+   Scan scan = new Scan();
+   for(String field : fieldNames) {
+   // select only the fields in the 'selectedFields'
+   String[] famCol = field.split(COLON);
+   scan.addColumn(Bytes.toBytes(famCol[0]), 
Bytes.toBytes(famCol[1]));
+   }
+   return scan;
+   }
+
+   private void connectToTable() {
+   //use files found in the classpath
+   if(this.conf == null) {
+   this.conf = 

[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource

[ramkrish86]

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015321
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSource.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.table.sources.BatchTableSource;
+import org.apache.flink.table.sources.ProjectableTableSource;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Creates a table source that helps to scan data from an hbase table
+ *
+ * Note : the colNames are specified along with a familyName and they are 
seperated by a ':'
+ * For eg, cf1:q1 - where cf1 is the familyName and q1 is the qualifier 
name
+ */
+public class HBaseTableSource implements BatchTableSource, 
ProjectableTableSource {
+
+   private Configuration conf;
+   private String tableName;
+   private byte[] rowKey;
+   private String[] colNames;
+   private TypeInformation[] colTypes;
+
+   public HBaseTableSource(Configuration conf, String tableName, byte[] 
rowKey, String[] colNames,
+   TypeInformation[] 
colTypes) {
+   this.conf = conf;
+   this.tableName = Preconditions.checkNotNull(tableName, "Table  
name");
+   this.rowKey = Preconditions.checkNotNull(rowKey, "Rowkey");
--- End diff --

Yes. That is true but do we always want full table scan? Actually in HBase 
it is better we specify start and end key. So how do we specify that? I have 
not used this rowKey now but I thought it is better to be used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2168) Add HBaseTableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831223#comment-15831223
 ] 

ASF GitHub Bot commented on FLINK-2168:
---

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015379
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java
 ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.RichInputFormat;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.flink.types.Row;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.ClusterConnection;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.HRegionLocator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * {@link InputFormat} subclass that wraps the access for HTables. Returns 
the result as {@link Row}
+ */
+public class HBaseTableSourceInputFormat extends RichInputFormat implements ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HBaseTableSourceInputFormat.class);
+   private String tableName;
+   private TypeInformation[] fieldTypeInfos;
+   private String[] fieldNames;
+   private transient Table table;
+   private transient Scan scan;
+   private transient Connection conn;
+   private ResultScanner resultScanner = null;
+
+   private byte[] lastRow;
+   private int scannedRows;
+   private boolean endReached = false;
+   private org.apache.hadoop.conf.Configuration conf;
+   private static final String COLON = ":";
+
+   public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration 
conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) {
+   this.conf = conf;
+   this.tableName = tableName;
+   this.fieldNames = fieldNames;
+   this.fieldTypeInfos = fieldTypeInfos;
+   }
+
+   @Override
+   public void configure(Configuration parameters) {
+   LOG.info("Initializing HBaseConfiguration");
+   connectToTable();
+   if(table != null) {
+   scan = createScanner();
+   }
+   }
+
+   private Scan createScanner() {
+   Scan scan = new Scan();
+   for(String field : fieldNames) {
+   // select only the fields in the 'selectedFields'
+   String[] famCol = field.split(COLON);
+   scan.addColumn(Bytes.toBytes(famCol[0]), 
Bytes.toBytes(famCol[1]));
  

[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource

[ramkrish86]

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015267
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java
 ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.RichInputFormat;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.flink.types.Row;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.ClusterConnection;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.HRegionLocator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * {@link InputFormat} subclass that wraps the access for HTables. Returns 
the result as {@link Row}
+ */
+public class HBaseTableSourceInputFormat extends RichInputFormat implements ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HBaseTableSourceInputFormat.class);
+   private String tableName;
+   private TypeInformation[] fieldTypeInfos;
+   private String[] fieldNames;
+   private transient Table table;
+   private transient Scan scan;
+   private transient Connection conn;
+   private ResultScanner resultScanner = null;
+
+   private byte[] lastRow;
+   private int scannedRows;
+   private boolean endReached = false;
+   private org.apache.hadoop.conf.Configuration conf;
+   private static final String COLON = ":";
+
+   public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration 
conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) {
+   this.conf = conf;
+   this.tableName = tableName;
+   this.fieldNames = fieldNames;
+   this.fieldTypeInfos = fieldTypeInfos;
+   }
+
+   @Override
+   public void configure(Configuration parameters) {
+   LOG.info("Initializing HBaseConfiguration");
+   connectToTable();
+   if(table != null) {
+   scan = createScanner();
+   }
+   }
+
+   private Scan createScanner() {
+   Scan scan = new Scan();
+   for(String field : fieldNames) {
+   // select only the fields in the 'selectedFields'
+   String[] famCol = field.split(COLON);
+   scan.addColumn(Bytes.toBytes(famCol[0]), 
Bytes.toBytes(famCol[1]));
+   }
+   return scan;
+   }
+
+   private void connectToTable() {
+   //use files found in the classpath
+   if(this.conf == null) {
+   this.conf = 

[jira] [Commented] (FLINK-2168) Add HBaseTableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831221#comment-15831221
 ] 

ASF GitHub Bot commented on FLINK-2168:
---

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015321
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSource.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.table.sources.BatchTableSource;
+import org.apache.flink.table.sources.ProjectableTableSource;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Creates a table source that helps to scan data from an hbase table
+ *
+ * Note : the colNames are specified along with a familyName and they are 
seperated by a ':'
+ * For eg, cf1:q1 - where cf1 is the familyName and q1 is the qualifier 
name
+ */
+public class HBaseTableSource implements BatchTableSource, 
ProjectableTableSource {
+
+   private Configuration conf;
+   private String tableName;
+   private byte[] rowKey;
+   private String[] colNames;
+   private TypeInformation[] colTypes;
+
+   public HBaseTableSource(Configuration conf, String tableName, byte[] 
rowKey, String[] colNames,
+   TypeInformation[] 
colTypes) {
+   this.conf = conf;
+   this.tableName = Preconditions.checkNotNull(tableName, "Table  
name");
+   this.rowKey = Preconditions.checkNotNull(rowKey, "Rowkey");
--- End diff --

Yes. That is true but do we always want full table scan? Actually in HBase 
it is better we specify start and end key. So how do we specify that? I have 
not used this rowKey now but I thought it is better to be used?


> Add HBaseTableSource
> 
>
> Key: FLINK-2168
> URL: https://issues.apache.org/jira/browse/FLINK-2168
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>  Labels: starter
>
> Add a {{HBaseTableSource}} to read data from a HBase table. The 
> {{HBaseTableSource}} should implement the {{ProjectableTableSource}} 
> (FLINK-3848) and {{FilterableTableSource}} (FLINK-3849) interfaces.
> The implementation can be based on Flink's {{TableInputFormat}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2168) Add HBaseTableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831218#comment-15831218
 ] 

ASF GitHub Bot commented on FLINK-2168:
---

Github user ramkrish86 commented on a diff in the pull request:

https://github.com/apache/flink/pull/3149#discussion_r97015267
  
--- Diff: 
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java
 ---
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.io.InputFormat;
+import org.apache.flink.api.common.io.LocatableInputSplitAssigner;
+import org.apache.flink.api.common.io.RichInputFormat;
+import org.apache.flink.api.common.io.statistics.BaseStatistics;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.io.InputSplitAssigner;
+import org.apache.flink.types.Row;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.ClusterConnection;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.HRegionLocator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * {@link InputFormat} subclass that wraps the access for HTables. Returns 
the result as {@link Row}
+ */
+public class HBaseTableSourceInputFormat extends RichInputFormat implements ResultTypeQueryable {
+
+   private static final long serialVersionUID = 1L;
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HBaseTableSourceInputFormat.class);
+   private String tableName;
+   private TypeInformation[] fieldTypeInfos;
+   private String[] fieldNames;
+   private transient Table table;
+   private transient Scan scan;
+   private transient Connection conn;
+   private ResultScanner resultScanner = null;
+
+   private byte[] lastRow;
+   private int scannedRows;
+   private boolean endReached = false;
+   private org.apache.hadoop.conf.Configuration conf;
+   private static final String COLON = ":";
+
+   public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration 
conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) {
+   this.conf = conf;
+   this.tableName = tableName;
+   this.fieldNames = fieldNames;
+   this.fieldTypeInfos = fieldTypeInfos;
+   }
+
+   @Override
+   public void configure(Configuration parameters) {
+   LOG.info("Initializing HBaseConfiguration");
+   connectToTable();
+   if(table != null) {
+   scan = createScanner();
+   }
+   }
+
+   private Scan createScanner() {
+   Scan scan = new Scan();
+   for(String field : fieldNames) {
+   // select only the fields in the 'selectedFields'
+   String[] famCol = field.split(COLON);
+   scan.addColumn(Bytes.toBytes(famCol[0]), 
Bytes.toBytes(famCol[1]));
  

[GitHub] flink issue #3150: [FLINK-4693][tableApi] Add session group-windows for batc...

[sunjincheng121]

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3150
  
@fhueske I have updated the PR according to your comments.The change 
information is as follows：
1. Fix "remember and read input objects across method calls".
2. Amend some code format.
thanks again. SunJincheng.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4693) Add session group-windows for batch tables

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831151#comment-15831151
 ] 

ASF GitHub Bot commented on FLINK-4693:
---

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3150
  
@fhueske I have updated the PR according to your comments.The change 
information is as follows：
1. Fix "remember and read input objects across method calls".
2. Amend some code format.
thanks again. SunJincheng.


> Add session group-windows for batch tables
> ---
>
> Key: FLINK-4693
> URL: https://issues.apache.org/jira/browse/FLINK-4693
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: sunjincheng
>
> Add Session group-windows for batch tables as described in 
> [FLIP-11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations].
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831133#comment-15831133
 ] 

ASF GitHub Bot commented on FLINK-5584:
---

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3175
  
Hi, @hongyuhong , thank your for your job.  Agree with @wuchong 's 
comments. I add same database OVER example for you:
```
Example data:
select * from  PeopleInfo
ID Name Gender Score
6   LiHuan  Man 80
7   LiHuan  Man 90
8   LiMing  Man 56
9   LiMing  Woman   60
10  WangHua Woman   80
  
```
```
 --Simple case 
SELECT name, gender, count(name) OVER () AS num FROM PeopleInfo

namegender num
LiHuan  Man 5
LiHuan  Man 5
LiMing  Man 5
LiMing  Woman   5
WangHuaWoman5

```
```
  --With ORDER BY case
SELECT name,gender,score ROW_NUMBER() OVER (ORDER BY score ASC) AS num FROM 
PeopleInfo

namegender   score   num
LiMing  Man 56  1
LiMing  Woman   60  2
WangHua Woman   80  3
LiHuan  Man 80  4
LiHuan  Man 90  5
```
  
   ```
--With both PARTITION BY and  ORDER BY case
SELECT [name],gender,score, ROW_NUMBER() OVER(PARTITION BY  Gender ORDER BY 
score ASC) as num
FROM PeopleInfo;

namegender   score   num
LiMing  Man 56  1
LiHuan  Man 80  2
LiHuan  Man 90  3
LiMing  Woman   60  1
WangHua Woman   80  2
```
  ```
 --With ROWS  PRECEDING and CURRENT ROW case
SELECT name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY
   id ASC  ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) as  sum
FROM PeopleInfo

name gender score sum
LiHuan  Man 80  80
LiHuan  Man 90  170
LiMing  Man 56  226
LiMing  Woman   60  60
WangHua Woman   80  140

SELECT name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as sum
FROM PeopleInfo

name genderscore sum
LiHuan  Man 80  80
LiHuan  Man 90  170
LiMing  Man 56  146
LiMing  Woman   60  60
WangHua Woman   80  140

```
  ```
--With ROWS FOLLOWING case   

SELECT id, name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER 
BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING ) as sum
FROM dbo.PeopleInfo

id   name genderscore  sum
6   LiHuan  Man 80  170
7   LiHuan  Man 90  226
8   LiMing  Man 56  146
9   LiMing  Woman   60  140
10  WangHua Woman   80  140

SELECT id,name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY
 id ASC  ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING ) as sum
FROM PeopleInfo
 
id   name gender score  sum
6   LiHuan  Man 80  170
7   LiHuan  Man 90  226
8   LiMing  Man 56  226
9   LiMing  Woman   60  140
10  WangHua Woman   80  140

SELECT id, name, gender, score,sum(score) OVER (PARTITION BY gender ORDER BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING ) as sum
FROM PeopleInfo

id   name gender  score  sum
8   LiMing  Man 56  146
7   LiHuan  Man 90  226
6   LiHuan  Man 80  226
10  WangHua Woman   80  140
9   LiMing  Woman   60  140
```
Thank you , SunJincheng.


> Support Sliding-count row-window on streaming sql
> -
>
> 

[GitHub] flink issue #3175: [FLINK-5584]support sliding-count row-window on streaming...

[sunjincheng121]

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/3175
  
Hi, @hongyuhong , thank your for your job.  Agree with @wuchong 's 
comments. I add same database OVER example for you:
```
Example data:
select * from  PeopleInfo
ID Name Gender Score
6   LiHuan  Man 80
7   LiHuan  Man 90
8   LiMing  Man 56
9   LiMing  Woman   60
10  WangHua Woman   80
  
```
```
 --Simple case 
SELECT name, gender, count(name) OVER () AS num FROM PeopleInfo

namegender num
LiHuan  Man 5
LiHuan  Man 5
LiMing  Man 5
LiMing  Woman   5
WangHuaWoman5

```
```
  --With ORDER BY case
SELECT name,gender,score ROW_NUMBER() OVER (ORDER BY score ASC) AS num FROM 
PeopleInfo

namegender   score   num
LiMing  Man 56  1
LiMing  Woman   60  2
WangHua Woman   80  3
LiHuan  Man 80  4
LiHuan  Man 90  5
```
  
   ```
--With both PARTITION BY and  ORDER BY case
SELECT [name],gender,score, ROW_NUMBER() OVER(PARTITION BY  Gender ORDER BY 
score ASC) as num
FROM PeopleInfo;

namegender   score   num
LiMing  Man 56  1
LiHuan  Man 80  2
LiHuan  Man 90  3
LiMing  Woman   60  1
WangHua Woman   80  2
```
  ```
 --With ROWS  PRECEDING and CURRENT ROW case
SELECT name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY
   id ASC  ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) as  sum
FROM PeopleInfo

name gender score sum
LiHuan  Man 80  80
LiHuan  Man 90  170
LiMing  Man 56  226
LiMing  Woman   60  60
WangHua Woman   80  140

SELECT name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as sum
FROM PeopleInfo

name genderscore sum
LiHuan  Man 80  80
LiHuan  Man 90  170
LiMing  Man 56  146
LiMing  Woman   60  60
WangHua Woman   80  140

```
  ```
--With ROWS FOLLOWING case   

SELECT id, name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER 
BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING ) as sum
FROM dbo.PeopleInfo

id   name genderscore  sum
6   LiHuan  Man 80  170
7   LiHuan  Man 90  226
8   LiMing  Man 56  146
9   LiMing  Woman   60  140
10  WangHua Woman   80  140

SELECT id,name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY
 id ASC  ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING ) as sum
FROM PeopleInfo
 
id   name gender score  sum
6   LiHuan  Man 80  170
7   LiHuan  Man 90  226
8   LiMing  Man 56  226
9   LiMing  Woman   60  140
10  WangHua Woman   80  140

SELECT id, name, gender, score,sum(score) OVER (PARTITION BY gender ORDER BY
   id ASC  ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING ) as sum
FROM PeopleInfo

id   name gender  score  sum
8   LiMing  Man 56  146
7   LiHuan  Man 90  226
6   LiHuan  Man 80  226
10  WangHua Woman   80  140
9   LiMing  Woman   60  140
```
Thank you , SunJincheng.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3153: [hotfix] [docs] Insert closing parentheses on "Flink Data...

[keijiyoshida]

Github user keijiyoshida commented on the issue:

https://github.com/apache/flink/pull/3153
  
Thanks for your checking and merging!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831112#comment-15831112
 ] 

ASF GitHub Bot commented on FLINK-5584:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3175
  
Hi @hongyuhong , thank your for your job. But it seems that you 
misunderstand the SQL OVER syntax. 
The OVER clause defines a window or user-specified set of rows within a 
query result set. A window function then computes a value for each row in the 
window. It is similar to Row-Window proposed in 
[FLIP-11](https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations),
 but is different with Sliding Row-count window.

For example, OVER (ROWS 2 PRECEDING) means that the window of rows that the 
function operates on is three rows in size, starting with 2 rows preceding 
until and including the current row.

Say we have a table `T1` 

```
t  a  
-
1  1 
2  5 
3  3 
4  5 
5  4 
6 11
```

and the following SQL will yield:

```sql
SELECT t, a, sum(a) OVER (ROWS 2 PRECEDING) FROM T1
```

```
t  a  avg
--
1  1  1
2  5  6
3  3  9
4  5  13
5  4  12
6 11  20
```

For Row-window, we would need something more complex, especially when we 
need to order by timestamp. For example, to support event-time count-window 
row-window, we need to create a custom operator that collects records in a 
priority queue ordered by timestamp. Once a watermark is received for the upper 
bound of a window, the priority queue is used to evaluate the window function 
(based on count) and to purge too old records. 

I would suggest this PR to wait for FLINK-4679. When FLINK-4679 is fixed, 
this PR can be easily supported IMO.


> Support Sliding-count row-window on streaming sql
> -
>
> Key: FLINK-5584
> URL: https://issues.apache.org/jira/browse/FLINK-5584
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Yuhong Hong
>
> Calcite has already support sliding-count row-window, the grammar look like:
> select sum(amount) over (rows 10 preceding) from Order;
> select sum(amount) over (partition by user rows 10 preceding) from Order;
> And it will parse the sql as a LogicalWindow relnode, the logical Window 
> contains aggregate func info and window info, it's similar to Flink 
> LogicalWIndowAggregate, so we can add an convert rule to directly convert 
> LogicalWindow into DataStreamAggregate relnode, and if Calcite support more 
> grammar, we can extend the convert rule.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3175: [FLINK-5584]support sliding-count row-window on streaming...

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3175
  
Hi @hongyuhong , thank your for your job. But it seems that you 
misunderstand the SQL OVER syntax. 
The OVER clause defines a window or user-specified set of rows within a 
query result set. A window function then computes a value for each row in the 
window. It is similar to Row-Window proposed in 
[FLIP-11](https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations),
 but is different with Sliding Row-count window.

For example, OVER (ROWS 2 PRECEDING) means that the window of rows that the 
function operates on is three rows in size, starting with 2 rows preceding 
until and including the current row.

Say we have a table `T1` 

```
t  a  
-
1  1 
2  5 
3  3 
4  5 
5  4 
6 11
```

and the following SQL will yield:

```sql
SELECT t, a, sum(a) OVER (ROWS 2 PRECEDING) FROM T1
```

```
t  a  avg
--
1  1  1
2  5  6
3  3  9
4  5  13
5  4  12
6 11  20
```

For Row-window, we would need something more complex, especially when we 
need to order by timestamp. For example, to support event-time count-window 
row-window, we need to create a custom operator that collects records in a 
priority queue ordered by timestamp. Once a watermark is received for the upper 
bound of a window, the priority queue is used to evaluate the window function 
(based on count) and to purge too old records. 

I would suggest this PR to wait for FLINK-4679. When FLINK-4679 is fixed, 
this PR can be easily supported IMO.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831036#comment-15831036
 ] 

ASF GitHub Bot commented on FLINK-5584:
---

GitHub user hongyuhong opened a pull request:

https://github.com/apache/flink/pull/3175

[FLINK-5584]support sliding-count row-window on streaming sql

Calcite has already support sliding-count row-window, the grammar look like:
select sum(amount) over (rows 10 preceding) from Order;
select sum(amount) over (partition by user rows 10 preceding) from Order;
And it will parse the sql as a LogicalWindow relnode, the logical Window 
contains aggregate func info and window info, it's similar to Flink 
LogicalWIndowAggregate, so we can add an convert rule to directly convert 
LogicalWindow into DataStreamAggregate relnode.

1. Add HepPlanner to do the window optimize, cause valcano planner can not 
choose the ProjectToWindow optimize as the best.
2. Add DataStreamWindowRule.scala to convert LogicalWindow to 
DataStreamAggregate.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hongyuhong/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3175.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3175


commit 2417e24ad676474df3c9fce6701024ca88c6a439
Author: hongyuhong 00223286 
Date:   2017-01-20T02:02:12Z

support sliding-count row-window on streaming sql




> Support Sliding-count row-window on streaming sql
> -
>
> Key: FLINK-5584
> URL: https://issues.apache.org/jira/browse/FLINK-5584
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Yuhong Hong
>
> Calcite has already support sliding-count row-window, the grammar look like:
> select sum(amount) over (rows 10 preceding) from Order;
> select sum(amount) over (partition by user rows 10 preceding) from Order;
> And it will parse the sql as a LogicalWindow relnode, the logical Window 
> contains aggregate func info and window info, it's similar to Flink 
> LogicalWIndowAggregate, so we can add an convert rule to directly convert 
> LogicalWindow into DataStreamAggregate relnode, and if Calcite support more 
> grammar, we can extend the convert rule.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3175: [FLINK-5584]support sliding-count row-window on st...

[hongyuhong]

GitHub user hongyuhong opened a pull request:

https://github.com/apache/flink/pull/3175

[FLINK-5584]support sliding-count row-window on streaming sql

Calcite has already support sliding-count row-window, the grammar look like:
select sum(amount) over (rows 10 preceding) from Order;
select sum(amount) over (partition by user rows 10 preceding) from Order;
And it will parse the sql as a LogicalWindow relnode, the logical Window 
contains aggregate func info and window info, it's similar to Flink 
LogicalWIndowAggregate, so we can add an convert rule to directly convert 
LogicalWindow into DataStreamAggregate relnode.

1. Add HepPlanner to do the window optimize, cause valcano planner can not 
choose the ProjectToWindow optimize as the best.
2. Add DataStreamWindowRule.scala to convert LogicalWindow to 
DataStreamAggregate.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hongyuhong/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3175.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3175


commit 2417e24ad676474df3c9fce6701024ca88c6a439
Author: hongyuhong 00223286 
Date:   2017-01-20T02:02:12Z

support sliding-count row-window on streaming sql




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5584) Support Sliding-count row-window on streaming sql

[Yuhong Hong (JIRA)]

Yuhong Hong created FLINK-5584:
--

 Summary: Support Sliding-count row-window on streaming sql
 Key: FLINK-5584
 URL: https://issues.apache.org/jira/browse/FLINK-5584
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Yuhong Hong


Calcite has already support sliding-count row-window, the grammar look like:
select sum(amount) over (rows 10 preceding) from Order;
select sum(amount) over (partition by user rows 10 preceding) from Order;

And it will parse the sql as a LogicalWindow relnode, the logical Window 
contains aggregate func info and window info, it's similar to Flink 
LogicalWIndowAggregate, so we can add an convert rule to directly convert 
LogicalWindow into DataStreamAggregate relnode, and if Calcite support more 
grammar, we can extend the convert rule.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3141: [FLINK-5520] [table] Disable outer joins with non-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3141


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5520) Disable outer joins with non-equality predicates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830760#comment-15830760
 ] 

ASF GitHub Bot commented on FLINK-5520:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3141


> Disable outer joins with non-equality predicates
> 
>
> Key: FLINK-5520
> URL: https://issues.apache.org/jira/browse/FLINK-5520
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Assignee: lincoln.lee
>Priority: Blocker
> Fix For: 1.2.0
>
>
> Outer joins with non-equality predicates (and at least one equality 
> predicate) compute incorrect results. 
> Since this is not a very common requirement, I propose to disable this 
> feature for the 1.2.0 release and correctly implement it for a later version.
> The fix should add checks in the Table API validation phase (to get a good 
> error message) and in the DataSetJoinRule to prevent translation of SQL 
> queries with non-equality predicates on outer joins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3095: [FLINK-5452] [table] Fix SortITCase which will fai...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3095


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5452) Make table unit tests pass under cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830759#comment-15830759
 ] 

ASF GitHub Bot commented on FLINK-5452:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3095


> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3153: [hotfix] [docs] Insert closing parentheses on "Fli...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3153


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reassigned FLINK-3679:
-

Assignee: Haohui Mai

> DeserializationSchema should handle zero or more outputs for every input
> 
>
> Key: FLINK-3679
> URL: https://issues.apache.org/jira/browse/FLINK-3679
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API, Kafka Connector
>Reporter: Jamie Grier
>Assignee: Haohui Mai
>
> There are a couple of issues with the DeserializationSchema API that I think 
> should be improved.  This request has come to me via an existing Flink user.
> The main issue is simply that the API assumes that there is a one-to-one 
> mapping between input and outputs.  In reality there are scenarios where one 
> input message (say from Kafka) might actually map to zero or more logical 
> elements in the pipeline.
> Particularly important here is the case where you receive a message from a 
> source (such as Kafka) and say the raw bytes don't deserialize properly.  
> Right now the only recourse is to throw IOException and therefore fail the 
> job.  
> This is definitely not good since bad data is a reality and failing the job 
> is not the right option.  If the job fails we'll just end up replaying the 
> bad data and the whole thing will start again.
> Instead in this case it would be best if the user could just return the empty 
> set.
> The other case is where one input message should logically be multiple output 
> messages.  This case is probably less important since there are other ways to 
> do this but in general it might be good to make the 
> DeserializationSchema.deserialize() method return a collection rather than a 
> single element.
> Maybe we need to support a DeserializationSchema variant that has semantics 
> more like that of FlatMap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-5557) Fix link in library methods

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-5557.
-
Resolution: Fixed

Fixed in master: 4833e74e7a8b514e8dfa243f3f196ce443cf1c50
1.2: 92e39350045f22942773c1605ca788a547c35f21

> Fix link in library methods
> ---
>
> Key: FLINK-5557
> URL: https://issues.apache.org/jira/browse/FLINK-5557
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.2.0
>
>
> The link to "Towards real-time community detection in large networks" is 
> padded with unnecessary and seemingly malformed text.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/library_methods.html#community-detection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5557) Fix link in library methods

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830546#comment-15830546
 ] 

ASF GitHub Bot commented on FLINK-5557:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3163


> Fix link in library methods
> ---
>
> Key: FLINK-5557
> URL: https://issues.apache.org/jira/browse/FLINK-5557
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Trivial
> Fix For: 1.2.0
>
>
> The link to "Towards real-time community detection in large networks" is 
> padded with unnecessary and seemingly malformed text.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/library_methods.html#community-detection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3163: [FLINK-5557] [gelly] Fix link in library methods

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3163


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-5579) Kerberos not working for Kafka connector using ticket cache

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai closed FLINK-5579.
--
Resolution: Not A Bug

> Kerberos not working for Kafka connector using ticket cache
> ---
>
> Key: FLINK-5579
> URL: https://issues.apache.org/jira/browse/FLINK-5579
> Project: Flink
>  Issue Type: Bug
>  Components: Security, YARN
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Setup: security using Kerberos ticket cache, single Flink job deployed on YARN
> The Kerberos ticket cache doesn't seem to be picked up / sent to TaskManager 
> containers when using the Kafka connector when deployed on YARN (when 
> deployed using standalone, this works normally).
> {code}
> Caused by: org.apache.kafka.common.KafkaException: 
> javax.security.auth.login.LoginException: Unable to obtain Princpal Name for 
> authentication
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60)
>   at 
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:271)
>   ... 23 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain 
> Princpal Name for authentication
>   at 
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:804)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:675)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:588)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
>   at 
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
>   at 
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
>   at 
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
>   at 
> javax.security.auth.login.LoginContext.login(LoginContext.java:595)
>   at 
> org.apache.kafka.common.security.kerberos.Login.login(Login.java:298)
>   at 
> org.apache.kafka.common.security.kerberos.Login.(Login.java:104)
>   at 
> org.apache.kafka.common.security.kerberos.LoginManager.(LoginManager.java:44)
>   at 
> org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85)
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5579) Kerberos not working for Kafka connector using ticket cache

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830523#comment-15830523
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-5579:


This actually isn't a bug. With YARN, Kerberos security for Kafka only works 
with keytabs. Please see FLINK-3929.
On the other hand, we should probably make this more clear in the documentation.

> Kerberos not working for Kafka connector using ticket cache
> ---
>
> Key: FLINK-5579
> URL: https://issues.apache.org/jira/browse/FLINK-5579
> Project: Flink
>  Issue Type: Bug
>  Components: Security, YARN
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Setup: security using Kerberos ticket cache, single Flink job deployed on YARN
> The Kerberos ticket cache doesn't seem to be picked up / sent to TaskManager 
> containers when using the Kafka connector when deployed on YARN (when 
> deployed using standalone, this works normally).
> {code}
> Caused by: org.apache.kafka.common.KafkaException: 
> javax.security.auth.login.LoginException: Unable to obtain Princpal Name for 
> authentication
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60)
>   at 
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:271)
>   ... 23 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain 
> Princpal Name for authentication
>   at 
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:804)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:675)
>   at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:588)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
>   at 
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
>   at 
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
>   at 
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at 
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
>   at 
> javax.security.auth.login.LoginContext.login(LoginContext.java:595)
>   at 
> org.apache.kafka.common.security.kerberos.Login.login(Login.java:298)
>   at 
> org.apache.kafka.common.security.kerberos.Login.(Login.java:104)
>   at 
> org.apache.kafka.common.security.kerberos.LoginManager.(LoginManager.java:44)
>   at 
> org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85)
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55)
>   ... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5583) Support flexible error handling in the Kafka consumer

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-5583:
-

 Summary: Support flexible error handling in the Kafka consumer
 Key: FLINK-5583
 URL: https://issues.apache.org/jira/browse/FLINK-5583
 Project: Flink
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


We found that it is valuable to allow the applications to handle errors and 
exceptions in the Kafka consumer in order to build a robust application in 
production.

The context is the following:

(1) We have schematized, Avro records flowing through Kafka.
(2) The decoder implements the DeserializationSchema to decode the records.
(3) Occasionally there are corrupted records (e.g., schema issues). The 
streaming pipeline might want to bail out (which is the current behavior) or to 
skip the corrupted records depending on the applications.

Two options are available:

(1) Have a variant of DeserializationSchema to return a FlatMap like structure 
as suggested in FLINK-3679.
(2) Allow the applications to catch and handle the exception by exposing some 
APIs that are similar to the {{ExceptionProxy}}.

Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading and event time issues

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-5459:
--
Description: 
{{docs/monitoring/debugging_classloading.md}}



  was:
{{docs/monitoring/debugging_classloading.md}}

{{docs/monitoring/debugging_event_time.md}}



> Add documentation how to debug classloading and event time issues
> -
>
> Key: FLINK-5459
> URL: https://issues.apache.org/jira/browse/FLINK-5459
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Stephan Ewen
> Fix For: 1.2.0, 1.3.0
>
>
> {{docs/monitoring/debugging_classloading.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-5459:
--
Summary: Add documentation how to debug classloading  (was: Add 
documentation how to debug classloading and event time issues)

> Add documentation how to debug classloading
> ---
>
> Key: FLINK-5459
> URL: https://issues.apache.org/jira/browse/FLINK-5459
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Stephan Ewen
> Fix For: 1.2.0, 1.3.0
>
>
> {{docs/monitoring/debugging_classloading.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading issues

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-5459:
--
Summary: Add documentation how to debug classloading issues  (was: Add 
documentation how to debug classloading)

> Add documentation how to debug classloading issues
> --
>
> Key: FLINK-5459
> URL: https://issues.apache.org/jira/browse/FLINK-5459
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Stephan Ewen
> Fix For: 1.2.0, 1.3.0
>
>
> {{docs/monitoring/debugging_classloading.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-5555) Add documentation about debugging watermarks

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-.
---
   Resolution: Fixed
Fix Version/s: 1.3.0

Resolved for 1.3 in master: 
http://git-wip-us.apache.org/repos/asf/flink/commit/b86a3d76

Resolved for 1.2 in 7f3c8ff811de620cb9f272d7b6a179bdb9aa33db


> Add documentation about debugging watermarks
> 
>
> Key: FLINK-
> URL: https://issues.apache.org/jira/browse/FLINK-
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.2.0, 1.3.0
>
>
> This was a frequent question on the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5459) Add documentation how to debug classloading and event time issues

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830408#comment-15830408
 ] 

Robert Metzger commented on FLINK-5459:
---

Event time debugging has been addressed in FLINK- already. I'll edit this 
JIRA.

> Add documentation how to debug classloading and event time issues
> -
>
> Key: FLINK-5459
> URL: https://issues.apache.org/jira/browse/FLINK-5459
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Stephan Ewen
> Fix For: 1.2.0, 1.3.0
>
>
> {{docs/monitoring/debugging_classloading.md}}
> {{docs/monitoring/debugging_event_time.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5555) Add documentation about debugging watermarks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830405#comment-15830405
 ] 

ASF GitHub Bot commented on FLINK-:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3170


> Add documentation about debugging watermarks
> 
>
> Key: FLINK-
> URL: https://issues.apache.org/jira/browse/FLINK-
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.2.0
>
>
> This was a frequent question on the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #3170: [FLINK-5555] Document how to debug event time usin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5555) Add documentation about debugging watermarks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830401#comment-15830401
 ] 

ASF GitHub Bot commented on FLINK-:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3170
  
Thanks a lot for the quick review. I'll merge it.


> Add documentation about debugging watermarks
> 
>
> Key: FLINK-
> URL: https://issues.apache.org/jira/browse/FLINK-
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.2.0
>
>
> This was a frequent question on the mailing list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3170: [FLINK-5555] Document how to debug event time using water...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3170
  
Thanks a lot for the quick review. I'll merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5355) Handle AmazonKinesisException gracefully in Kinesis Streaming Connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830326#comment-15830326
 ] 

ASF GitHub Bot commented on FLINK-5355:
---

Github user skidder commented on the issue:

https://github.com/apache/flink/pull/3078
  
Thank you @tzulitai for the feedback on the styling! I've pushed a commit 
that addresses your comments.


> Handle AmazonKinesisException gracefully in Kinesis Streaming Connector
> ---
>
> Key: FLINK-5355
> URL: https://issues.apache.org/jira/browse/FLINK-5355
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Scott Kidder
>Assignee: Scott Kidder
>
> My Flink job that consumes from a Kinesis stream must be restarted at least 
> once daily due to an uncaught AmazonKinesisException when reading from 
> Kinesis. The complete stacktrace looks like:
> {noformat}
> com.amazonaws.services.kinesis.model.AmazonKinesisException: null (Service: 
> AmazonKinesis; Status Code: 500; Error Code: InternalFailure; Request ID: 
> dc1b7a1a-1b97-1a32-8cd5-79a896a55223)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
>   at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
>   at 
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
>   at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1747)
>   at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1723)
>   at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:858)
>   at 
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getRecords(KinesisProxy.java:193)
>   at 
> org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.getRecords(ShardConsumer.java:268)
>   at 
> org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.run(ShardConsumer.java:176)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It's interesting that the Kinesis endpoint returned a 500 status code, but 
> that's outside the scope of this issue.
> I think we can handle this exception in the same manner as a 
> ProvisionedThroughputException: performing an exponential backoff and 
> retrying a finite number of times before throwing an exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3078: [FLINK-5355] Handle AmazonKinesisException gracefully in ...

[skidder]

Github user skidder commented on the issue:

https://github.com/apache/flink/pull/3078
  
Thank you @tzulitai for the feedback on the styling! I've pushed a commit 
that addresses your comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5452) Make table unit tests pass under cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830287#comment-15830287
 ] 

ASF GitHub Bot commented on FLINK-5452:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3095
  
Thanks for fixing the tests @KurtYoung!
+1 to merge


> Make table unit tests pass under cluster mode
> -
>
> Key: FLINK-5452
> URL: https://issues.apache.org/jira/browse/FLINK-5452
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Kurt Young
>Assignee: Kurt Young
>
> Currently if we change the test execution mode to 
> {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will 
> fail. Need to figure out whether it's the case design problem or there are 
> some bugs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #3095: [FLINK-5452] [table] Fix SortITCase which will fail under...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3095
  
Thanks for fixing the tests @KurtYoung!
+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---