[jira] [Commented] (FLINK-9172) Support external catalog factory that comes default with SQL-Client

2018-11-03 Thread Rong Rong (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674101#comment-16674101
 ] 

Rong Rong commented on FLINK-9172:
--

Hi [~xuefuz] I think the scope of this JIRA goes beyond supporting for Hive 
catalog only. There were also some discussions relevant to this on [~twalthr]'s 
recent proposal for unified SQL connector API. 

We also have some internal implementations for this JIRA that works on our 
proprietary catalog system in house. I've summarized some of my very higher 
level insights previously as well. Maybe a short design doc regarding this 
configuration can be drafted to align the goals and together? What do you guys 
think [~suez1224] [~xuefuz].

> Support external catalog factory that comes default with SQL-Client
> ---
>
> Key: FLINK-9172
> URL: https://issues.apache.org/jira/browse/FLINK-9172
> Project: Flink
>  Issue Type: Sub-task
>  Components: SQL Client
>Reporter: Rong Rong
>Assignee: Eron Wright 
>Priority: Major
>
> It doesn't seem that the configuration (YAML) file allows specifications of 
> external catalogs currently. The request here is to add support for external 
> catalog specifications in YAML file. User should also be able to specify one 
> catalog is the default.
> It will be great to have SQL-Client to support some external catalogs 
> out-of-the-box for SQL users to configure and utilize easily. I am currently 
> think of having an external catalog factory that spins up both streaming and 
> batch external catalog table sources and sinks. This could greatly unify and 
> provide easy access for SQL users. 
> The catalog-related configurations then need to be processed and passed to 
> TableEnvironment accordingly by calling relevant APIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10694) ZooKeeperHaServices Cleanup

2018-11-03 Thread Mikhail Pryakhin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Pryakhin updated FLINK-10694:
-
Description: 
When a streaming job with Zookeeper-HA enabled gets cancelled all the 
job-related Zookeeper nodes are not removed. Is there a reason behind that? 
 I noticed that Zookeeper paths are created of type "Container Node" (an 
Ephemeral node that can have nested nodes) and fall back to Persistent node 
type in case Zookeeper doesn't support this sort of nodes. 
 But anyway, it is worth removing the job Zookeeper node when a job is 
cancelled, isn't it?

zookeeper version 3.4.10
 flink version 1.6.1
 # The job is deployed as a YARN cluster with the following properties set
{noformat}
 high-availability: zookeeper
 high-availability.zookeeper.quorum: 
 high-availability.zookeeper.storageDir: hdfs:///
 high-availability.zookeeper.path.root: 
 high-availability.zookeeper.path.namespace: 
{noformat}

 # The job is cancelled via flink cancel  command.

What I've noticed:
 when the job is running the following directory structure is created in 
zookeeper
{noformat}
///leader/resource_manager_lock
///leader/rest_server_lock
///leader/dispatcher_lock
///leader/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///leaderlatch/resource_manager_lock
///leaderlatch/rest_server_lock
///leaderlatch/dispatcher_lock
///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///checkpoints/5c21f00b9162becf5ce25a1cf0e67cde/041
///checkpoint-counter/5c21f00b9162becf5ce25a1cf0e67cde
///running_job_registry/5c21f00b9162becf5ce25a1cf0e67cde
{noformat}
when the job is cancelled some ephemeral nodes disappear, but most of them are 
still there:
{noformat}
///leader/5c21f00b9162becf5ce25a1cf0e67cde
///leaderlatch/resource_manager_lock
///leaderlatch/rest_server_lock
///leaderlatch/dispatcher_lock
///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///checkpoints/
///checkpoint-counter/
///running_job_registry/
{noformat}
Here is the method [1] responsible for cleaning zookeeper folders up [1] which 
is called when a job manager has stopped [2]. 
 And it seems it only cleans up the *running_job_registry* folder, other 
folders stay untouched. I suppose that everything under the 
*///* folder should be cleaned up when the job 
is cancelled.

[1] 
[https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperRunningJobsRegistry.java#L107]
 [2] 
[https://github.com/apache/flink/blob/f087f57749004790b6f5b823d66822c36ae09927/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L332]

  was:
When a streaming job with Zookeeper-HA enabled gets cancelled all the 
job-related Zookeeper nodes are not removed. Is there a reason behind that? 
 I noticed that Zookeeper paths are created of type "Container Node" (an 
Ephemeral node that can have nested nodes) and fall back to Persistent node 
type in case Zookeeper doesn't support this sort of nodes. 
 But anyway, it is worth removing the job Zookeeper node when a job is 
cancelled, isn't it?

zookeeper version 3.4.10
 flink version 1.6.1
 # The job is deployed as a YARN cluster with the following properties set
{noformat}
 high-availability: zookeeper
 high-availability.zookeeper.quorum: 
 high-availability.zookeeper.storageDir: hdfs:///
 high-availability.zookeeper.path.root: 
 high-availability.zookeeper.path.namespace: 
{noformat}

 # The job is cancelled via flink cancel  command.

What I've noticed:
 when the job is running the following directory structure is created in 
zookeeper
{noformat}
///leader/resource_manager_lock
///leader/rest_server_lock
///leader/dispatcher_lock
///leader/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///leaderlatch/resource_manager_lock
///leaderlatch/rest_server_lock
///leaderlatch/dispatcher_lock
///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///checkpoints/5c21f00b9162becf5ce25a1cf0e67cde/041
///checkpoint-counter/5c21f00b9162becf5ce25a1cf0e67cde
///running_job_registry/5c21f00b9162becf5ce25a1cf0e67cde
{noformat}
when the job is cancelled some ephemeral nodes disappear, but most of them are 
still there:
{noformat}
///leader/5c21f00b9162becf5ce25a1cf0e67cde
///leaderlatch/resource_manager_lock
///leaderlatch/rest_server_lock
///leaderlatch/dispatcher_lock
///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
///checkpoints/
///checkpoint-counter/
///running_job_registry/
{noformat}
Here is the method [1] responsible for cleaning zookeeper folders up [1] which 
is called when the job manager has stopped [2]. 
 And it seems it only cleans up the folder *running_job_registry*, other 
folders stay untouched. I suppose that everything under the 
*///* folder is cleaned up when the job is 
cancelled.

[1]

[jira] [Assigned] (FLINK-10306) Support the display of log file from any position on webUI

2018-11-03 Thread Jiayi Liao (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiayi Liao reassigned FLINK-10306:
--

Assignee: zhangxinyu  (was: Jiayi Liao)

> Support the display of log file from any position on webUI
> --
>
> Key: FLINK-10306
> URL: https://issues.apache.org/jira/browse/FLINK-10306
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Jiayi Liao
>Assignee: zhangxinyu
>Priority: Major
>
> Although we copy the whole log files from taskmanager to blob service host, 
> sometimes we may not be able to read the whole file's content on WebUI 
> because of the browser's load.
> We already use RandomAccessFile to read files, so I think we need to support 
> read the log file from any row of it by adding a parameter at the end of url 
> like http://xxx/#/taskmanager/container_xx/log/100.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-10351) Ignore specific operator's state when restore from savepoint

2018-11-03 Thread Jiayi Liao (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiayi Liao closed FLINK-10351.
--
Resolution: Not A Problem

> Ignore specific operator's state when restore from savepoint
> 
>
> Key: FLINK-10351
> URL: https://issues.apache.org/jira/browse/FLINK-10351
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.6.0
>Reporter: Jiayi Liao
>Assignee: Jiayi Liao
>Priority: Major
>
> The story is that I want to change autoWatermarkInterval, but I find that it 
> didn't help because the processing time service is restored from the state 
> backend.
> So I wonder if we can provide a command like --ignoreState  to 
> let users abandon the state they don't want?
> I think it'll be helpful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10770) Some generated functions are not opened properly.

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674050#comment-16674050
 ] 

ASF GitHub Bot commented on FLINK-10770:


jrthe42 opened a new pull request #7014: [FLINK-10770] [Table & SQL] Some 
generated functions are not opened properly.
URL: https://github.com/apache/flink/pull/7014
 
 
   ## What is the purpose of the change
   When transforming sql to execution plan, some generated functions' `open` 
method are not called., this may cause exception if the UDFs need be opened. 
This PR fix this issue.
   
   ## Brief change log
- Call `open` method after generated functions are complied and 
instantiated.
   
   ## Verifying this change
   This change added tests and can be verified as follows:
 - org.apache.flink.table.runtime.stream.table.UdfITCase
 - org.apache.flink.table.runtime.batch.table.UdfITCase
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (**not** applicable / docs / 
JavaDocs / not documented)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Some generated functions are not opened properly.
> -
>
> Key: FLINK-10770
> URL: https://issues.apache.org/jira/browse/FLINK-10770
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Affects Versions: 1.6.2, 1.7.0
>Reporter: wangsan
>Assignee: wangsan
>Priority: Major
>  Labels: pull-request-available
>
> Recently I found sometimes UDFs are not open properly. It turns out when 
> transforming sql to execution plan, some generated functions' *open* method 
> are not called. e.g. *NonWindowJoin*, *TimeBoundedStreamJoin*, 
> *FlatJoinRunner*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10770) Some generated functions are not opened properly.

2018-11-03 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-10770:
---
Labels: pull-request-available  (was: )

> Some generated functions are not opened properly.
> -
>
> Key: FLINK-10770
> URL: https://issues.apache.org/jira/browse/FLINK-10770
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Affects Versions: 1.6.2, 1.7.0
>Reporter: wangsan
>Assignee: wangsan
>Priority: Major
>  Labels: pull-request-available
>
> Recently I found sometimes UDFs are not open properly. It turns out when 
> transforming sql to execution plan, some generated functions' *open* method 
> are not called. e.g. *NonWindowJoin*, *TimeBoundedStreamJoin*, 
> *FlatJoinRunner*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] jrthe42 opened a new pull request #7014: [FLINK-10770] [Table & SQL] Some generated functions are not opened properly.

2018-11-03 Thread GitBox 
jrthe42 opened a new pull request #7014: [FLINK-10770] [Table & SQL] Some 
generated functions are not opened properly.
URL: https://github.com/apache/flink/pull/7014
 
 
   ## What is the purpose of the change
   When transforming sql to execution plan, some generated functions' `open` 
method are not called., this may cause exception if the UDFs need be opened. 
This PR fix this issue.
   
   ## Brief change log
- Call `open` method after generated functions are complied and 
instantiated.
   
   ## Verifying this change
   This change added tests and can be verified as follows:
 - org.apache.flink.table.runtime.stream.table.UdfITCase
 - org.apache.flink.table.runtime.batch.table.UdfITCase
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (**not** applicable / docs / 
JavaDocs / not documented)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-5601) Window operator does not checkpoint watermarks

2018-11-03 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-5601:
--
Labels: pull-request-available  (was: )

> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Ufuk Celebi
>Assignee: Jiayi Liao
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.8.0
>
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that 
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an 
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and 
> rescaling and ran into failures, because the data generator required 
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not 
> dropped, because the watermarks needed to be re-established after restore 
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not 
> checkpointing watermarks?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-5601) Window operator does not checkpoint watermarks

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674048#comment-16674048
 ] 

ASF GitHub Bot commented on FLINK-5601:
---

buptljy opened a new pull request #7013: [FLINK-5601][Checkpointing] Watermark 
checkpointing
URL: https://github.com/apache/flink/pull/7013
 
 
   ## What is the purpose of the change
   
   This pull request uses ListState in 
TimestampsAndPeriodicWatermarksOperator.java and 
TimestampsAndPunctuatedWatermarksOperator.java to checkpoint watermark. During 
recovering process, it uses union state to find the lowest watermark and emit 
it immediately.
   
   ## Brief change log
   * Add ListState to store watermark in 
TimestampsAndPeriodicWatermarksOperator.java and 
TimestampsAndPunctuatedWatermarksOperator.java.
   
   ## Verifying this change
   
   * Unit testing in WatermarkCheckpointingITCase.java. 
   * TimestampsAndPeriodicWatermarksOperator testing:  Send five "Five", 
then checkpoint, then make the job fail, then recover the job, and send five 
"Three", the sum should be 25.
   * TimestampsAndPunctuatedWatermarksOperator testing: Similar testing 
logic machenism as above.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Ufuk Celebi
>Assignee: Jiayi Liao
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.8.0
>
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that 
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an 
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and 
> rescaling and ran into failures, because the data generator required 
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not 
> dropped, because the watermarks needed to be re-established after restore 
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not 
> checkpointing watermarks?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] buptljy opened a new pull request #7013: [FLINK-5601][Checkpointing] Watermark checkpointing

2018-11-03 Thread GitBox 
buptljy opened a new pull request #7013: [FLINK-5601][Checkpointing] Watermark 
checkpointing
URL: https://github.com/apache/flink/pull/7013
 
 
   ## What is the purpose of the change
   
   This pull request uses ListState in 
TimestampsAndPeriodicWatermarksOperator.java and 
TimestampsAndPunctuatedWatermarksOperator.java to checkpoint watermark. During 
recovering process, it uses union state to find the lowest watermark and emit 
it immediately.
   
   ## Brief change log
   * Add ListState to store watermark in 
TimestampsAndPeriodicWatermarksOperator.java and 
TimestampsAndPunctuatedWatermarksOperator.java.
   
   ## Verifying this change
   
   * Unit testing in WatermarkCheckpointingITCase.java. 
   * TimestampsAndPeriodicWatermarksOperator testing:  Send five "Five", 
then checkpoint, then make the job fail, then recover the job, and send five 
"Three", the sum should be 25.
   * TimestampsAndPunctuatedWatermarksOperator testing: Similar testing 
logic machenism as above.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-5601) Window operator does not checkpoint watermarks

2018-11-03 Thread Jiayi Liao (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674042#comment-16674042
 ] 

Jiayi Liao commented on FLINK-5601:
---

[~aljoscha] Sorry for attaching this again. Could you tell me more details 
about the backwards compatibility you mentioned? I inspected the source code 
today and found that users don't need to change anything with this new 
improvement, because the context.isRestored returns false. And, I succeeded to 
recover a job from new flink-dist jar(watermark checkpointing) from a 
savepoint(no watermark checkpointing).


> Window operator does not checkpoint watermarks
> --
>
> Key: FLINK-5601
> URL: https://issues.apache.org/jira/browse/FLINK-5601
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Ufuk Celebi
>Assignee: Jiayi Liao
>Priority: Critical
> Fix For: 1.8.0
>
>
> During release testing [~stefanrichte...@gmail.com] and I noticed that 
> watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an 
> adjusted {{SessionWindowITCase}} via Kafka for testing migration and 
> rescaling and ran into failures, because the data generator required 
> determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not 
> dropped, because the watermarks needed to be re-established after restore 
> first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not 
> checkpointing watermarks?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10694) ZooKeeperHaServices Cleanup

2018-11-03 Thread vinoyang (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674029#comment-16674029
 ] 

vinoyang commented on FLINK-10694:
--

[~till.rohrmann] I think this cleaning operation is necessary. Before I 
implement it, I should confirm it with you first.

> ZooKeeperHaServices Cleanup
> ---
>
> Key: FLINK-10694
> URL: https://issues.apache.org/jira/browse/FLINK-10694
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.6.1, 1.7.0
>Reporter: Mikhail Pryakhin
>Assignee: vinoyang
>Priority: Critical
> Fix For: 1.6.3, 1.7.0
>
>
> When a streaming job with Zookeeper-HA enabled gets cancelled all the 
> job-related Zookeeper nodes are not removed. Is there a reason behind that? 
>  I noticed that Zookeeper paths are created of type "Container Node" (an 
> Ephemeral node that can have nested nodes) and fall back to Persistent node 
> type in case Zookeeper doesn't support this sort of nodes. 
>  But anyway, it is worth removing the job Zookeeper node when a job is 
> cancelled, isn't it?
> zookeeper version 3.4.10
>  flink version 1.6.1
>  # The job is deployed as a YARN cluster with the following properties set
> {noformat}
>  high-availability: zookeeper
>  high-availability.zookeeper.quorum: 
>  high-availability.zookeeper.storageDir: hdfs:///
>  high-availability.zookeeper.path.root: 
>  high-availability.zookeeper.path.namespace: 
> {noformat}
>  # The job is cancelled via flink cancel  command.
> What I've noticed:
>  when the job is running the following directory structure is created in 
> zookeeper
> {noformat}
> ///leader/resource_manager_lock
> ///leader/rest_server_lock
> ///leader/dispatcher_lock
> ///leader/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///leaderlatch/resource_manager_lock
> ///leaderlatch/rest_server_lock
> ///leaderlatch/dispatcher_lock
> ///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///checkpoints/5c21f00b9162becf5ce25a1cf0e67cde/041
> ///checkpoint-counter/5c21f00b9162becf5ce25a1cf0e67cde
> ///running_job_registry/5c21f00b9162becf5ce25a1cf0e67cde
> {noformat}
> when the job is cancelled some ephemeral nodes disappear, but most of them 
> are still there:
> {noformat}
> ///leader/5c21f00b9162becf5ce25a1cf0e67cde
> ///leaderlatch/resource_manager_lock
> ///leaderlatch/rest_server_lock
> ///leaderlatch/dispatcher_lock
> ///leaderlatch/5c21f00b9162becf5ce25a1cf0e67cde/job_manager_lock
> ///checkpoints/
> ///checkpoint-counter/
> ///running_job_registry/
> {noformat}
> Here is the method [1] responsible for cleaning zookeeper folders up [1] 
> which is called when the job manager has stopped [2]. 
>  And it seems it only cleans up the folder *running_job_registry*, other 
> folders stay untouched. I suppose that everything under the 
> *///* folder is cleaned up when the job is 
> cancelled.
> [1] 
> [https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperRunningJobsRegistry.java#L107]
>  [2] 
> [https://github.com/apache/flink/blob/f087f57749004790b6f5b823d66822c36ae09927/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L332]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-9999) Add ISNUMERIC supported in Table API/SQL

2018-11-03 Thread vinoyang (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang closed FLINK-.
---
Resolution: Won't Do

> Add ISNUMERIC supported in Table API/SQL
> 
>
> Key: FLINK-
> URL: https://issues.apache.org/jira/browse/FLINK-
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>  Labels: pull-request-available
>
> ISNUMERIC function used to verify a expression is a valid numberic type.
> documentation : 
> https://docs.microsoft.com/en-us/sql/t-sql/functions/isnumeric-transact-sql?view=sql-server-2017



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-10685) Support history server on YARN

2018-11-03 Thread vinoyang (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang closed FLINK-10685.

Resolution: Cannot Reproduce

> Support history server on YARN
> --
>
> Key: FLINK-10685
> URL: https://issues.apache.org/jira/browse/FLINK-10685
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (FLINK-10770) Some generated functions are not opened properly.

2018-11-03 Thread wangsan (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wangsan reassigned FLINK-10770:
---

Assignee: wangsan

> Some generated functions are not opened properly.
> -
>
> Key: FLINK-10770
> URL: https://issues.apache.org/jira/browse/FLINK-10770
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Affects Versions: 1.6.2, 1.7.0
>Reporter: wangsan
>Assignee: wangsan
>Priority: Major
>
> Recently I found sometimes UDFs are not open properly. It turns out when 
> transforming sql to execution plan, some generated functions' *open* method 
> are not called. e.g. *NonWindowJoin*, *TimeBoundedStreamJoin*, 
> *FlatJoinRunner*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-10770) Some generated functions are not opened properly.

2018-11-03 Thread wangsan (JIRA) 
wangsan created FLINK-10770:
---

 Summary: Some generated functions are not opened properly.
 Key: FLINK-10770
 URL: https://issues.apache.org/jira/browse/FLINK-10770
 Project: Flink
  Issue Type: Bug
  Components: Table API  SQL
Affects Versions: 1.6.2, 1.7.0
Reporter: wangsan


Recently I found sometimes UDFs are not open properly. It turns out when 
transforming sql to execution plan, some generated functions' *open* method are 
not called. e.g. *NonWindowJoin*, *TimeBoundedStreamJoin*, *FlatJoinRunner*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10757) TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673983#comment-16673983
 ] 

ASF GitHub Bot commented on FLINK-10757:


asfgit closed pull request #6998: [FLINK-10757] [tests] Avoid port conflicts in 
AbstractTaskManagerProc…
URL: https://github.com/apache/flink/pull/6998
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
 
b/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
index 83298aa78ec..8065d1ad6c7 100644
--- 
a/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
+++ 
b/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
@@ -24,6 +24,7 @@
 import org.apache.flink.configuration.HeartbeatManagerOptions;
 import org.apache.flink.configuration.HighAvailabilityOptions;
 import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.configuration.TaskManagerOptions;
 import org.apache.flink.runtime.clusterframework.types.ResourceID;
 import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
@@ -98,6 +99,7 @@ public void testTaskManagerProcessFailure() throws Exception {
Configuration config = new Configuration();
config.setString(AkkaOptions.ASK_TIMEOUT, "100 s");
config.setString(JobManagerOptions.ADDRESS, "localhost");
+   config.setInteger(RestOptions.PORT, 0);
config.setLong(HeartbeatManagerOptions.HEARTBEAT_INTERVAL, 
500L);
config.setLong(HeartbeatManagerOptions.HEARTBEAT_TIMEOUT, 
1L);
config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the 
> cluster entrypoint StandaloneSessionClusterEntrypoint.
> 
>
> Key: FLINK-10757
> URL: https://issues.apache.org/jira/browse/FLINK-10757
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Bowen Li
>Assignee: TisonKun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {code:java}
> Failed tests: 
>   ...
>  
> TaskManagerProcessFailureStreamingRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure:223
>  Failed to initialize the cluster entrypoint 
> StandaloneSessionClusterEntrypoint.
> Tests in error: 
> {code}
> https://travis-ci.org/apache/flink/jobs/449439623



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] asfgit closed pull request #6998: [FLINK-10757] [tests] Avoid port conflicts in AbstractTaskManagerProc…

2018-11-03 Thread GitBox 
asfgit closed pull request #6998: [FLINK-10757] [tests] Avoid port conflicts in 
AbstractTaskManagerProc…
URL: https://github.com/apache/flink/pull/6998
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
 
b/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
index 83298aa78ec..8065d1ad6c7 100644
--- 
a/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
+++ 
b/flink-tests/src/test/java/org/apache/flink/test/recovery/AbstractTaskManagerProcessFailureRecoveryTest.java
@@ -24,6 +24,7 @@
 import org.apache.flink.configuration.HeartbeatManagerOptions;
 import org.apache.flink.configuration.HighAvailabilityOptions;
 import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.configuration.RestOptions;
 import org.apache.flink.configuration.TaskManagerOptions;
 import org.apache.flink.runtime.clusterframework.types.ResourceID;
 import org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint;
@@ -98,6 +99,7 @@ public void testTaskManagerProcessFailure() throws Exception {
Configuration config = new Configuration();
config.setString(AkkaOptions.ASK_TIMEOUT, "100 s");
config.setString(JobManagerOptions.ADDRESS, "localhost");
+   config.setInteger(RestOptions.PORT, 0);
config.setLong(HeartbeatManagerOptions.HEARTBEAT_INTERVAL, 
500L);
config.setLong(HeartbeatManagerOptions.HEARTBEAT_TIMEOUT, 
1L);
config.setString(HighAvailabilityOptions.HA_MODE, "zookeeper");


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10715) E2e tests fail with ConcurrentModificationException in MetricRegistryImpl

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673984#comment-16673984
 ] 

ASF GitHub Bot commented on FLINK-10715:


asfgit closed pull request #7008: [FLINK-10715][metrics] Suppress 
ConcurrentModificationException thrown by the Slf4jReporter
URL: https://github.com/apache/flink/pull/7008
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
 
b/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
index 124efe227a3..ce6acb0c9ac 100644
--- 
a/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
+++ 
b/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
@@ -33,6 +33,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ConcurrentModificationException;
 import java.util.Map;
 
 /**
@@ -75,6 +76,16 @@ public void close() {
 
@Override
public void report() {
+   try {
+   tryReport();
+   }
+   catch (ConcurrentModificationException ignored) {
+   // at tryReport() we don't synchronize while iterating 
over the various maps which might cause a
+   // ConcurrentModificationException to be thrown, if 
concurrently a metric is being added or removed.
+   }
+   }
+
+   private void tryReport() {
// initialize with previous size to avoid repeated resizing of 
backing array
// pad the size to allow deviations in the final string, for 
example due to different double value representations
StringBuilder builder = new StringBuilder((int) (previousSize * 
1.1));


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
> -
>
> Key: FLINK-10715
> URL: https://issues.apache.org/jira/browse/FLINK-10715
> Project: Flink
>  Issue Type: Bug
>  Components: E2E Tests, Metrics
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Assignee: Igal Shilman
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> Couple of e2e tests that rely on metrics fail with exception:
> {code}
> 2018-10-29 11:40:32,781 WARN  
> org.apache.flink.runtime.metrics.MetricRegistryImpl   - Error while 
> reporting metrics
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
>   at 
> org.apache.flink.metrics.slf4j.Slf4jReporter.report(Slf4jReporter.java:101)
>   at 
> org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:427)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Tests that failed:
> *  'Resuming Externalized Checkpoint (file, sync, no parallelism change) 
> end-to-end test
> * 'State TTL Heap backend end-to-end test'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-10715) E2e tests fail with ConcurrentModificationException in MetricRegistryImpl

2018-11-03 Thread Till Rohrmann (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-10715.
---
   Resolution: Fixed
Fix Version/s: 1.7.0
   1.6.3
   1.5.6

Fixed via 

1.7.0: e91848858cbb760f4429ab306c82232bf9854cb8
1.6.3: 246fd8e212eff48505fe037a783ed03288a4cae6
1.5.6: 04496e912d58b4056f7b98b0b7f0c017db6eea1b

> E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
> -
>
> Key: FLINK-10715
> URL: https://issues.apache.org/jira/browse/FLINK-10715
> Project: Flink
>  Issue Type: Bug
>  Components: E2E Tests, Metrics
>Affects Versions: 1.7.0
>Reporter: Dawid Wysakowicz
>Assignee: Igal Shilman
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> Couple of e2e tests that rely on metrics fail with exception:
> {code}
> 2018-10-29 11:40:32,781 WARN  
> org.apache.flink.runtime.metrics.MetricRegistryImpl   - Error while 
> reporting metrics
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
>   at 
> org.apache.flink.metrics.slf4j.Slf4jReporter.report(Slf4jReporter.java:101)
>   at 
> org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:427)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Tests that failed:
> *  'Resuming Externalized Checkpoint (file, sync, no parallelism change) 
> end-to-end test
> * 'State TTL Heap backend end-to-end test'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] asfgit closed pull request #7008: [FLINK-10715][metrics] Suppress ConcurrentModificationException thrown by the Slf4jReporter

2018-11-03 Thread GitBox 
asfgit closed pull request #7008: [FLINK-10715][metrics] Suppress 
ConcurrentModificationException thrown by the Slf4jReporter
URL: https://github.com/apache/flink/pull/7008
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
 
b/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
index 124efe227a3..ce6acb0c9ac 100644
--- 
a/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
+++ 
b/flink-metrics/flink-metrics-slf4j/src/main/java/org/apache/flink/metrics/slf4j/Slf4jReporter.java
@@ -33,6 +33,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ConcurrentModificationException;
 import java.util.Map;
 
 /**
@@ -75,6 +76,16 @@ public void close() {
 
@Override
public void report() {
+   try {
+   tryReport();
+   }
+   catch (ConcurrentModificationException ignored) {
+   // at tryReport() we don't synchronize while iterating 
over the various maps which might cause a
+   // ConcurrentModificationException to be thrown, if 
concurrently a metric is being added or removed.
+   }
+   }
+
+   private void tryReport() {
// initialize with previous size to avoid repeated resizing of 
backing array
// pad the size to allow deviations in the final string, for 
example due to different double value representations
StringBuilder builder = new StringBuilder((int) (previousSize * 
1.1));


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (FLINK-10757) TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the cluster entrypoint StandaloneSessionClusterEntrypoint.

2018-11-03 Thread Till Rohrmann (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-10757.
---
Resolution: Fixed

Fixed via d3696e54d4e2a59268b283db1bbd0085f8ff168a

> TaskManagerProcessFailureStreamingRecoveryITCase failed to initialize the 
> cluster entrypoint StandaloneSessionClusterEntrypoint.
> 
>
> Key: FLINK-10757
> URL: https://issues.apache.org/jira/browse/FLINK-10757
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Bowen Li
>Assignee: TisonKun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> {code:java}
> Failed tests: 
>   ...
>  
> TaskManagerProcessFailureStreamingRecoveryITCase>AbstractTaskManagerProcessFailureRecoveryTest.testTaskManagerProcessFailure:223
>  Failed to initialize the cluster entrypoint 
> StandaloneSessionClusterEntrypoint.
> Tests in error: 
> {code}
> https://travis-ci.org/apache/flink/jobs/449439623



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673973#comment-16673973
 ] 

ASF GitHub Bot commented on FLINK-10463:


xccui commented on issue #6888: [FLINK-10463][table] Null literal cannot be 
properly parsed in Java Table API function call
URL: https://github.com/apache/flink/pull/6888#issuecomment-435572138
 
 
   Sorry for the late response @hequn8128. As illustrated in [this 
stackoverflow 
question](https://stackoverflow.com/questions/26453870/scala-packratparsers-does-not-backtrack-as-it-should),
 patterns reordering could be the proper way to deal with this problem. +1 to 
your solution.
   
   BTW, @twalthr, it seems that the doc for expression syntax in `TableAPI.md` 
has not been maintained for a long time. As the syntax being more and more 
complicated, do you think we should still keep them “strongly consistent” with 
the codes? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
>  Issue Type: Bug
>  Components: Table API  SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws 
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse 
> expression at column 13: string matching regex 
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
>   at 
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
>   at 
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] xccui commented on issue #6888: [FLINK-10463][table] Null literal cannot be properly parsed in Java Table API function call

2018-11-03 Thread GitBox 
xccui commented on issue #6888: [FLINK-10463][table] Null literal cannot be 
properly parsed in Java Table API function call
URL: https://github.com/apache/flink/pull/6888#issuecomment-435572138
 
 
   Sorry for the late response @hequn8128. As illustrated in [this 
stackoverflow 
question](https://stackoverflow.com/questions/26453870/scala-packratparsers-does-not-backtrack-as-it-should),
 patterns reordering could be the proper way to deal with this problem. +1 to 
your solution.
   
   BTW, @twalthr, it seems that the doc for expression syntax in `TableAPI.md` 
has not been maintained for a long time. As the syntax being more and more 
complicated, do you think we should still keep them “strongly consistent” with 
the codes? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10769) port InMemoryExternalCatalog to java

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673959#comment-16673959
 ] 

ASF GitHub Bot commented on FLINK-10769:


TisonKun commented on a change in pull request #7012: [FLINK-10769][Table & 
SQL] port InMemoryExternalCatalog to java
URL: https://github.com/apache/flink/pull/7012#discussion_r230547952
 
 

 ##
 File path: 
flink-libraries/flink-table/src/main/java/org/apache/flink/table/catalog/InMemoryExternalCatalog.java
 ##
 @@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.table.api.CatalogAlreadyExistException;
+import org.apache.flink.table.api.CatalogNotExistException;
+import org.apache.flink.table.api.TableAlreadyExistException;
+import org.apache.flink.table.api.TableNotExistException;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This class is an in-memory implementation of [[ExternalCatalog]].
 
 Review comment:
   `{@link ExternalCatalog}`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> port InMemoryExternalCatalog to java
> 
>
> Key: FLINK-10769
> URL: https://issues.apache.org/jira/browse/FLINK-10769
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.8.0
>
>
> In the Flink-Hive integration design, we propose a new FlinkInMemoryCatalog 
> (FLINK-10697) for production use. FlinkInMemoryCatalog will share some part 
> with the existing InMemoryExternalCatalog, thus we need to make changes to 
> InMemoryExternalCatalog.
> As we are moving away from Scala to Java, we should write all new 
> code/feature in Java. Therefore, we will port InMemoryExternalCatalog to java 
> first without any feature or behavior change.
> This is a pre-requisite for FLINK-10697



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] TisonKun commented on a change in pull request #7012: [FLINK-10769][Table & SQL] port InMemoryExternalCatalog to java

2018-11-03 Thread GitBox 
TisonKun commented on a change in pull request #7012: [FLINK-10769][Table & 
SQL] port InMemoryExternalCatalog to java
URL: https://github.com/apache/flink/pull/7012#discussion_r230547952
 
 

 ##
 File path: 
flink-libraries/flink-table/src/main/java/org/apache/flink/table/catalog/InMemoryExternalCatalog.java
 ##
 @@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.table.api.CatalogAlreadyExistException;
+import org.apache.flink.table.api.CatalogNotExistException;
+import org.apache.flink.table.api.TableAlreadyExistException;
+import org.apache.flink.table.api.TableNotExistException;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This class is an in-memory implementation of [[ExternalCatalog]].
 
 Review comment:
   `{@link ExternalCatalog}`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-10769) port InMemoryExternalCatalog to java

2018-11-03 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/FLINK-10769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-10769:
---
Labels: pull-request-available  (was: )

> port InMemoryExternalCatalog to java
> 
>
> Key: FLINK-10769
> URL: https://issues.apache.org/jira/browse/FLINK-10769
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.8.0
>
>
> In the Flink-Hive integration design, we propose a new FlinkInMemoryCatalog 
> (FLINK-10697) for production use. FlinkInMemoryCatalog will share some part 
> with the existing InMemoryExternalCatalog, thus we need to make changes to 
> InMemoryExternalCatalog.
> As we are moving away from Scala to Java, we should write all new 
> code/feature in Java. Therefore, we will port InMemoryExternalCatalog to java 
> first without any feature or behavior change.
> This is a pre-requisite for FLINK-10697



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10769) port InMemoryExternalCatalog to java

2018-11-03 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/FLINK-10769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673944#comment-16673944
 ] 

ASF GitHub Bot commented on FLINK-10769:


bowenli86 opened a new pull request #7012: [FLINK-10769] port 
InMemoryExternalCatalog to java
URL: https://github.com/apache/flink/pull/7012
 
 
   ## What is the purpose of the change
   
   In the Flink-Hive integration design, we propose a new 
`FlinkInMemoryCatalog` (FLINK-10697) for production use, unlink 
`InMemoryExternalCatalog` which is only used for testing and dev work. 
`FlinkInMemoryCatalog` will share some part with the existing 
`InMemoryExternalCatalog`, thus we need to make changes to 
`InMemoryExternalCatalog`.
   
   As discussed with @twalthr , we are moving away from Scala to Java, and we 
should write all new code/feature in Java to minimize migration efforts. 
Therefore, we will port `InMemoryExternalCatalog` to java first. This PR  only 
port scala to java with NO feature or behavior change.
   
   This is a pre-requisite for FLINK-10697
   
   ## Brief change log
   
   port `InMemoryExternalCatalog` and `InMemoryExternalCatalogTest` to java
   
   ## Verifying this change
   
   This change is already covered by existing tests, such as 
*InMemoryExternalCatalog*.
   
   ## Does this pull request potentially affect one of the following parts:
   
   none
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> port InMemoryExternalCatalog to java
> 
>
> Key: FLINK-10769
> URL: https://issues.apache.org/jira/browse/FLINK-10769
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API  SQL
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.8.0
>
>
> In the Flink-Hive integration design, we propose a new FlinkInMemoryCatalog 
> (FLINK-10697) for production use. FlinkInMemoryCatalog will share some part 
> with the existing InMemoryExternalCatalog, thus we need to make changes to 
> InMemoryExternalCatalog.
> As we are moving away from Scala to Java, we should write all new 
> code/feature in Java. Therefore, we will port InMemoryExternalCatalog to java 
> first without any feature or behavior change.
> This is a pre-requisite for FLINK-10697



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] bowenli86 opened a new pull request #7012: [FLINK-10769] port InMemoryExternalCatalog to java

2018-11-03 Thread GitBox 
bowenli86 opened a new pull request #7012: [FLINK-10769] port 
InMemoryExternalCatalog to java
URL: https://github.com/apache/flink/pull/7012
 
 
   ## What is the purpose of the change
   
   In the Flink-Hive integration design, we propose a new 
`FlinkInMemoryCatalog` (FLINK-10697) for production use, unlink 
`InMemoryExternalCatalog` which is only used for testing and dev work. 
`FlinkInMemoryCatalog` will share some part with the existing 
`InMemoryExternalCatalog`, thus we need to make changes to 
`InMemoryExternalCatalog`.
   
   As discussed with @twalthr , we are moving away from Scala to Java, and we 
should write all new code/feature in Java to minimize migration efforts. 
Therefore, we will port `InMemoryExternalCatalog` to java first. This PR  only 
port scala to java with NO feature or behavior change.
   
   This is a pre-requisite for FLINK-10697
   
   ## Brief change log
   
   port `InMemoryExternalCatalog` and `InMemoryExternalCatalogTest` to java
   
   ## Verifying this change
   
   This change is already covered by existing tests, such as 
*InMemoryExternalCatalog*.
   
   ## Does this pull request potentially affect one of the following parts:
   
   none
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services