[jira] [Created] (FLINK-4837) flink-streaming-akka source connector
Subhobrata Dey created FLINK-4837: - Summary: flink-streaming-akka source connector Key: FLINK-4837 URL: https://issues.apache.org/jira/browse/FLINK-4837 Project: Flink Issue Type: New Feature Components: Streaming Connectors Reporter: Subhobrata Dey Hello, This issue is created to propose the idea of having a flink-streaming-akka source connector. The source connector can be used to receive messages from an Akka feeder or publisher actor & these messages can then be processed using flink streaming. The source connector has the following features. 1. It can supports several different message formats like iterable data, bytes array & data with timestamp. 2. It can send back acknowledgements to the feeder actor. Thanks & regards, Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3857) Add reconnect attempt to Elasticsearch host
[
https://issues.apache.org/jira/browse/FLINK-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269345#comment-15269345
]
Subhobrata Dey commented on FLINK-3857:
---
Hello [~fhueske],
I'm interested in the task & assigning it to myself.
> Add reconnect attempt to Elasticsearch host
> ---
>
> Key: FLINK-3857
> URL: https://issues.apache.org/jira/browse/FLINK-3857
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Affects Versions: 1.1.0, 1.0.2
>Reporter: Fabian Hueske
>
> Currently, the connection to the Elasticsearch host is opened in
> {{ElasticsearchSink.open()}}. In case the connection is lost (maybe due to a
> changed DNS entry), the sink fails.
> I propose to catch the Exception for lost connections in the {{invoke()}}
> method and try to re-open the connection for a configurable number of times
> with a certain delay.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-3857) Add reconnect attempt to Elasticsearch host
[
https://issues.apache.org/jira/browse/FLINK-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhobrata Dey reassigned FLINK-3857:
-
Assignee: Subhobrata Dey
> Add reconnect attempt to Elasticsearch host
> ---
>
> Key: FLINK-3857
> URL: https://issues.apache.org/jira/browse/FLINK-3857
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Affects Versions: 1.1.0, 1.0.2
>Reporter: Fabian Hueske
>Assignee: Subhobrata Dey
>
> Currently, the connection to the Elasticsearch host is opened in
> {{ElasticsearchSink.open()}}. In case the connection is lost (maybe due to a
> changed DNS entry), the sink fails.
> I propose to catch the Exception for lost connections in the {{invoke()}}
> method and try to re-open the connection for a configurable number of times
> with a certain delay.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer
[
https://issues.apache.org/jira/browse/FLINK-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182331#comment-15182331
]
Subhobrata Dey commented on FLINK-3387:
---
Hello [~StephanEwen],
Thanks for replying. I need some help.
Can you kindly help me in finding the class(as an example) where I should
create an instance of 'DataInputDeserializer' ?
I can proceed then. Thanks in advance.
> Replace all ByteArrayOutputStream and DataOutputStream use in State Backends
> by DataOutputSerializer
>
>
> Key: FLINK-3387
> URL: https://issues.apache.org/jira/browse/FLINK-3387
> Project: Flink
> Issue Type: Bug
> Components: state backends
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Subhobrata Dey
>
> Currently, the {{byte[]}} to object conversion and vice versa is always done
> by creating new byte array streams and Data Input/Output streams. That is
> pretty wasteful, we can optimize this with Flink's own classes:
> - {{DataInputSerializer}}
> - {{DataOutputSerializer}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer
[
https://issues.apache.org/jira/browse/FLINK-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156147#comment-15156147
]
Subhobrata Dey commented on FLINK-3387:
---
Hello [~StephanEwen],
I see that each class is creating an instance of `DataInputDeserializer`. So,
from what I understand, you want a static method like `getInstance` in class
`DataInputDeserializer` which will be be used by all the classes for calling
`deserializer.setBuffer(bytes, 0, bytes.length)`. Is this understanding
correct?
> Replace all ByteArrayOutputStream and DataOutputStream use in State Backends
> by DataOutputSerializer
>
>
> Key: FLINK-3387
> URL: https://issues.apache.org/jira/browse/FLINK-3387
> Project: Flink
> Issue Type: Bug
> Components: state backends
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Subhobrata Dey
>
> Currently, the {{byte[]}} to object conversion and vice versa is always done
> by creating new byte array streams and Data Input/Output streams. That is
> pretty wasteful, we can optimize this with Flink's own classes:
> - {{DataInputSerializer}}
> - {{DataOutputSerializer}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer
[
https://issues.apache.org/jira/browse/FLINK-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147597#comment-15147597
]
Subhobrata Dey commented on FLINK-3387:
---
Hello [~StephanEwen],
if nobody is working on this issue, I would like to work on it.
> Replace all ByteArrayOutputStream and DataOutputStream use in State Backends
> by DataOutputSerializer
>
>
> Key: FLINK-3387
> URL: https://issues.apache.org/jira/browse/FLINK-3387
> Project: Flink
> Issue Type: Bug
> Components: state backends
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>
> Currently, the {{byte[]}} to object conversion and vice versa is always done
> by creating new byte array streams and Data Input/Output streams. That is
> pretty wasteful, we can optimize this with Flink's own classes:
> - {{DataInputSerializer}}
> - {{DataOutputSerializer}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3035) Redis as State Backend
[ https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141013#comment-15141013 ] Subhobrata Dey commented on FLINK-3035: --- [~mjsax] [~aljoscha] I'll move this discussion to dev mailing list. Thanks for your review & comments. > Redis as State Backend > -- > > Key: FLINK-3035 > URL: https://issues.apache.org/jira/browse/FLINK-3035 > Project: Flink > Issue Type: New Feature > Components: Streaming >Reporter: Matthias J. Sax >Assignee: Subhobrata Dey >Priority: Minor > > Add Redis as a state backend for distributed snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3035) Redis as State Backend
[ https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140239#comment-15140239 ] Subhobrata Dey commented on FLINK-3035: --- Hello [~mjsax] Thanks for replying. I adopted & created a PR out of my implementation. Would love to know your views. > Redis as State Backend > -- > > Key: FLINK-3035 > URL: https://issues.apache.org/jira/browse/FLINK-3035 > Project: Flink > Issue Type: New Feature > Components: Streaming >Reporter: Matthias J. Sax >Assignee: Subhobrata Dey >Priority: Minor > > Add Redis as a state backend for distributed snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3035) Redis as State Backend
[ https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136075#comment-15136075 ] Subhobrata Dey commented on FLINK-3035: --- Hello [~mjsax], I had a look into the rocksdb state backend implementation & found that rocksdb is used in embedded mode. However, there are not many popular libraries which allow Redis to be used in embedded mode. One such project I found is: https://github.com/kstyrc/embedded-redis So, would you suggest to use the above project for using redis in embedded mode or can we start redis server separately (externally)? Looking forward to your suggestions. > Redis as State Backend > -- > > Key: FLINK-3035 > URL: https://issues.apache.org/jira/browse/FLINK-3035 > Project: Flink > Issue Type: New Feature > Components: Streaming >Reporter: Matthias J. Sax >Assignee: Subhobrata Dey >Priority: Minor > > Add Redis as a state backend for distributed snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2678) DataSet API does not support multi-dimensional arrays as keys
[ https://issues.apache.org/jira/browse/FLINK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131661#comment-15131661 ] Subhobrata Dey commented on FLINK-2678: --- [~till.rohrmann] Please help in reviewing the new commit. > DataSet API does not support multi-dimensional arrays as keys > - > > Key: FLINK-2678 > URL: https://issues.apache.org/jira/browse/FLINK-2678 > Project: Flink > Issue Type: Wish > Components: DataSet API >Reporter: Till Rohrmann >Assignee: Subhobrata Dey >Priority: Minor > > The DataSet API does not support grouping/sorting on field which are > multi-dimensional arrays. It could be helpful to also support these types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-3035) Redis as State Backend
[ https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subhobrata Dey reassigned FLINK-3035: - Assignee: Subhobrata Dey > Redis as State Backend > -- > > Key: FLINK-3035 > URL: https://issues.apache.org/jira/browse/FLINK-3035 > Project: Flink > Issue Type: New Feature > Components: Streaming >Reporter: Matthias J. Sax >Assignee: Subhobrata Dey >Priority: Minor > > Add Redis as a state backend for distributed snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3035) Redis as State Backend
[ https://issues.apache.org/jira/browse/FLINK-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128367#comment-15128367 ] Subhobrata Dey commented on FLINK-3035: --- Hello [~mjsax], Yes, I'm interested in the issue & would assign myself to the ticket. Thanks. > Redis as State Backend > -- > > Key: FLINK-3035 > URL: https://issues.apache.org/jira/browse/FLINK-3035 > Project: Flink > Issue Type: New Feature > Components: Streaming >Reporter: Matthias J. Sax >Priority: Minor > > Add Redis as a state backend for distributed snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3276) Move runtime parts of flink-streaming-java to flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126524#comment-15126524
]
Subhobrata Dey commented on FLINK-3276:
---
Hello Maximilian,
If nobody else is working on this issue, I would like to take this issue up.
Kindly let me know.
> Move runtime parts of flink-streaming-java to flink-runtime
> ---
>
> Key: FLINK-3276
> URL: https://issues.apache.org/jira/browse/FLINK-3276
> Project: Flink
> Issue Type: Bug
> Components: Java API, Streaming
>Reporter: Maximilian Michels
>Priority: Critical
> Fix For: 1.0.0
>
>
> All runtime parts of {{flink-streaming-java}} should be moved to
> {{flink-runtime}}.
> Most difficult part of these changes I see in the JobGraph generation /
> Execution parts. We need an interface like the {{PlanExecutor}} in
> {{flink-java}} to delegate the runtime logic of streaming jobs to
> {{flink-runtime}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2678) DataSet API does not support multi-dimensional arrays as keys
[ https://issues.apache.org/jira/browse/FLINK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122486#comment-15122486 ] Subhobrata Dey commented on FLINK-2678: --- Hello Till, I was checking the implementation of PrimitiveArrayTypeInfo.java ( which is the TypeInformation implementation of 1-d array ) The tasks mention about multidimensional arrays. By multidimensional do you intend to restrict the solution to a particular dimension like 2-d or 3-d? or do you have some elegant solution to make the dimension dynamic? Kindly let me know because I plan to implement the solution. > DataSet API does not support multi-dimensional arrays as keys > - > > Key: FLINK-2678 > URL: https://issues.apache.org/jira/browse/FLINK-2678 > Project: Flink > Issue Type: Wish > Components: DataSet API >Reporter: Till Rohrmann >Assignee: Subhobrata Dey >Priority: Minor > > The DataSet API does not support grouping/sorting on field which are > multi-dimensional arrays. It could be helpful to also support these types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3033) Redis Source Connector
[ https://issues.apache.org/jira/browse/FLINK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122563#comment-15122563 ] Subhobrata Dey commented on FLINK-3033: --- Hello, If nobody is working on this issue, I would like to work on this issue. Thanks & regards, Subhobrata > Redis Source Connector > -- > > Key: FLINK-3033 > URL: https://issues.apache.org/jira/browse/FLINK-3033 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Matthias J. Sax >Priority: Minor > > Flink does not provide a source connector for Redis. > See FLINK-3034 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3034) Redis SInk Connector
[ https://issues.apache.org/jira/browse/FLINK-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122562#comment-15122562 ] Subhobrata Dey commented on FLINK-3034: --- Hello, if nobody is working on this issue, I would like to work on it. Thanks & regards, Subhobrata > Redis SInk Connector > > > Key: FLINK-3034 > URL: https://issues.apache.org/jira/browse/FLINK-3034 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Matthias J. Sax >Priority: Minor > > Flink does not provide a sink connector for Redis. > See FLINK-3033 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2678) DataSet API does not support multi-dimensional arrays as keys
[ https://issues.apache.org/jira/browse/FLINK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121366#comment-15121366 ] Subhobrata Dey commented on FLINK-2678: --- Hello Till, If no one is working on this issue, can I take up this issue to contribute? Kindly let me know. Thanks & regards, Subhobrata > DataSet API does not support multi-dimensional arrays as keys > - > > Key: FLINK-2678 > URL: https://issues.apache.org/jira/browse/FLINK-2678 > Project: Flink > Issue Type: Wish > Components: DataSet API >Reporter: Till Rohrmann >Priority: Minor > > The DataSet API does not support grouping/sorting on field which are > multi-dimensional arrays. It could be helpful to also support these types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2678) DataSet API does not support multi-dimensional arrays as keys
[ https://issues.apache.org/jira/browse/FLINK-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121516#comment-15121516 ] Subhobrata Dey commented on FLINK-2678: --- Hello Till, Please assign me the issue. I would like to work on it. > DataSet API does not support multi-dimensional arrays as keys > - > > Key: FLINK-2678 > URL: https://issues.apache.org/jira/browse/FLINK-2678 > Project: Flink > Issue Type: Wish > Components: DataSet API >Reporter: Till Rohrmann >Priority: Minor > > The DataSet API does not support grouping/sorting on field which are > multi-dimensional arrays. It could be helpful to also support these types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2985) Allow different field names for unionAll() in Table API
[ https://issues.apache.org/jira/browse/FLINK-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120520#comment-15120520 ] Subhobrata Dey commented on FLINK-2985: --- Hello, I want to work on this issue. Can somebody kindly assign me this issue? Thanks & regards, Subhobrata > Allow different field names for unionAll() in Table API > --- > > Key: FLINK-2985 > URL: https://issues.apache.org/jira/browse/FLINK-2985 > Project: Flink > Issue Type: Improvement > Components: Table API >Reporter: Timo Walther >Priority: Minor > > The recently merged `unionAll` operator checks if the field names of the left > and right side are equal. Actually, this is not necessary. The union operator > in SQL checks only the types and uses the names of left side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3292) Bug in flink-jdbc. Not all JDBC drivers supported
Subhobrata Dey created FLINK-3292: - Summary: Bug in flink-jdbc. Not all JDBC drivers supported Key: FLINK-3292 URL: https://issues.apache.org/jira/browse/FLINK-3292 Project: Flink Issue Type: Bug Components: other Affects Versions: 1.0.0 Reporter: Subhobrata Dey Priority: Minor Fix For: 1.0.0 Hello, In method open in JDBCInputFormat.java, while using dbConn.createStatement, the resultSetType & resultSetConcurrency are hardcoded. These two fields may vary with different JDBC drivers & hence it fails in a few cases like SAP HANA Jdbc driver. There are two variants of the method dbCon.createStatement, one with parameters & the other without parameters. Both should be supported. Thanks & regards, Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
