[jira] [Created] (FLINK-4609) Remove redundant check for null in CrossOperator

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4609:
--

 Summary: Remove redundant check for null in CrossOperator
 Key: FLINK-4609
 URL: https://issues.apache.org/jira/browse/FLINK-4609
 Project: Flink
  Issue Type: Bug
  Components: Java API
Affects Versions: 1.1.2
Reporter: Alexander Pivovarov
Priority: Trivial


CrossOperator checks input1 and input2 for null after they were dereferenced



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4608) Use short-circuit AND in Max/Min AggregationFunction

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4608:
--

 Summary: Use short-circuit AND in Max/Min AggregationFunction
 Key: FLINK-4608
 URL: https://issues.apache.org/jira/browse/FLINK-4608
 Project: Flink
  Issue Type: Bug
  Components: Java API
Affects Versions: 1.1.2
Reporter: Alexander Pivovarov
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4607) Close FileInputStream in ParameterTool and other

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4607:
--

 Summary: Close FileInputStream in ParameterTool and other
 Key: FLINK-4607
 URL: https://issues.apache.org/jira/browse/FLINK-4607
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.1.2
Reporter: Alexander Pivovarov
Priority: Trivial


ParameterTool and some tests do not close FileInputStream
{code}
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java
flink-java/src/test/java/org/apache/flink/api/java/utils/ParameterToolTest.java
flink-java8/src/test/java/org/apache/flink/runtime/util/JarFileCreatorLambdaTest.java
flink-runtime/src/test/java/org/apache/flink/runtime/util/JarFileCreatorTest.java
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4606) Integrate the new ResourceManager with the existing FlinkResourceManager

[zhangjing (JIRA)]

zhangjing created FLINK-4606:


 Summary: Integrate the new ResourceManager with the existing 
FlinkResourceManager
 Key: FLINK-4606
 URL: https://issues.apache.org/jira/browse/FLINK-4606
 Project: Flink
  Issue Type: Sub-task
  Components: Cluster Management
Reporter: zhangjing
Assignee: zhangjing


Integrate the new ResourceManager with the existing FlinkResourceManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Implementing FLIP-2 and FLIP-4

[AJ Heller]

Thank you for offering your support, I'm excited to dig in!

I have some work to do getting up to speed on the windowing internals. And
I still need to get my bearing on the Evictor changes, I plan to read
through the list archive and documents today. Vishnu, are your changes
already publicly viewable?

Regarding the window modifications in FLIP-2, I see Vishnu that you've
suggested an interface for the EvictorContext object, and Aljoscha, you
suggested an abstract Context class. Does it make sense for them to agree?
The other big difference I've seen in the signatures is wheter the Window
is contained in the context or not.

Have you considered modifying the signature of the methods to accept `` or ``? At least in terms of
FLIP-2, this would allow each process window function to define and work
with its own context (without downcasting, anyway), and similarly in the
future, there'd be less work in changing Context subclasses when new
abstract methods are added to Context.

But I may be getting ahead of myself. Could you point me towards where
contexts are/would be created? I'm not clear on the ownership and lifecycle
of these objects yet.

[jira] [Created] (FLINK-4605) Add an expression that returns the return type of an expression

[Timo Walther (JIRA)]

Timo Walther created FLINK-4605:
---

 Summary: Add an expression that returns the return type of an 
expression
 Key: FLINK-4605
 URL: https://issues.apache.org/jira/browse/FLINK-4605
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Timo Walther


Esp. for Java users of the Table API it is hard to obtain the return type of an 
expression. I propose to implement an expression that returns the type of an 
input expression as a string.

{{myLong.getType()}} could call the toString method of TypeInformation.

This could also be useful to distinguish between different subtypes of POJOs 
etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flink HA on AWS: Network related issue

[Deepak Jha]

Hi Till,
I'm getting following message in Jobmanager log

2016-09-09 07:46:55,093 PDT [WARN]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-985] akka.remote.RemoteWatcher - *Detected
unreachable: [akka.tcp://flink@10.8.4.57:6121
]*
2016-09-09 07:46:55,094 PDT [INFO]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-985]
o.a.f.runtime.jobmanager.JobManager - Task manager akka.tcp://
flink@10.8.4.57:6121/user/taskmanager terminated.
2016-09-09 07:46:55,094 PDT [INFO]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-985] o.a.f.r.instance.InstanceManager
- Unregistered task manager akka.tcp://flink@10.8.4.57:6121/user/taskmanager.
Number of registered task managers 2. Number of available slots 4.
2016-09-09 07:46:55,096 PDT [WARN]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-982] Remoting - Association to
[akka.tcp://flink@10.8.4.57:6121] having UID [-1223410403] is irrecoverably
failed. *UID is now quarantined and all messages to this UID will be
delivered to dead letters. Remote actorsystem must be restarted to recover
from this situation.*
2016-09-09 07:46:55,097 PDT [INFO]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-982] akka.actor.LocalActorRef -
Message [akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://flink/deadLetters] to
Actor[akka://flink/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fflink%4010.8.4.57%3A6121-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fflink%4010.8.4.57%3A6121-0#393939009]
was not delivered. [54] dead letters encountered. This logging can be
turned off or adjusted with configuration settings 'akka.log-dead-letters'
and 'akka.log-dead-letters-during-shutdown'.
2016-09-09 07:46:55,098 PDT [INFO]  ip-10-8-11-249
[flink-akka.actor.default-dispatcher-985] akka.actor.LocalActorRef -
Message [akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://flink/deadLetters] to
Actor[akka://flink/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fflink%4010.8.4.57%3A51291-2#1151730456]
was not delivered. [55] dead letters encountered. This logging can be
turned off or adjusted with configuration settings 'akka.log-dead-letters'
and 'akka.log-dead-letters-during-shutdown'.
2016-09-09 07:46:58,479 PDT [INFO]  ip-10-8-11-249
[ForkJoinPool-3-worker-1] o.a.f.r.c.ZooKeeperCompletedCheckpointStore -
Recovering checkpoints from ZooKeeper.

Hope it helps. I'm using Flink 1.0.2

On Fri, Sep 9, 2016 at 12:34 AM, Till Rohrmann  wrote:

> Hi Deepak,
>
> could you check the logs whether the JobManager has been quarantined and
> thus, cannot be connected to anymore? The logs should at least contain a
> hint why the TaskManager lost the connection initially.
>
> Cheers,
> Till
>
> On Thu, Sep 8, 2016 at 7:08 PM, Deepak Jha  wrote:
>
> > Hi,
> > I've setup Flink HA on AWS ( 3 Taskmanagers and 2 Jobmanagers each are on
> > EC2 m4.large instance with checkpoint enabled on S3 ). My topology works
> > fine, but after few hours I do see that Taskmanagers gets detached with
> > Jobmanager. I tried to reach Jobmanager using telnet at the same time and
> > it worked but Taskmanager does not succeed in connecting again. It
> attaches
> > only after I restart it. I tried following settings but still the problem
> > persists.
> >
> > akka.ask.timeout: 20 s
> > akka.lookup.timeout: 20 s
> > akka.watch.heartbeat.interval: 20 s
> >
> > Please find attached snapshot on one of the Taskmanager. Is there any
> > setting that I need to do ?
> >
> > --
> > Thanks,
> > Deepak Jha
> >
> >
>



-- 
Thanks,
Deepak Jha

Contribution

[Hasan Gürcan]

Hi devs,

i contributed to Stratosphere as I was studying computer science at the FU 
Berlin. Currently i am working as data engineer and i am very interested in 
contributing to your project.
Do someone have an idea for getting started in contributing to Flink like a 
certain jira task?

Best Regards
Hasan Gürcan

Re: Contribution

[Hasan Gürcan]

Hi,

Thanks Gábor i will search for some tasks with this label.
@Till: Thanks too :). Tendencially to backend.

Best Regards
Hasan

> Am 09.09.2016 um 14:17 schrieb Till Rohrmann :
> 
> Welcome back Hasan :-)
> 
> What are you interested in (e.g. libraries, backend, frontend)?
> 
> Cheers,
> Till
> 
> On Fri, Sep 9, 2016 at 2:02 PM, Gábor Gévay  wrote:
> 
>> Hi Hasan,
>> 
>> Welcome!
>> There is the "starter" label on some Jiras, which means that the issue
>> is good for getting started.
>> 
>> Best,
>> Gábor
>> 
>> 
>> 
>> 
>> 2016-09-09 13:46 GMT+02:00 Hasan Gürcan
>> :
>>> Hi devs,
>>> 
>>> i contributed to Stratosphere as I was studying computer science at the
>> FU Berlin. Currently i am working as data engineer and i am very interested
>> in contributing to your project.
>>> Do someone have an idea for getting started in contributing to Flink
>> like a certain jira task?
>>> 
>>> Best Regards
>>> Hasan Gürcan
>> 

Re: Contribution

[Till Rohrmann]

Welcome back Hasan :-)

What are you interested in (e.g. libraries, backend, frontend)?

Cheers,
Till

On Fri, Sep 9, 2016 at 2:02 PM, Gábor Gévay  wrote:

> Hi Hasan,
>
> Welcome!
> There is the "starter" label on some Jiras, which means that the issue
> is good for getting started.
>
> Best,
> Gábor
>
>
>
>
> 2016-09-09 13:46 GMT+02:00 Hasan Gürcan
> :
> > Hi devs,
> >
> > i contributed to Stratosphere as I was studying computer science at the
> FU Berlin. Currently i am working as data engineer and i am very interested
> in contributing to your project.
> > Do someone have an idea for getting started in contributing to Flink
> like a certain jira task?
> >
> > Best Regards
> > Hasan Gürcan
>

Re: Contribution

[Gábor Gévay]

Hi Hasan,

Welcome!
There is the "starter" label on some Jiras, which means that the issue
is good for getting started.

Best,
Gábor




2016-09-09 13:46 GMT+02:00 Hasan Gürcan
:
> Hi devs,
>
> i contributed to Stratosphere as I was studying computer science at the FU 
> Berlin. Currently i am working as data engineer and i am very interested in 
> contributing to your project.
> Do someone have an idea for getting started in contributing to Flink like a 
> certain jira task?
>
> Best Regards
> Hasan Gürcan

[jira] [Created] (FLINK-4604) Add support for standard deviation/variance

[Timo Walther (JIRA)]

Timo Walther created FLINK-4604:
---

 Summary: Add support for standard deviation/variance
 Key: FLINK-4604
 URL: https://issues.apache.org/jira/browse/FLINK-4604
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Timo Walther


Calcite's {{AggregateReduceFunctionsRule}} can convert SQL {{AVG, STDDEV_POP, 
STDDEV_SAMP, VAR_POP, VAR_SAMP}} to sum/count functions. We should add, test 
and document this rule. 

If we also want to add this aggregates to Table API is up for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Contribution

[Hasan Gürcan]

Hi devs,

i contributed to Stratosphere as I was studying computer science at the FU 
Berlin. Currently i am working as data engineer and i am very interested in 
contributing to your project.
Do someone have an idea for getting started in contributing to Flink like a 
certain jira task?

Best Regards
Hasan Gürcan

[jira] [Created] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-4603:


 Summary: KeyedStateBackend cannot restore user code classes
 Key: FLINK-4603
 URL: https://issues.apache.org/jira/browse/FLINK-4603
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.2.0
Reporter: Till Rohrmann
Priority: Blocker
 Fix For: 1.2.0


A user reported that he cannot restore keyed state which contains user code 
classes. I suspect that we don't use the user code class loader to deserialize 
the state.

The solution seems to be to forward the user code class loader to the 
{{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flink HA on AWS: Network related issue

[Till Rohrmann]

Hi Deepak,

could you check the logs whether the JobManager has been quarantined and
thus, cannot be connected to anymore? The logs should at least contain a
hint why the TaskManager lost the connection initially.

Cheers,
Till

On Thu, Sep 8, 2016 at 7:08 PM, Deepak Jha  wrote:

> Hi,
> I've setup Flink HA on AWS ( 3 Taskmanagers and 2 Jobmanagers each are on
> EC2 m4.large instance with checkpoint enabled on S3 ). My topology works
> fine, but after few hours I do see that Taskmanagers gets detached with
> Jobmanager. I tried to reach Jobmanager using telnet at the same time and
> it worked but Taskmanager does not succeed in connecting again. It attaches
> only after I restart it. I tried following settings but still the problem
> persists.
>
> akka.ask.timeout: 20 s
> akka.lookup.timeout: 20 s
> akka.watch.heartbeat.interval: 20 s
>
> Please find attached snapshot on one of the Taskmanager. Is there any
> setting that I need to do ?
>
> --
> Thanks,
> Deepak Jha
>
>

[jira] [Created] (FLINK-4601) Check for empty string properly

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4601:
--

 Summary: Check for empty string properly
 Key: FLINK-4601
 URL: https://issues.apache.org/jira/browse/FLINK-4601
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.1.2
Reporter: Alexander Pivovarov
Priority: Trivial


UdfAnalyzerExamplesTest.java and UdfAnalyzerTest.java use == to check for empty 
string. We should use isEmpty() instead



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)