[jira] [Comment Edited] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-12 Thread Subhankar Biswas (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326810#comment-15326810
 ] 

Subhankar Biswas edited comment on FLINK-4051 at 6/13/16 4:43 AM:
--

nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
   lock.lock();
   try {
 E x;
 while ( (x = unlinkFirst()) == null)
 notEmpty.await();
 return x;
} finally {
lock.unlock();
}
{code}
{code:java}
//Code of Poll
 lock.lockInterruptibly();
 try {
   E x;
   while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
   }
 return x;
   } finally {
lock.unlock();
   }
{code}
So if the queue is empty the thread will be await mode. IMO a good 
implementation is to use the poll(timeout, unit).


was (Author: neo20iitkgp):
nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
   lock.lock();
   try {
 E x;
 while ( (x = unlinkFirst()) == null)
 notEmpty.await();
 return x;
} finally {
lock.unlock();
}
{code}
{code:java}
//Code of Poll
 lock.lockInterruptibly();
 try {
   E x;
   while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
   }
 return x;
   } finally {
lock.unlock();
   }
{code}
So if the queue is empty the thread will be await mode.

> RabbitMQ Source might not react to cancel signal
> 
>
> Key: FLINK-4051
> URL: https://issues.apache.org/jira/browse/FLINK-4051
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>
> As reported here 
> https://issues.apache.org/jira/browse/FLINK-3763?focusedCommentId=15322517=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322517,
>  the RabbitMQ source might block forever / ignore the cancelling signal, if 
> its listening to an empty queue.
> Fix: call nextDelivery() with a timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-12 Thread Subhankar Biswas (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326810#comment-15326810
 ] 

Subhankar Biswas edited comment on FLINK-4051 at 6/13/16 4:42 AM:
--

nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
   lock.lock();
   try {
 E x;
 while ( (x = unlinkFirst()) == null)
 notEmpty.await();
 return x;
} finally {
lock.unlock();
}
{code}
{code:java}
//Code of Poll
 lock.lockInterruptibly();
 try {
   E x;
   while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
   }
 return x;
   } finally {
lock.unlock();
   }
{code}
So if the queue is empty the thread will be await mode.


was (Author: neo20iitkgp):
nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.

> RabbitMQ Source might not react to cancel signal
> 
>
> Key: FLINK-4051
> URL: https://issues.apache.org/jira/browse/FLINK-4051
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>
> As reported here 
> https://issues.apache.org/jira/browse/FLINK-3763?focusedCommentId=15322517=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322517,
>  the RabbitMQ source might block forever / ignore the cancelling signal, if 
> its listening to an empty queue.
> Fix: call nextDelivery() with a timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-12 Thread Subhankar Biswas (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326810#comment-15326810
 ] 

Subhankar Biswas edited comment on FLINK-4051 at 6/13/16 4:31 AM:
--

nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.


was (Author: neo20iitkgp):
 {code:java}nextDelivery(){code} use take() method of BlockingQueue, while 
nextDelivery(timeout) use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.

> RabbitMQ Source might not react to cancel signal
> 
>
> Key: FLINK-4051
> URL: https://issues.apache.org/jira/browse/FLINK-4051
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>
> As reported here 
> https://issues.apache.org/jira/browse/FLINK-3763?focusedCommentId=15322517=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322517,
>  the RabbitMQ source might block forever / ignore the cancelling signal, if 
> its listening to an empty queue.
> Fix: call nextDelivery() with a timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-12 Thread Subhankar Biswas (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326810#comment-15326810
 ] 

Subhankar Biswas edited comment on FLINK-4051 at 6/13/16 4:30 AM:
--

 {code:java}nextDelivery(){code} use take() method of BlockingQueue, while 
nextDelivery(timeout) use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.


was (Author: neo20iitkgp):
 nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.

> RabbitMQ Source might not react to cancel signal
> 
>
> Key: FLINK-4051
> URL: https://issues.apache.org/jira/browse/FLINK-4051
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>
> As reported here 
> https://issues.apache.org/jira/browse/FLINK-3763?focusedCommentId=15322517=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322517,
>  the RabbitMQ source might block forever / ignore the cancelling signal, if 
> its listening to an empty queue.
> Fix: call nextDelivery() with a timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4051) RabbitMQ Source might not react to cancel signal

2016-06-12 Thread Subhankar Biswas (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326810#comment-15326810
 ] 

Subhankar Biswas commented on FLINK-4051:
-

 nextDelivery() use take() method of BlockingQueue, while nextDelivery(timeout) 
use poll() method. 
{code:java}
//Code of take
public E takeFirst() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lock();
try {
E x;
while ( (x = unlinkFirst()) == null)
notEmpty.await();
return x;
} finally {
lock.unlock();
}
}
{code}
{code:java}
//Code of Poll 
public E pollFirst(long timeout, TimeUnit unit)
throws InterruptedException {
long nanos = unit.toNanos(timeout);
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
E x;
while ( (x = unlinkFirst()) == null) {
if (nanos <= 0)
return null;
nanos = notEmpty.awaitNanos(nanos);
}
return x;
} finally {
lock.unlock();
}
}
{code}
So if the queue is empty the thread will be await mode.

> RabbitMQ Source might not react to cancel signal
> 
>
> Key: FLINK-4051
> URL: https://issues.apache.org/jira/browse/FLINK-4051
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Subhankar Biswas
>
> As reported here 
> https://issues.apache.org/jira/browse/FLINK-3763?focusedCommentId=15322517=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322517,
>  the RabbitMQ source might block forever / ignore the cancelling signal, if 
> its listening to an empty queue.
> Fix: call nextDelivery() with a timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3801) Upgrade Joda-Time library to 2.9.3

2016-06-12 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLINK-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-3801:
--
Description: 
Currently yoda-time 2.5 is used which was very old.


We should upgrade to 2.9.3

  was:
Currently yoda-time 2.5 is used which was very old.

We should upgrade to 2.9.3


> Upgrade Joda-Time library to 2.9.3
> --
>
> Key: FLINK-3801
> URL: https://issues.apache.org/jira/browse/FLINK-3801
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Minor
>
> Currently yoda-time 2.5 is used which was very old.
> We should upgrade to 2.9.3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3702) DataStream API PojoFieldAccessor doesn't support nested POJOs

2016-06-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326578#comment-15326578
 ] 

ASF GitHub Bot commented on FLINK-3702:
---

GitHub user ggevay opened a pull request:

https://github.com/apache/flink/pull/2094

[FLINK-3702] Make FieldAccessors support nested field expressions.

I finally had some time to complete this, sorry it took so long.

I have added `getFieldAccessor` to `TypeInformation`, which creates a 
`FieldAccessor` from a position or a (possibly nested) field expression. It 
uses recursion in the nested case, and also supports the heterogeneous case, 
e.g. pojo inside tuple inside pojo.

Additionally, I have noticed that the code to serialize/deserialize a 
`java.lang.reflect.Field` is duplicated at several places 
(`readObject`/`writeObject` of `PojoSerializer`, `PojoComparator`, 
`PojoField`). I also needed it in `PojoFieldAccessor`, so instead of adding 
more duplication, I factored it out into a new class (`FieldSerializer`).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ggevay/flink FieldAccessorRefactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2094


commit e8b584791144a2e9a22e39242010726fc76ae2a3
Author: Gabor Gevay 
Date:   2016-05-22T17:48:50Z

[FLINK-3702] Make FieldAccessors support nested field expressions.




> DataStream API PojoFieldAccessor doesn't support nested POJOs
> -
>
> Key: FLINK-3702
> URL: https://issues.apache.org/jira/browse/FLINK-3702
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.0.0
>Reporter: Robert Metzger
>Assignee: Gabor Gevay
>
> The {{PojoFieldAccessor}} (which is used by {{.sum(String)}} and similar 
> methods) doesn't support nested POJOs right now.
> As part of FLINK-3697 I'll add a check for a nested POJO and fail with an 
> exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2094: [FLINK-3702] Make FieldAccessors support nested fi...

2016-06-12 Thread ggevay 
GitHub user ggevay opened a pull request:

https://github.com/apache/flink/pull/2094

[FLINK-3702] Make FieldAccessors support nested field expressions.

I finally had some time to complete this, sorry it took so long.

I have added `getFieldAccessor` to `TypeInformation`, which creates a 
`FieldAccessor` from a position or a (possibly nested) field expression. It 
uses recursion in the nested case, and also supports the heterogeneous case, 
e.g. pojo inside tuple inside pojo.

Additionally, I have noticed that the code to serialize/deserialize a 
`java.lang.reflect.Field` is duplicated at several places 
(`readObject`/`writeObject` of `PojoSerializer`, `PojoComparator`, 
`PojoField`). I also needed it in `PojoFieldAccessor`, so instead of adding 
more duplication, I factored it out into a new class (`FieldSerializer`).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ggevay/flink FieldAccessorRefactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2094


commit e8b584791144a2e9a22e39242010726fc76ae2a3
Author: Gabor Gevay 
Date:   2016-05-22T17:48:50Z

[FLINK-3702] Make FieldAccessors support nested field expressions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

2016-06-12 Thread Ivan Mushketyk (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326546#comment-15326546
 ] 

Ivan Mushketyk edited comment on FLINK-3677 at 6/12/16 5:26 PM:


Could you please assign this issue to me?


was (Author: ivan.mushketyk):
Could you please assign this to me?

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Priority: Minor
>  Labels: starter
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns

2016-06-12 Thread Ivan Mushketyk (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326546#comment-15326546
 ] 

Ivan Mushketyk commented on FLINK-3677:
---

Could you please assign this to me?

> FileInputFormat: Allow to specify include/exclude file name patterns
> 
>
> Key: FLINK-3677
> URL: https://issues.apache.org/jira/browse/FLINK-3677
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.0
>Reporter: Maximilian Michels
>Priority: Minor
>  Labels: starter
>
> It would be nice to be able to specify a regular expression to filter files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3839) Support wildcards in classpath parameters

2016-06-12 Thread Ken Krugler (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326513#comment-15326513
 ] 

Ken Krugler commented on FLINK-3839:


Hi Robert,

>From my email on this...

bq. But this doesn’t seem to work. I believe it’s because JDK tools do the 
expansion before creating the URLs used by the classloader, but Flink code 
doesn’t do any such special processing, and just creates URLs - see 
ProgramOptions.java, via  classpaths.add(new URL(path));

So take a look at ProgramOptions.java as a place to start.

> Support wildcards in classpath parameters
> -
>
> Key: FLINK-3839
> URL: https://issues.apache.org/jira/browse/FLINK-3839
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ken Krugler
>Assignee: Robert Thorman
>Priority: Minor
>
> Currently you can only specify a single explict jar with the CLI --classpath 
> file:// parameter.Java (since 1.6) has allowed you to use -cp 
> /* as a way of adding every file that ends in .jar in a 
> directory.
> This would simplify things, e.g. when running on EMR you have to add roughly 
> 120 jars explicitly, but these are all located in just two directories.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3311) Add a connector for streaming data into Cassandra

2016-06-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326399#comment-15326399
 ] 

ASF GitHub Bot commented on FLINK-3311:
---

Github user theomega commented on the issue:

https://github.com/apache/flink/pull/1771
  
I tried out this branch and it works like it should in the scenario I set 
up:
I wrote (so only using the Sink) to a quiet complex columnfamily in a 8 
node cassandra cluster. I was using a complex setup of windowed streams and all 
the data appeared and was perfectly readable as expected. I also had multiple 
sinks at the same time which also worked perfectly. I could not test the 
scenario @rmetzger is mentioning.

Overall, I agree with @rmetzger that it should be considered to merge this 
to get more users to test it and report issues. 


> Add a connector for streaming data into Cassandra
> -
>
> Key: FLINK-3311
> URL: https://issues.apache.org/jira/browse/FLINK-3311
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Andrea Sella
>
> We had users in the past asking for a Flink+Cassandra integration.
> It seems that there is a well-developed java client for connecting into 
> Cassandra: https://github.com/datastax/java-driver (ASL 2.0)
> There are also tutorials out there on how to start a local cassandra instance 
> (for the tests): 
> http://prettyprint.me/prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/index.html
> For the data types, I think we should support TupleX types, and map standard 
> java types to the respective cassandra types.
> In addition, it seems that there is a object mapper from datastax to store 
> POJOs in Cassandra (there are annotations for defining the primary key and 
> types)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1771: [FLINK-3311/FLINK-3332] Add Cassandra connector

2016-06-12 Thread theomega 
Github user theomega commented on the issue:

https://github.com/apache/flink/pull/1771
  
I tried out this branch and it works like it should in the scenario I set 
up:
I wrote (so only using the Sink) to a quiet complex columnfamily in a 8 
node cassandra cluster. I was using a complex setup of windowed streams and all 
the data appeared and was perfectly readable as expected. I also had multiple 
sinks at the same time which also worked perfectly. I could not test the 
scenario @rmetzger is mentioning.

Overall, I agree with @rmetzger that it should be considered to merge this 
to get more users to test it and report issues. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---