date:20161011

This is one of my challenges too-

1. The JavaScript rules are only applicable inside one operator (next,
followedBy, notFollowedBy). And the JavaScript rules can apply to only the
event in that operator. I make it a little more dynamic by creating a Rules
HashMap and add rules with the names "first", "next", "followedBy1" (same
as the ones I use for the pattern operator). This way the rules attached to
a particular operator can be easily changed via a connected stream.

I think the feature where other events in the pattern are accessible is
being added. Currently you can only look inside one event in the "where"
clause attached to a CEP pattern operator. For example, if I check two
consecutive credit card events for a user to calculate the straight line
distance between then to divide by the difference in time I cannot do that
unless I fire the pattern for every pair and check this condition in the
PatternStream's select operator where all the events are accessible.

2. The second problem I have is that I cannot change the rules applied to
the pattern stream. For example if I have next.followedBy and I want to add
another followedBy it is a compile time change. The JavaScript engine helps
me with the first issue but this one just needs a recompile unless you have
another Flink Pipeline deployed which can check against that pattern as
well. I am guess at this point you will need to take a SavePoint, stop your
pipeline, redeploy this new pipeline(with a new pattern configuration) and
start again.

I would like to know if there is a cleaner solution but the above is my
fallback.

Sameer

On Tue, Oct 11, 2016 at 5:51 PM,  wrote:

> Hi Sameer,
>
> I just replied to the earlier post, but I will copy it here:
>
> We also have the same requirement - we want to allow the user to change
> the matching patterns and have them take effect immediately. I'm wondering
> whether the proposed trigger DSL takes us one step closer:(I don't think it
> solves the problem) or we have to dynamically generate Flink job JAR files
> when the matching rules/patterns are changed and submit them to Flink.
>
> I had thought about using a similar approach, but it is quite restrictive
> because you cannot use the power for Flink CEP engine with this approach.
> For example, I want to be able to use followedBy, next, notFollowedBy (in
> future) operators to detect the patterns and these matching patterns need
> to be user-cofigurable/dynamic/hot deployable. The simple attribute-based
> rules/patterns that you specified can be made dynamic as you mentioned but
> the rules/patterns that use not just the current event attributes, but also
> past events (e.g. followedBy) are much harder to make them dynamic without
> some help from Flink that implements the CEP operators.
>
> - LF
>
>
>
>
> --
> *From:* Sameer W 
> *To:* "user@flink.apache.org" 
> *Sent:* Tuesday, October 11, 2016 2:23 PM
> *Subject:* Re: What is the best way to load/add patterns dynamically (at
> runtime) with Flink?
>
> I have used a JavaScript engine in my CEP to evaluate my patterns. Each
> event is a list of named attributes (HashMap like). And event is attached
> to a list of rules expressed as JavaScript code (See example below with one
> rule but I can match as many rules).  The rules are distributed over a
> connected stream which allow it to change over time. This is how I do it to
> keep my rules dynamic. If someone has a better way I would love to hear it
> as well.
>
>
> private transient ScriptEngineManager factory = new ScriptEngineManager();
> private transient ScriptEngine engine = factory.getEngineByName("
> JavaScript");
> /*Inside open*/
> factory = new ScriptEngineManager();
>
> /*Close open*/
>
> /*Inside my operator*/
> engine = factory.getEngineByName("JavaScript");
> engine.put("evt", value.f1); //value.f1 contains a JSON version of my
> HashMap of attributes
> engine.eval(value.f2.rule); //f2.last contains the rule which is evaluated
> by the JavaScript Engine
> /*
> Sample JavaScript contained in the call - engine.eval(value.f2.rule); is
> shown below (not the "evt" variable in the JavaScript and the the preceding
> line - engine.put("evt", value.f1);
>
> *var evt=JSON.parse(evt);var result = evt.temperature>50 &&
> evt.pressure<900*
> */
> boolean ret = (boolean)engine.get("result");
>
> if(ret) /*Rule is Matched*/
>
>
>
> > On Oct 11, 2016, at 5:01 PM, PedroMrChaves 
> wrote:
> >
> > Hello,
> >
> > I am new to Apache Flink and am trying to build a CEP using Flink's API.
> One
> > of the requirements is the ability to add/change patterns at runtime for
> > anomaly detection (maintaining the systems availability). Any Ideas of
> how
> > could I do that?
> >
> > For instance, If I have a stream of security events (accesses,
> > authentications ,etc.) and a pattern for detecting anomalies I would
> like to
> > be able to change that

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

2016-10-11 Thread lgfmt

Hi Sameer,
I just replied to the earlier post, but I will copy it here:
We also have the same requirement - we want to allow the user to change the 
matching patterns and have them take effect immediately. I'm wondering whether 
the proposed trigger DSL takes us one step closer:(I don't think it solves the 
problem) or we have to dynamically generate Flink job JAR files when the 
matching rules/patterns are changed and submit them to Flink.

I had thought about using a similar approach, but it is quite restrictive 
because you cannot use the power for Flink CEP engine with this approach. For 
example, I want to be able to use followedBy, next, notFollowedBy (in future) 
operators to detect the patterns and these matching patterns need to be 
user-cofigurable/dynamic/hot deployable. The simple attribute-based 
rules/patterns that you specified can be made dynamic as you mentioned but the 
rules/patterns that use not just the current event attributes, but also past 
events (e.g. followedBy) are much harder to make them dynamic without some help 
from Flink that implements the CEP operators.

- LF

  From: Sameer W 
 To: "user@flink.apache.org"  
 Sent: Tuesday, October 11, 2016 2:23 PM
 Subject: Re: What is the best way to load/add patterns dynamically (at 
runtime) with Flink?

I have used a JavaScript engine in my CEP to evaluate my patterns. Each event 
is a list of named attributes (HashMap like). And event is attached to a list 
of rules expressed as JavaScript code (See example below with one rule but I 
can match as many rules).  The rules are distributed over a connected stream 
which allow it to change over time. This is how I do it to keep my rules 
dynamic. If someone has a better way I would love to hear it as well.

private transient ScriptEngineManager factory = new ScriptEngineManager();
private transient ScriptEngine engine = factory.getEngineByName("JavaScript");
/*Inside open*/
factory = new ScriptEngineManager();

/*Close open*/

/*Inside my operator*/
engine = factory.getEngineByName("JavaScript");
engine.put("evt", value.f1); //value.f1 contains a JSON version of my HashMap 
of attributes
engine.eval(value.f2.rule); //f2.last contains the rule which is evaluated by 
the JavaScript Engine/*Sample JavaScript contained in the call - 
engine.eval(value.f2.rule); is shown below (not the "evt" variable in the 
JavaScript and the the preceding line - engine.put("evt", value.f1);
var evt=JSON.parse(evt);var result = evt.temperature>50 && evt.pressure<900
*/
boolean ret = (boolean)engine.get("result");
if(ret) /*Rule is Matched*/

> On Oct 11, 2016, at 5:01 PM, PedroMrChaves  wrote:
>
> Hello,
>
> I am new to Apache Flink and am trying to build a CEP using Flink's API. One
> of the requirements is the ability to add/change patterns at runtime for
> anomaly detection (maintaining the systems availability). Any Ideas of how
> could I do that?
>
> For instance, If I have a stream of security events (accesses,
> authentications ,etc.) and a pattern for detecting anomalies I would like to
> be able to change that pattern parameters, for instance instead of detecting
> the occurrence of events A->B->C I would like to change the condition on B
> to B’ in order to have a new rule. Moreover, I would like to be able to
> create new patterns dynamically as new use cases arise.
>
> Best Regards,
> Pedro Chaves
>
>
>
>
> --
> View this message in context: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at 
> Nabble.com.

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

2016-10-11 Thread lgfmt

We also have the same requirement - we want to allow the user to change the
matching patterns and have them take effect immediately. I'm wondering whether
the proposed trigger DSL takes us one step closer:(I don't think it solves the
problem) or we have to dynamically generate Flink job JAR files when the
matching rules/patterns are changed and submit them to Flink.

- LF

From: PedroMrChaves
To: user@flink.apache.org
Sent: Tuesday, October 11, 2016 2:01 PM
Subject: What is the best way to load/add patterns dynamically (at runtime)
with Flink?

Hello,

I am new to Apache Flink and am trying to build a CEP using Flink's API. One
of the requirements is the ability to add/change patterns at runtime for
anomaly detection (maintaining the systems availability). Any Ideas of how
could I do that?

For instance, If I have a stream of security events (accesses,
authentications ,etc.) and a pattern for detecting anomalies I would like to
be able to change that pattern parameters, for instance instead of detecting
the occurrence of events A->B->C I would like to change the condition on B
to B’ in order to have a new rule. Moreover, I would like to be able to
create new patterns dynamically as new use cases arise.

Best Regards,
Pedro Chaves

--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.

mapreduce.HadoopOutputFormat config value issue

2016-10-11 Thread Shannon Carey

In Flink 1.1.1, I am seeing what looks like a serialization issue of 
org.apache.hadoop.conf.Configuration or when used with 
org.apache.flink.api.scala.hadoop.mapreduce.HadoopOutputFormat. When I use the 
mapred.HadoopOutputFormat version, it works just fine.

Specifically, the job fails because "java.lang.UnsupportedOperationException: 
You must set the ColumnFamily schema using setColumnFamilySchema." I am 
definitely setting that property, and it appears to be getting serialized, but 
when the config deserializes the setting is gone. Anybody have any ideas? In 
the meantime, I will continue using the "mapred" package.

Stack trace:
java.lang.UnsupportedOperationException: You must set the ColumnFamily schema 
using setColumnFamilySchema.
at 
org.apache.cassandra.hadoop.cql3.CqlBulkOutputFormat.getColumnFamilySchema(CqlBulkOutputFormat.java:184)
at 
org.apache.cassandra.hadoop.cql3.CqlBulkRecordWriter.setConfigs(CqlBulkRecordWriter.java:94)
at 
org.apache.cassandra.hadoop.cql3.CqlBulkRecordWriter.(CqlBulkRecordWriter.java:74)
at 
org.apache.cassandra.hadoop.cql3.CqlBulkOutputFormat.getRecordWriter(CqlBulkOutputFormat.java:86)
at 
org.apache.cassandra.hadoop.cql3.CqlBulkOutputFormat.getRecordWriter(CqlBulkOutputFormat.java:52)
at 
org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormatBase.open(HadoopOutputFormatBase.java:146)
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:176)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)


Code that works:

val insertStmt = s"INSERT INTO $keyspace.$colFamily (user_id, updated_time, 
value) VALUES (?, ?, ?)"
val config = new JobConf()

ConfigHelper.setOutputInitialAddress(config, 
initialOutputAddress.getHostAddress)

CqlBulkOutputFormat.setColumnFamilySchema(config, colFamily, cqlTableSchema)
CqlBulkOutputFormat.setColumnFamilyInsertStatement(config, colFamily, 
insertStmt)
CqlBulkOutputFormat.setDeleteSourceOnSuccess(config, true)
ConfigHelper.setOutputColumnFamily(config,
  keyspace,
  colFamily)
ConfigHelper.setOutputPartitioner(config, partitionerClass)

val outputFormat = new mapred.HadoopOutputFormat[Object, 
java.util.List[ByteBuffer]](
  new CqlBulkOutputFormat,
  config)

Code that doesn't work:

val insertStmt = s"INSERT INTO $keyspace.$colFamily (user_id, updated_time, 
value) VALUES (?, ?, ?)"
val config = new Configuration()

ConfigHelper.setOutputInitialAddress(config, 
initialOutputAddress.getHostAddress)

CqlBulkOutputFormat.setColumnFamilySchema(config, colFamily, cqlTableSchema)
CqlBulkOutputFormat.setColumnFamilyInsertStatement(config, colFamily, 
insertStmt)
CqlBulkOutputFormat.setDeleteSourceOnSuccess(config, true)
ConfigHelper.setOutputColumnFamily(config,
  keyspace,
  colFamily)
ConfigHelper.setOutputPartitioner(config, partitionerClass)

val hadoopJob: Job = Job.getInstance(config)

val outputFormat = new mapreduce.HadoopOutputFormat[Object, 
java.util.List[ByteBuffer]](
  new CqlBulkOutputFormat,
  hadoopJob)

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

I have used a JavaScript engine in my CEP to evaluate my patterns. Each
event is a list of named attributes (HashMap like). And event is attached
to a list of rules expressed as JavaScript code (See example below with one
rule but I can match as many rules).  The rules are distributed over a
connected stream which allow it to change over time. This is how I do it to
keep my rules dynamic. If someone has a better way I would love to hear it
as well.


private transient ScriptEngineManager factory = new ScriptEngineManager();
private transient ScriptEngine engine =
factory.getEngineByName("JavaScript");
/*Inside open*/
factory = new ScriptEngineManager();

/*Close open*/

/*Inside my operator*/
engine = factory.getEngineByName("JavaScript");
engine.put("evt", value.f1); //value.f1 contains a JSON version of my
HashMap of attributes
engine.eval(value.f2.rule); //f2.last contains the rule which is evaluated
by the JavaScript Engine
/*
Sample JavaScript contained in the call - engine.eval(value.f2.rule); is
shown below (not the "evt" variable in the JavaScript and the the preceding
line - engine.put("evt", value.f1);

*var evt=JSON.parse(evt);var result = evt.temperature>50 &&
evt.pressure<900*
*/
boolean ret = (boolean)engine.get("result");

if(ret) /*Rule is Matched*/



> On Oct 11, 2016, at 5:01 PM, PedroMrChaves 
wrote:
>
> Hello,
>
> I am new to Apache Flink and am trying to build a CEP using Flink's API.
One
> of the requirements is the ability to add/change patterns at runtime for
> anomaly detection (maintaining the systems availability). Any Ideas of how
> could I do that?
>
> For instance, If I have a stream of security events (accesses,
> authentications ,etc.) and a pattern for detecting anomalies I would like
to
> be able to change that pattern parameters, for instance instead of
detecting
> the occurrence of events A->B->C I would like to change the condition on B
> to B’ in order to have a new rule. Moreover, I would like to be able to
> create new patterns dynamically as new use cases arise.
>
> Best Regards,
> Pedro Chaves
>
>
>
>
> --
> View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
> Sent from the Apache Flink User Mailing List archive. mailing list
archive at Nabble.com.

What is the best way to load/add patterns dynamically (at runtime) with Flink?

2016-10-11 Thread PedroMrChaves

Hello,

I am new to Apache Flink and am trying to build a CEP using Flink's API. One
of the requirements is the ability to add/change patterns at runtime for
anomaly detection (maintaining the systems availability). Any Ideas of how
could I do that?

For instance, If I have a stream of security events (accesses,
authentications ,etc.) and a pattern for detecting anomalies I would like to
be able to change that pattern parameters, for instance instead of detecting
the occurrence of events A->B->C I would like to change the condition on B
to B’ in order to have a new rule. Moreover, I would like to be able to
create new patterns dynamically as new use cases arise. 

Best Regards,
Pedro Chaves
 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Listening to timed-out patterns in Flink CEP

Assuming an element with timestamp which is later than the last emitted
watermark arrives, would it just be dropped because the PatternStream does
not have a max allowed lateness method? In that case it appears that CEP
cannot handle late events yet out of the box.

If we do want to support late events can we chain a
keyBy().timeWindow().allowedLateness(x).map().assignTimestampsAndWatermarks().keyBy()
again before handing it to the CEP operator. This way we may have the
patterns fired multiple times but it allows an event to be late and out of
order. It looks like it will work but is there a less convoluted way.

Thanks,
Sameer

On Tue, Oct 11, 2016 at 12:17 PM, Till Rohrmann 
wrote:

> But then no element later than the last emitted watermark must be issued
> by the sources. If that is the case, then this solution should work.
>
> Cheers,
> Till
>
> On Tue, Oct 11, 2016 at 4:50 PM, Sameer W  wrote:
>
>> Hi,
>>
>> If you know that the events are arriving in order and a consistent lag,
>> why not just increment the watermark time every time the
>> getCurrentWatermark() method is invoked based on the autoWatermarkInterval
>> (or less to be conservative).
>>
>> You can check if the watermark has changed since the arrival of the last
>> event and if not increment it in the getCurrentWatermark() method.
>> Otherwise the watermark will never increase until an element arrive and if
>> the stream partition stalls for some reason the whole pipeline freezes.
>>
>> Sameer
>>
>>
>> On Tue, Oct 11, 2016 at 6:04 AM, Till Rohrmann 
>> wrote:
>>
>>> Hi David,
>>>
>>> the problem is still that there is no corresponding watermark saying
>>> that 4 seconds have now passed. With your code, watermarks will be
>>> periodically emitted but the same watermark will be emitted until a new
>>> element arrives which will reset the watermark. Thus, the system can never
>>> know until this watermark is seen whether there will be an earlier event or
>>> not. I fear that this is a fundamental problem with stream processing.
>>>
>>> You're right that the negation operator won't solve the problem. It will
>>> indeed suffer from the same problem.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Sun, Oct 9, 2016 at 7:37 PM,  wrote:
>>>
 >>FLINK-3320  (CEP
 "not" operator) does not address this because again, how would the "not
 match" be triggered if no event at all occurs?

 Good question.

 I'm not sure whether the following will work:

 This could be done by creating a CEP matching pattern that uses both of
 "notNext" (or "notFollowedBy") and "within" constructs. Something like 
 this:

 Pattern pattern = Pattern.begin("first")
 .notNext("second")
 .within(Time.seconds(3));

 I'm hoping Flink CEP experts (Till?) will comment on this.

 Note: I have requested these negation patterns to be implemented in
 Flink CEP, but notNext/notFollowedBy are not yet implemented in Flink..

 - LF

 --
 *From:* David Koch 
 *To:* user@flink.apache.org; lg...@yahoo.com
 *Sent:* Sunday, October 9, 2016 5:51 AM

 *Subject:* Re: Listening to timed-out patterns in Flink CEP

 Hello,

 Thank you for the explanation as well as the link to the other post.
 Interesting to learn about some of the open JIRAs.

 Indeed, I was not using event time, but processing time. However, even
 when using event time I only get notified of timeouts upon subsequent
 events.

 The link  contains an example where I
 read   from a socket, wrap this in a custom "event" with
 timestamp, key the resultant stream by  and attempt to detect 
 instances no further than 3 seconds apart using CEP.

 Apart from the fact that results are only printed when I close the
 socket (normal?) I don't observe any change in behaviour

 So event-time/watermarks or not: SOME event has to occur for the
 timeout to be triggered.

 FLINK-3320  (CEP
 "not" operator) does not address this because again, how would the "not
 match" be triggered if no event at all occurs?

 On Sat, Oct 8, 2016 at 12:50 AM,  wrote:

 The following is a better link:

 http://mail-archives.apache. org/mod_mbox/flink-user/
 201609.mbox/%3CCAC27z% 3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g%
 40mail.gmail.com%3E

 - LF

 --
 *From:* "lg...@yahoo.com"

Re: Exception when restoring state from RocksDB - how to recover?

Ah ok great, thanks! I will try upgrading sometime this week then.

Cheers,
Josh

On Tue, Oct 11, 2016 at 5:37 PM, Stephan Ewen  wrote:

> Hi Josh!
>
> I think the master has gotten more stable with respect to that. The issue
> you mentioned should be fixed.
>
> Another big set of changes (the last big batch) is going in in the next
> days - this time for re-sharding timers (window operator) and other state
> that is not organized by key.
>
> If you want to be a bit conservative, give it a few days before jumping
> onto the latest master. If you are brave, give it a shot now ;-)
>
> Greetings,
> Stephan
>
>
> On Tue, Oct 11, 2016 at 5:43 PM, Josh  wrote:
>
>> Hi Stephan,
>>
>> Thanks, that sounds good!
>>
>> I'm planning to upgrade to Flink 1.2-SNAPSHOT as soon as possible - I was
>> delaying upgrading due to the issues with restoring operator state you
>> mentioned on my other thread here:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Flink-job-fails-to-restore-RocksDB-state-after-
>> upgrading-to-1-2-SNAPSHOT-td9110.html
>>
>> Sorry to jump around but do you know if that's fixed in the latest
>> 1.2-SNAPSHOT? Was it resolved by Flink-4788?
>>
>> Thanks,
>> Josh
>>
>> On Tue, Oct 11, 2016 at 4:13 PM, Stephan Ewen  wrote:
>>
>>> Hi Josh!
>>>
>>> There are two ways to improve the RocksDB / S3 behavior
>>>
>>> (1) Use the FullyAsync mode. It stores the data in one file, not in a
>>> directory. Since directories are the "eventual consistent" part of S3, this
>>> prevents many issues.
>>>
>>> (2) Flink 1.2-SNAPSHOT has some additional fixes that circumvent
>>> additional S3 issues.
>>>
>>> Hope that helps,
>>> Stephan
>>>
>>>
>>> On Tue, Oct 11, 2016 at 4:42 PM, Josh  wrote:
>>>
 Hi Aljoscha,

 Yeah I'm using S3. Is this a known problem when using S3? Do you have
 any ideas on how to restore my job from this state, or prevent it from
 happening again?

 Thanks,
 Josh


 On Tue, Oct 11, 2016 at 1:58 PM, Aljoscha Krettek 
 wrote:

> Hi,
> you are using S3 to store the checkpoints, right? It might be that
> you're running into a problem with S3 "directory listings" not being
> consistent.
>
> Cheers,
> Aljoscha
>
> On Tue, 11 Oct 2016 at 12:40 Josh  wrote:
>
> Hi all,
>
>
> I just have a couple of questions about checkpointing and restoring state 
> from RocksDB.
>
>
> 1) In some cases, I find that it is impossible to restore a job from a 
> checkpoint, due to an exception such as the one pasted below[*]. In this 
> case, it appears that the last checkpoint is somehow corrupt. Does anyone 
> know why this might happen?
>
>
> 2) When the above happens, I have no choice but to cancel the job, as it 
> repeatedly attempts to restart and keeps getting the same exception. 
> Given that no savepoint was taken recently, is it possible for me to 
> restore the job from an older checkpoint (e.g. the second-last 
> checkpoint)?
>
>
> The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.
>
>
> Thanks,
>
> Josh
>
>
> [*]The exception when restoring state:
>
> java.lang.Exception: Could not restore checkpointed state to operators 
> and functions
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Error while restoring RocksDB 
> state from 
> /mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
>   ... 3 more
> Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
>   at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
>   at 
>

Re: Exception when restoring state from RocksDB - how to recover?

2016-10-11 Thread Stephan Ewen

Hi Josh!

I think the master has gotten more stable with respect to that. The issue
you mentioned should be fixed.

Another big set of changes (the last big batch) is going in in the next
days - this time for re-sharding timers (window operator) and other state
that is not organized by key.

If you want to be a bit conservative, give it a few days before jumping
onto the latest master. If you are brave, give it a shot now ;-)

Greetings,
Stephan

On Tue, Oct 11, 2016 at 5:43 PM, Josh  wrote:

> Hi Stephan,
>
> Thanks, that sounds good!
>
> I'm planning to upgrade to Flink 1.2-SNAPSHOT as soon as possible - I was
> delaying upgrading due to the issues with restoring operator state you
> mentioned on my other thread here:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Flink-job-fails-to-restore-RocksDB-state-
> after-upgrading-to-1-2-SNAPSHOT-td9110.html
>
> Sorry to jump around but do you know if that's fixed in the latest
> 1.2-SNAPSHOT? Was it resolved by Flink-4788?
>
> Thanks,
> Josh
>
> On Tue, Oct 11, 2016 at 4:13 PM, Stephan Ewen  wrote:
>
>> Hi Josh!
>>
>> There are two ways to improve the RocksDB / S3 behavior
>>
>> (1) Use the FullyAsync mode. It stores the data in one file, not in a
>> directory. Since directories are the "eventual consistent" part of S3, this
>> prevents many issues.
>>
>> (2) Flink 1.2-SNAPSHOT has some additional fixes that circumvent
>> additional S3 issues.
>>
>> Hope that helps,
>> Stephan
>>
>>
>> On Tue, Oct 11, 2016 at 4:42 PM, Josh  wrote:
>>
>>> Hi Aljoscha,
>>>
>>> Yeah I'm using S3. Is this a known problem when using S3? Do you have
>>> any ideas on how to restore my job from this state, or prevent it from
>>> happening again?
>>>
>>> Thanks,
>>> Josh
>>>
>>>
>>> On Tue, Oct 11, 2016 at 1:58 PM, Aljoscha Krettek 
>>> wrote:
>>>
 Hi,
 you are using S3 to store the checkpoints, right? It might be that
 you're running into a problem with S3 "directory listings" not being
 consistent.

 Cheers,
 Aljoscha

 On Tue, 11 Oct 2016 at 12:40 Josh  wrote:

 Hi all,

 I just have a couple of questions about checkpointing and restoring state 
 from RocksDB.

 1) In some cases, I find that it is impossible to restore a job from a 
 checkpoint, due to an exception such as the one pasted below[*]. In this 
 case, it appears that the last checkpoint is somehow corrupt. Does anyone 
 know why this might happen?

 2) When the above happens, I have no choice but to cancel the job, as it 
 repeatedly attempts to restart and keeps getting the same exception. Given 
 that no savepoint was taken recently, is it possible for me to restore the 
 job from an older checkpoint (e.g. the second-last checkpoint)?

 The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.

 Thanks,

 Josh

 [*]The exception when restoring state:

 java.lang.Exception: Could not restore checkpointed state to operators and 
 functions
at 
 org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
at 
 org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.RuntimeException: Error while restoring RocksDB state 
 from 
 /mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
at 
 org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
at 
 org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
at 
 org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
at 
 org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
at 
 org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
... 3 more
 Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
at 
 org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
at 
 org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
... 7 more

>>>
>>
>

Re: CEP and slightly out of order elements

Thanks Till - This is helpful to know.

Sameer

On Tue, Oct 11, 2016 at 12:20 PM, Till Rohrmann 
wrote:

> Hi Sameer,
>
> the CEP operator will take care of ordering the elements.
>
> Internally what happens is that the elements are buffered before being
> applied to the state machine. The operator only applies the elements after
> it has seen a watermark which is greater than the timestamps of the
> elements being applied to the NFA. Since the elements are kept in a
> priority queue wrt the timestamp they will be in order.
>
> Cheers,
> Till
>
> On Tue, Oct 11, 2016 at 1:51 PM, Sameer W  wrote:
>
>> Hi,
>>
>> If using CEP with event-time I have events which can be slightly out of
>> order and I want to sort them by timestamp within their time-windows before
>> applying CEP-
>>
>> For example, if using 5 second windows and I use the following
>>
>> ds2 = ds.keyBy.window(TumblingWindow(10 seconds).apply(/*Sort by
>> Timestamp*/);
>>
>> Next assign watermarks again on ds2 (because elements in ds2 will all
>> have the same timestamp of WINDOW_END_TIME-1ms)
>> ds2.assignTimestampsAndWatermarks()
>>
>> Finally apply CEP on ds2 with a WITHIN window of 5 seconds (shorter
>> timestamp than the one I used earlier).
>>
>> The reasoning is, if I am using the next() operator in CEP, the events
>> should be in the order of their timestamps.
>>
>> Is this the right way to handle this problem? I have heard people say
>> that assigning watermarks twice can lead to wrong results. But don't I need
>> to assign timestamps once more in this scenario.
>>
>> Thanks,
>> Sameer
>>
>>
>>
>>
>>
>

Re: Listening to timed-out patterns in Flink CEP

2016-10-11 Thread David Koch

I will give it a try, my current time/watermark assigner extends
AscendingTimestampExtractor so I can't override setting the watermark to
the last seen event timestamp.

Thanks for your replies.

/David

On Tue, Oct 11, 2016 at 6:17 PM, Till Rohrmann 
wrote:

> But then no element later than the last emitted watermark must be issued
> by the sources. If that is the case, then this solution should work.
>
> Cheers,
> Till
>
> On Tue, Oct 11, 2016 at 4:50 PM, Sameer W  wrote:
>
>> Hi,
>>
>> If you know that the events are arriving in order and a consistent lag,
>> why not just increment the watermark time every time the
>> getCurrentWatermark() method is invoked based on the autoWatermarkInterval
>> (or less to be conservative).
>>
>> You can check if the watermark has changed since the arrival of the last
>> event and if not increment it in the getCurrentWatermark() method.
>> Otherwise the watermark will never increase until an element arrive and if
>> the stream partition stalls for some reason the whole pipeline freezes.
>>
>> Sameer
>>
>>
>> On Tue, Oct 11, 2016 at 6:04 AM, Till Rohrmann 
>> wrote:
>>
>>> Hi David,
>>>
>>> the problem is still that there is no corresponding watermark saying
>>> that 4 seconds have now passed. With your code, watermarks will be
>>> periodically emitted but the same watermark will be emitted until a new
>>> element arrives which will reset the watermark. Thus, the system can never
>>> know until this watermark is seen whether there will be an earlier event or
>>> not. I fear that this is a fundamental problem with stream processing.
>>>
>>> You're right that the negation operator won't solve the problem. It will
>>> indeed suffer from the same problem.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Sun, Oct 9, 2016 at 7:37 PM,  wrote:
>>>
 >>FLINK-3320  (CEP
 "not" operator) does not address this because again, how would the "not
 match" be triggered if no event at all occurs?

 Good question.

 I'm not sure whether the following will work:

 This could be done by creating a CEP matching pattern that uses both of
 "notNext" (or "notFollowedBy") and "within" constructs. Something like 
 this:

 Pattern pattern = Pattern.begin("first")
 .notNext("second")
 .within(Time.seconds(3));

 I'm hoping Flink CEP experts (Till?) will comment on this.

 Note: I have requested these negation patterns to be implemented in
 Flink CEP, but notNext/notFollowedBy are not yet implemented in Flink..

 - LF

 --
 *From:* David Koch 
 *To:* user@flink.apache.org; lg...@yahoo.com
 *Sent:* Sunday, October 9, 2016 5:51 AM

 *Subject:* Re: Listening to timed-out patterns in Flink CEP

 Hello,

 Thank you for the explanation as well as the link to the other post.
 Interesting to learn about some of the open JIRAs.

 Indeed, I was not using event time, but processing time. However, even
 when using event time I only get notified of timeouts upon subsequent
 events.

 The link  contains an example where I
 read   from a socket, wrap this in a custom "event" with
 timestamp, key the resultant stream by  and attempt to detect 
 instances no further than 3 seconds apart using CEP.

 Apart from the fact that results are only printed when I close the
 socket (normal?) I don't observe any change in behaviour

 So event-time/watermarks or not: SOME event has to occur for the
 timeout to be triggered.

 FLINK-3320  (CEP
 "not" operator) does not address this because again, how would the "not
 match" be triggered if no event at all occurs?

 On Sat, Oct 8, 2016 at 12:50 AM,  wrote:

 The following is a better link:

 http://mail-archives.apache. org/mod_mbox/flink-user/
 201609.mbox/%3CCAC27z% 3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g%
 40mail.gmail.com%3E

 - LF

 --
 *From:* "lg...@yahoo.com" 
 *To:* "user@flink.apache.org" 
 *Sent:* Friday, October 7, 2016 3:36 PM

 *Subject:* Re: Listening to timed-out patterns in Flink CEP

 Isn't the upcoming CEP negation (absence of an event) feature solve
 this issue?

 See this discussion thread:
 http://mail-archives.apache. org/mod_mbox/flink-user/
 201609.mbox/%3CCAC27z%3DOD% 2BTq8twBw_ 1YKni5sWAU3g1S9WDpJw0DUwgiG9YX

Re: CEP and slightly out of order elements

Hi Sameer,

the CEP operator will take care of ordering the elements.

Internally what happens is that the elements are buffered before being
applied to the state machine. The operator only applies the elements after
it has seen a watermark which is greater than the timestamps of the
elements being applied to the NFA. Since the elements are kept in a
priority queue wrt the timestamp they will be in order.

Cheers,
Till

On Tue, Oct 11, 2016 at 1:51 PM, Sameer W  wrote:

> Hi,
>
> If using CEP with event-time I have events which can be slightly out of
> order and I want to sort them by timestamp within their time-windows before
> applying CEP-
>
> For example, if using 5 second windows and I use the following
>
> ds2 = ds.keyBy.window(TumblingWindow(10 seconds).apply(/*Sort by
> Timestamp*/);
>
> Next assign watermarks again on ds2 (because elements in ds2 will all have
> the same timestamp of WINDOW_END_TIME-1ms)
> ds2.assignTimestampsAndWatermarks()
>
> Finally apply CEP on ds2 with a WITHIN window of 5 seconds (shorter
> timestamp than the one I used earlier).
>
> The reasoning is, if I am using the next() operator in CEP, the events
> should be in the order of their timestamps.
>
> Is this the right way to handle this problem? I have heard people say that
> assigning watermarks twice can lead to wrong results. But don't I need to
> assign timestamps once more in this scenario.
>
> Thanks,
> Sameer
>
>
>
>
>

Re: Listening to timed-out patterns in Flink CEP

But then no element later than the last emitted watermark must be issued by
the sources. If that is the case, then this solution should work.

Cheers,
Till

On Tue, Oct 11, 2016 at 4:50 PM, Sameer W  wrote:

> Hi,
>
> If you know that the events are arriving in order and a consistent lag,
> why not just increment the watermark time every time the
> getCurrentWatermark() method is invoked based on the autoWatermarkInterval
> (or less to be conservative).
>
> You can check if the watermark has changed since the arrival of the last
> event and if not increment it in the getCurrentWatermark() method.
> Otherwise the watermark will never increase until an element arrive and if
> the stream partition stalls for some reason the whole pipeline freezes.
>
> Sameer
>
>
> On Tue, Oct 11, 2016 at 6:04 AM, Till Rohrmann 
> wrote:
>
>> Hi David,
>>
>> the problem is still that there is no corresponding watermark saying that
>> 4 seconds have now passed. With your code, watermarks will be periodically
>> emitted but the same watermark will be emitted until a new element arrives
>> which will reset the watermark. Thus, the system can never know until this
>> watermark is seen whether there will be an earlier event or not. I fear
>> that this is a fundamental problem with stream processing.
>>
>> You're right that the negation operator won't solve the problem. It will
>> indeed suffer from the same problem.
>>
>> Cheers,
>> Till
>>
>> On Sun, Oct 9, 2016 at 7:37 PM,  wrote:
>>
>>> >>FLINK-3320  (CEP
>>> "not" operator) does not address this because again, how would the "not
>>> match" be triggered if no event at all occurs?
>>>
>>> Good question.
>>>
>>> I'm not sure whether the following will work:
>>>
>>> This could be done by creating a CEP matching pattern that uses both of
>>> "notNext" (or "notFollowedBy") and "within" constructs. Something like this:
>>>
>>> Pattern pattern = Pattern.begin("first")
>>> .notNext("second")
>>> .within(Time.seconds(3));
>>>
>>> I'm hoping Flink CEP experts (Till?) will comment on this.
>>>
>>> Note: I have requested these negation patterns to be implemented in
>>> Flink CEP, but notNext/notFollowedBy are not yet implemented in Flink..
>>>
>>>
>>> - LF
>>>
>>>
>>>
>>>
>>> --
>>> *From:* David Koch 
>>> *To:* user@flink.apache.org; lg...@yahoo.com
>>> *Sent:* Sunday, October 9, 2016 5:51 AM
>>>
>>> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>>>
>>> Hello,
>>>
>>> Thank you for the explanation as well as the link to the other post.
>>> Interesting to learn about some of the open JIRAs.
>>>
>>> Indeed, I was not using event time, but processing time. However, even
>>> when using event time I only get notified of timeouts upon subsequent
>>> events.
>>>
>>> The link  contains an example where I
>>> read   from a socket, wrap this in a custom "event" with
>>> timestamp, key the resultant stream by  and attempt to detect 
>>> instances no further than 3 seconds apart using CEP.
>>>
>>> Apart from the fact that results are only printed when I close the
>>> socket (normal?) I don't observe any change in behaviour
>>>
>>> So event-time/watermarks or not: SOME event has to occur for the timeout
>>> to be triggered.
>>>
>>> FLINK-3320  (CEP
>>> "not" operator) does not address this because again, how would the "not
>>> match" be triggered if no event at all occurs?
>>>
>>> On Sat, Oct 8, 2016 at 12:50 AM,  wrote:
>>>
>>> The following is a better link:
>>>
>>> http://mail-archives.apache. org/mod_mbox/flink-user/
>>> 201609.mbox/%3CCAC27z% 3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g%
>>> 40mail.gmail.com%3E
>>> 
>>>
>>>
>>> - LF
>>>
>>>
>>>
>>>
>>> --
>>> *From:* "lg...@yahoo.com" 
>>> *To:* "user@flink.apache.org" 
>>> *Sent:* Friday, October 7, 2016 3:36 PM
>>>
>>> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>>>
>>> Isn't the upcoming CEP negation (absence of an event) feature solve
>>> this issue?
>>>
>>> See this discussion thread:
>>> http://mail-archives.apache. org/mod_mbox/flink-user/
>>> 201609.mbox/%3CCAC27z%3DOD% 2BTq8twBw_ 1YKni5sWAU3g1S9WDpJw0DUwgiG9YX
>>> 9Fg%40mail.gmail.com%3E
>>> 
>>>
>>>
>>>
>>> //  Atul
>>>
>>>
>>> --
>>> *From:* Till Rohrmann 
>>> *To:* user@flink.apache.org
>>> *Sent:* Friday, October 7, 2016 12:58 AM
>>> *Subject:* Re: Listening to timed-out patterns

Re: PathIsNotEmptyDirectoryException in Namenode HDFS log when using Jobmanager HA in YARN

2016-10-11 Thread Stephan Ewen

Hi!

I think to some extend this is expected. There is some cleanup code that
deletes files and then  issues parent directory remove requests. It relies
on the fact that the parent directory is only removed if it is empty (after
the last file was deleted).

Is this a problem right now, or just a confusing behavior?

Greetings,
Stephan


On Tue, Oct 11, 2016 at 5:25 PM, static-max 
wrote:

> Hi,
>
> I get many (multiple times per minute) errors in my Namenode HDFS logfile:
>
> 2016-10-11 17:17:07,596 INFO  ipc.Server (Server.java:logException(2401))
> - IPC Server handler 295 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.delete
> from datanode1:34872 Call#2361 Retry#0
> org.apache.hadoop.fs.PathIsNotEmptyDirectoryException: `/flink/recovery
> is non empty': Directory is not empty
> at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(
> FSDirDeleteOp.java:89)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(
> FSNamesystem.java:3829)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> delete(NameNodeRpcServer.java:1071)
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.delete(
> ClientNamenodeProtocolServerSideTranslatorPB.java:619)
> at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
>
> That is the directory I configured for Jobmanager HA. I deleted it before
> starting the YARN session but that did not help. The folder gets created by
> Flink without problems.
>
> I'm using latest Flink Master (Commit: 6731ec1) and build it for Hadoop
> 2.7.3.
>
> Any idea is highly appreciated. Thanks a lot!
>

Re: more complex patterns for CEP (was: CEP two transitions to the same state)

2016-10-11 Thread lgfmt

Thanks, Till. I will wait for your response.
- LF

  From: Till Rohrmann 
 To: user@flink.apache.org; lg...@yahoo.com 
 Sent: Tuesday, October 11, 2016 2:49 AM
 Subject: Re: more complex patterns for CEP (was: CEP two transitions to the 
same state)

The timeline is hard to predict to be honest. It depends a little bit on how 
fast the community can proceed with these things. At the moment I'm personally 
involved in other issues and, thus, cannot work on the CEP library. I hope to 
get back to it soon.
Cheers,Till
On Sat, Oct 8, 2016 at 12:42 AM,  wrote:

hi Till,
Thanks for the detailed response.
I'm looking forward to seeing these features implemented in Flink. Can anyone 
provide timelines for the 3 tickets that you mentioned in your response?  - LF

  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Tuesday, September 20, 2016 7:13 AM
 Subject: Re: more complex patterns for CEP (was: CEP two transitions to the 
same state)

Hi Frank,
thanks for sharing your analysis. It indeed pinpoints some of the current CEP 
library's shortcomings.
Let me address your points:
1. Lack of not operator
The functionality to express events which must not occur in a pattern is 
missing. We've currently a JIRA [1] which addresses exactly this. For the 
notFollowedBy operator, we should discard all patterns where we've seen a 
matching event for the not state. I think it could be implemented like a 
special terminal state where we prune the partial pattern.
For the notNext operator, we could think about keeping the event which has not 
matched the notNext state and return it as part of the fully matched pattern. 
Alternatively, we could simply forget about it once we've assured that it does 
not match.
2. Allow functions to access fields of previous events
This hasn't been implemented yet because it is a quite expensive operation. 
Before calling the filter function you always have to reconstruct the current 
partial pattern and then give it to the filter function. But I agree that the 
user should be allowed to use such a functionality (and then pay the price for 
it in terms of efficiency). Giving access to the partially matched fields via a 
Map would be a way to solve the problem on the API level.
I think that almost all functionality for this feature is already in place. We 
simply would have to check the filter condition whether they require access to 
previous events and then compute the partial pattern.
3. Support for recursive patterns
The underlying SharedBuffer implementation should allow recursive event 
patterns. Once we have support for branching CEP patterns [2] which allow to 
connect different states this should also be possible with some minor changes.
However, a more interesting way to specify recursive CEP patterns is to use 
regular expression syntax (Kleene star, bounded occurrences) to express 
recursive parts of a pattern. I think this makes specifying such a pattern 
easier and more intuitive for the user. We've also a JIRA issue to track the 
process there [3] and Ivan is already working on this.
If you want to get involved in Flink's CEP development, then feel free to take 
over any free JIRA issue or create one yourself :-)
[1] https://issues.apache.org/ jira/browse/FLINK-3320
[2] https://issues.apache.org/ jira/browse/FLINK-4641[3] 
https://issues.apache.org/ jira/browse/FLINK-3318
Cheers,Till
On Fri, Sep 16, 2016 at 10:04 PM, Frank Dekervel  wrote:

Hello,
i did some more analysis wrt the problem i'm facing and the flink CEP api.
In order to complete the problem i'm facing using flink CEP i would need 3 
additions to the API (i think). I tried to understand the NFA logic, and i 
think 2 of them should be doable without fundamental changes.
First is to add a "negative" pattern (notFollowedBy / notNext):
Reason is the flow below: i have a start and a termination event, and an 
optional "failure" event in between. i want all succesful termination events, 
so i want to express there should not be a failure event between the start and 
the termination event. Note that there is no "success" event in this case on 
which i could match.

To implement, upon checking whether a transition would be possible, one would 
first need to check if it was not already dead-ended by a notFollowedBy / 
notNext. This would add a bit of complexity to the logic (when seeing if a 
transition is valid for a state, first check if on this state there was not 
already a match made to an notFollowedBy/notNext state. in that case one would 
reject the match)
Second is to allow the filterfunction to inspect the partial match made, so one 
would be able to filter based on the already-matched event. Reason is the 
following (hypothetical) example where we would match arrivals of a trains in a 
station. We cannot keyBy train (because the "occupied" events of the station 
don't have train information), neither can we keyBy

Re: Exception when restoring state from RocksDB - how to recover?

Hi Stephan,

Thanks, that sounds good!

I'm planning to upgrade to Flink 1.2-SNAPSHOT as soon as possible - I was
delaying upgrading due to the issues with restoring operator state you
mentioned on my other thread here:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-job-fails-to-restore-RocksDB-state-after-upgrading-to-1-2-SNAPSHOT-td9110.html

Sorry to jump around but do you know if that's fixed in the latest
1.2-SNAPSHOT? Was it resolved by Flink-4788?

Thanks,
Josh

On Tue, Oct 11, 2016 at 4:13 PM, Stephan Ewen  wrote:

> Hi Josh!
>
> There are two ways to improve the RocksDB / S3 behavior
>
> (1) Use the FullyAsync mode. It stores the data in one file, not in a
> directory. Since directories are the "eventual consistent" part of S3, this
> prevents many issues.
>
> (2) Flink 1.2-SNAPSHOT has some additional fixes that circumvent
> additional S3 issues.
>
> Hope that helps,
> Stephan
>
>
> On Tue, Oct 11, 2016 at 4:42 PM, Josh  wrote:
>
>> Hi Aljoscha,
>>
>> Yeah I'm using S3. Is this a known problem when using S3? Do you have any
>> ideas on how to restore my job from this state, or prevent it from
>> happening again?
>>
>> Thanks,
>> Josh
>>
>>
>> On Tue, Oct 11, 2016 at 1:58 PM, Aljoscha Krettek 
>> wrote:
>>
>>> Hi,
>>> you are using S3 to store the checkpoints, right? It might be that
>>> you're running into a problem with S3 "directory listings" not being
>>> consistent.
>>>
>>> Cheers,
>>> Aljoscha
>>>
>>> On Tue, 11 Oct 2016 at 12:40 Josh  wrote:
>>>
>>> Hi all,
>>>
>>>
>>> I just have a couple of questions about checkpointing and restoring state 
>>> from RocksDB.
>>>
>>>
>>> 1) In some cases, I find that it is impossible to restore a job from a 
>>> checkpoint, due to an exception such as the one pasted below[*]. In this 
>>> case, it appears that the last checkpoint is somehow corrupt. Does anyone 
>>> know why this might happen?
>>>
>>>
>>> 2) When the above happens, I have no choice but to cancel the job, as it 
>>> repeatedly attempts to restart and keeps getting the same exception. Given 
>>> that no savepoint was taken recently, is it possible for me to restore the 
>>> job from an older checkpoint (e.g. the second-last checkpoint)?
>>>
>>>
>>> The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.
>>>
>>>
>>> Thanks,
>>>
>>> Josh
>>>
>>>
>>> [*]The exception when restoring state:
>>>
>>> java.lang.Exception: Could not restore checkpointed state to operators and 
>>> functions
>>> at 
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
>>> at 
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.RuntimeException: Error while restoring RocksDB state 
>>> from 
>>> /mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
>>> at 
>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
>>> at 
>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
>>> at 
>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
>>> at 
>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
>>> at 
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
>>> ... 3 more
>>> Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
>>> at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
>>> at 
>>> org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
>>> at 
>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
>>> ... 7 more
>>>
>>>
>>>
>>
>

PathIsNotEmptyDirectoryException in Namenode HDFS log when using Jobmanager HA in YARN

2016-10-11 Thread static-max

Hi,

I get many (multiple times per minute) errors in my Namenode HDFS logfile:

2016-10-11 17:17:07,596 INFO  ipc.Server (Server.java:logException(2401)) -
IPC Server handler 295 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.delete from datanode1:34872
Call#2361 Retry#0
org.apache.hadoop.fs.PathIsNotEmptyDirectoryException: `/flink/recovery is
non empty': Directory is not empty
at
org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:89)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3829)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:1071)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:619)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)

That is the directory I configured for Jobmanager HA. I deleted it before
starting the YARN session but that did not help. The folder gets created by
Flink without problems.

I'm using latest Flink Master (Commit: 6731ec1) and build it for Hadoop
2.7.3.

Any idea is highly appreciated. Thanks a lot!

Re: Exception when restoring state from RocksDB - how to recover?

2016-10-11 Thread Stephan Ewen

Hi Josh!

There are two ways to improve the RocksDB / S3 behavior

(1) Use the FullyAsync mode. It stores the data in one file, not in a
directory. Since directories are the "eventual consistent" part of S3, this
prevents many issues.

(2) Flink 1.2-SNAPSHOT has some additional fixes that circumvent additional
S3 issues.

Hope that helps,
Stephan


On Tue, Oct 11, 2016 at 4:42 PM, Josh  wrote:

> Hi Aljoscha,
>
> Yeah I'm using S3. Is this a known problem when using S3? Do you have any
> ideas on how to restore my job from this state, or prevent it from
> happening again?
>
> Thanks,
> Josh
>
>
> On Tue, Oct 11, 2016 at 1:58 PM, Aljoscha Krettek 
> wrote:
>
>> Hi,
>> you are using S3 to store the checkpoints, right? It might be that you're
>> running into a problem with S3 "directory listings" not being consistent.
>>
>> Cheers,
>> Aljoscha
>>
>> On Tue, 11 Oct 2016 at 12:40 Josh  wrote:
>>
>> Hi all,
>>
>>
>> I just have a couple of questions about checkpointing and restoring state 
>> from RocksDB.
>>
>>
>> 1) In some cases, I find that it is impossible to restore a job from a 
>> checkpoint, due to an exception such as the one pasted below[*]. In this 
>> case, it appears that the last checkpoint is somehow corrupt. Does anyone 
>> know why this might happen?
>>
>>
>> 2) When the above happens, I have no choice but to cancel the job, as it 
>> repeatedly attempts to restart and keeps getting the same exception. Given 
>> that no savepoint was taken recently, is it possible for me to restore the 
>> job from an older checkpoint (e.g. the second-last checkpoint)?
>>
>>
>> The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.
>>
>>
>> Thanks,
>>
>> Josh
>>
>>
>> [*]The exception when restoring state:
>>
>> java.lang.Exception: Could not restore checkpointed state to operators and 
>> functions
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
>>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.RuntimeException: Error while restoring RocksDB state 
>> from 
>> /mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
>>  at 
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
>>  at 
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
>>  at 
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
>>  at 
>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
>>  at 
>> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
>>  ... 3 more
>> Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
>>  at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
>>  at 
>> org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
>>  at 
>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
>>  ... 7 more
>>
>>
>>
>

Re: Listening to timed-out patterns in Flink CEP

Hi,

If you know that the events are arriving in order and a consistent lag, why
not just increment the watermark time every time the getCurrentWatermark()
method is invoked based on the autoWatermarkInterval (or less to be
conservative).

You can check if the watermark has changed since the arrival of the last
event and if not increment it in the getCurrentWatermark() method.
Otherwise the watermark will never increase until an element arrive and if
the stream partition stalls for some reason the whole pipeline freezes.

Sameer


On Tue, Oct 11, 2016 at 6:04 AM, Till Rohrmann 
wrote:

> Hi David,
>
> the problem is still that there is no corresponding watermark saying that
> 4 seconds have now passed. With your code, watermarks will be periodically
> emitted but the same watermark will be emitted until a new element arrives
> which will reset the watermark. Thus, the system can never know until this
> watermark is seen whether there will be an earlier event or not. I fear
> that this is a fundamental problem with stream processing.
>
> You're right that the negation operator won't solve the problem. It will
> indeed suffer from the same problem.
>
> Cheers,
> Till
>
> On Sun, Oct 9, 2016 at 7:37 PM,  wrote:
>
>> >>FLINK-3320  (CEP
>> "not" operator) does not address this because again, how would the "not
>> match" be triggered if no event at all occurs?
>>
>> Good question.
>>
>> I'm not sure whether the following will work:
>>
>> This could be done by creating a CEP matching pattern that uses both of
>> "notNext" (or "notFollowedBy") and "within" constructs. Something like this:
>>
>> Pattern pattern = Pattern.begin("first")
>> .notNext("second")
>> .within(Time.seconds(3));
>>
>> I'm hoping Flink CEP experts (Till?) will comment on this.
>>
>> Note: I have requested these negation patterns to be implemented in Flink
>> CEP, but notNext/notFollowedBy are not yet implemented in Flink..
>>
>>
>> - LF
>>
>>
>>
>>
>> --
>> *From:* David Koch 
>> *To:* user@flink.apache.org; lg...@yahoo.com
>> *Sent:* Sunday, October 9, 2016 5:51 AM
>>
>> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>>
>> Hello,
>>
>> Thank you for the explanation as well as the link to the other post.
>> Interesting to learn about some of the open JIRAs.
>>
>> Indeed, I was not using event time, but processing time. However, even
>> when using event time I only get notified of timeouts upon subsequent
>> events.
>>
>> The link  contains an example where I read
>>   from a socket, wrap this in a custom "event" with timestamp,
>> key the resultant stream by  and attempt to detect  instances no
>> further than 3 seconds apart using CEP.
>>
>> Apart from the fact that results are only printed when I close the socket
>> (normal?) I don't observe any change in behaviour
>>
>> So event-time/watermarks or not: SOME event has to occur for the timeout
>> to be triggered.
>>
>> FLINK-3320  (CEP "not"
>> operator) does not address this because again, how would the "not match" be
>> triggered if no event at all occurs?
>>
>> On Sat, Oct 8, 2016 at 12:50 AM,  wrote:
>>
>> The following is a better link:
>>
>> http://mail-archives.apache. org/mod_mbox/flink-user/
>> 201609.mbox/%3CCAC27z% 3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g%
>> 40mail.gmail.com%3E
>> 
>>
>>
>> - LF
>>
>>
>>
>>
>> --
>> *From:* "lg...@yahoo.com" 
>> *To:* "user@flink.apache.org" 
>> *Sent:* Friday, October 7, 2016 3:36 PM
>>
>> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>>
>> Isn't the upcoming CEP negation (absence of an event) feature solve this
>> issue?
>>
>> See this discussion thread:
>> http://mail-archives.apache. org/mod_mbox/flink-user/
>> 201609.mbox/%3CCAC27z%3DOD% 2BTq8twBw_ 1YKni5sWAU3g1S9WDpJw0DUwgiG9YX
>> 9Fg%40mail.gmail.com%3E
>> 
>>
>>
>>
>> //  Atul
>>
>>
>> --
>> *From:* Till Rohrmann 
>> *To:* user@flink.apache.org
>> *Sent:* Friday, October 7, 2016 12:58 AM
>> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>>
>> Hi David,
>>
>> in case of event time, the timeout will be detected when the first
>> watermark exceeding the timeout value is received. Thus, it depends a
>> little bit how you generate watermarks (e.g. periodically, watermark per
>> event).
>>
>> In case of processing time, the time is only updated whenever a new
>> element arrives. Thus, if you have

Re: Exception when restoring state from RocksDB - how to recover?

Hi Aljoscha,

Yeah I'm using S3. Is this a known problem when using S3? Do you have any
ideas on how to restore my job from this state, or prevent it from
happening again?

Thanks,
Josh


On Tue, Oct 11, 2016 at 1:58 PM, Aljoscha Krettek 
wrote:

> Hi,
> you are using S3 to store the checkpoints, right? It might be that you're
> running into a problem with S3 "directory listings" not being consistent.
>
> Cheers,
> Aljoscha
>
> On Tue, 11 Oct 2016 at 12:40 Josh  wrote:
>
> Hi all,
>
>
> I just have a couple of questions about checkpointing and restoring state 
> from RocksDB.
>
>
> 1) In some cases, I find that it is impossible to restore a job from a 
> checkpoint, due to an exception such as the one pasted below[*]. In this 
> case, it appears that the last checkpoint is somehow corrupt. Does anyone 
> know why this might happen?
>
>
> 2) When the above happens, I have no choice but to cancel the job, as it 
> repeatedly attempts to restart and keeps getting the same exception. Given 
> that no savepoint was taken recently, is it possible for me to restore the 
> job from an older checkpoint (e.g. the second-last checkpoint)?
>
>
> The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.
>
>
> Thanks,
>
> Josh
>
>
> [*]The exception when restoring state:
>
> java.lang.Exception: Could not restore checkpointed state to operators and 
> functions
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Error while restoring RocksDB state 
> from 
> /mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
>   ... 3 more
> Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
>   at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
>   at 
> org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
>   at 
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
>   ... 7 more
>
>
>

Re: Handling decompression exceptions

2016-10-11 Thread Yassine MARZOUGUI

Thank you Fabian and Flavio for your help.

Best,
Yassine

2016-10-11 14:02 GMT+02:00 Flavio Pompermaier :

> I posted a workaround for that at https://github.com/okkam-it/
> flink-examples/blob/master/src/main/java/it/okkam/datalinks/batch/flink/
> datasourcemanager/importers/Csv2RowExample.java
>
> On 11 Oct 2016 1:57 p.m., "Fabian Hueske"  wrote:
>
>> Hi,
>>
>> Flink's String parser does not support escaped quotes. You data contains
>> a double double quote (""). The parser identifies this as the end of the
>> string field.
>> As a workaround, you can read the file as a regular text file, line by
>> line and do the parsing in a MapFunction.
>>
>> Best, Fabian
>>
>> 2016-10-11 13:37 GMT+02:00 Yassine MARZOUGUI :
>>
>>> Forgot to add parseQuotedStrings('"'). After adding it I'm getting the
>>> same exception with the second code too.
>>>
>>> 2016-10-11 13:29 GMT+02:00 Yassine MARZOUGUI 
>>> :
>>>
 Hi Fabian,

 I tried to debug the code, and it turns out a line in my csv data is
 causing the ArrayIndexOutOfBoundsException, here is the exception
 stacktrace:

 java.lang.ArrayIndexOutOfBoundsException: -1
 at org.apache.flink.types.parser.StringParser.parseField(String
 Parser.java:49)
 at org.apache.flink.types.parser.StringParser.parseField(String
 Parser.java:28)
 at org.apache.flink.types.parser.FieldParser.resetErrorStateAnd
 Parse(FieldParser.java:98)
 at org.apache.flink.api.common.io.GenericCsvInputFormat.parseRe
 cord(GenericCsvInputFormat.java:395)
 at org.apache.flink.api.java.io.CsvInputFormat.readRecord(CsvIn
 putFormat.java:110)
 at org.apache.flink.api.common.io.DelimitedInputFormat.nextReco
 rd(DelimitedInputFormat.java:470)
 at org.apache.flink.api.java.io.CsvInputFormat.nextRecord(CsvIn
 putFormat.java:78)
 at org.myorg.quickstart.MyCsvInputFormat.nextRecord(MyCsvInputF
 ormat.java:106)
 at org.apache.flink.runtime.operators.DataSourceTask.invoke(Dat
 aSourceTask.java:162)
 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
 at java.lang.Thread.run(Thread.java:745)

 And here is a sample CSV:

 timestamp,url,id
 2016-08-31 12:08:11.223,"https://www.toyota.fr/hybrid-innovation/infogr
 aphie.jsontcgcc, ce)_13h00 /""=/-3h00 %=) 1",000

 Using my code, I get the previous exception when parsing the sample
 CSV. If I use the following code, I get an incorrect result : (2016-08-31
 12:08:11.223, ce)_13h00 /""=/-3h00 %=) 1") instead of (2016-08-31
 12:08:11.223, 000)

 DataSet> withReadCSV = 
 env.readCsvFile("C:\\Users\\yassine\\Desktop\\test.csv")
 .ignoreFirstLine()
 .fieldDelimiter(",")
 .includeFields("101")
 .ignoreInvalidLines()
 .types(String.class, String.class);
 withReadCSV.writeAsText("C:\\Users\\yassine\\Desktop\\withreadcsv.txt", 
 FileSystem.WriteMode.OVERWRITE).setParallelism(1);


 Is it a bug in Flink or is my data not compliant with the csv standards?

 Thanks,
 Yassine


 2016-10-11 11:21 GMT+02:00 Fabian Hueske :

> Hi Yassine,
>
> I ran your code without problems and got the correct result.
> Can you provide the Stacktrace of the Exception?
>
> Thanks, Fabian
>
> 2016-10-10 10:57 GMT+02:00 Yassine MARZOUGUI <
> y.marzou...@mindlytix.com>:
>
>> Thank you Fabian and Stephan for the suggestions.
>> I couldn't override "readLine()" because it's final, so went with
>> Fabian's solution, but I'm struggling with csv field masks. Any help is
>> appreciated.
>> I created an Input Format which is basically TupleCsvInputFormat for
>> which I overrode the nextRecord() method to catch the exceptions. But I'm
>> having a *java.lang.ArrayIndexOutOfBoundsException*  when I add a
>> boolean[]{true, false, true} field mask . If I add a int[]{1,0,1} field
>> mask, the job succeeds but outputs the first and second columns. Here is 
>> my
>> code:
>>
>> TupleTypeInfo> typeInfo =
>> TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
>> Path histPath = new Path("hdfs:///shared/file.csv");
>>
>> CsvInputFormat > myInputFormt = new
>> MyCsvInputFormat<>(histPath, typeInfo, new boolean[]{true, false, true});
>> myInputFormt.enableQuotedStringParsing('"');
>> myInputFormt.setSkipFirstLineAsHeader(true);
>> myInputFormt.setLenient(true);
>>
>> DataSet> test = env.createInput(myInputFormt,t
>> ypeInfo).withParameters(parameters);
>> test.writeAsText("E:\\data\\test.csv", FileSystem.WriteMode.OVERWRITE
>> );
>>
>> and here is the  custom input format:

re: About Sliding window

2016-10-11 Thread Zhangrucong

Hi Kostas:
Thank you for your rapid response!

My use-case is that :
For every incoming event, we want to age the out-of-date event , count the 
event in window and send the result.

For example:
The events are coming as flowing:
[cid:image002.png@01D22401.7DD230E0]

We want flowing result:
[cid:image004.png@01D22402.EF14D4A0]


By the way, In StreamSQL API, in FILP11, It will realize row window. It seems 
that the function of Slide Event-time row-window suits my use-case. Does data 
stream API  support row window?

Thanks !

发件人: Kostas Kloudas [mailto:k.klou...@data-artisans.com]
发送时间: 2016年10月11日 19:38
收件人: user@flink.apache.org
主题: Re: About Sliding window

Hi Zhangrucong,

Sliding windows only support time-based slide.
So your use-case is not supported out-of-the-box.

But, if you describe a bit more what you want to do,
we may be able to find a way together to do your job using
the currently offered functionality.

Kostas

On Oct 11, 2016, at 1:20 PM, Zhangrucong 
> wrote:

Hello everyone:
  Now, I am want to use DataStream sliding window API. I look at the API and I 
have a question, dose the sliding time window support sliding by every incoming 
event?

Thanks in advance!



image001.emz
Description: image001.emz


oledata.mso
Description: oledata.mso


image003.emz
Description: image003.emz

Re: Exception when restoring state from RocksDB - how to recover?

2016-10-11 Thread Aljoscha Krettek

Hi,
you are using S3 to store the checkpoints, right? It might be that you're
running into a problem with S3 "directory listings" not being consistent.

Cheers,
Aljoscha

On Tue, 11 Oct 2016 at 12:40 Josh  wrote:

Hi all,


I just have a couple of questions about checkpointing and restoring
state from RocksDB.


1) In some cases, I find that it is impossible to restore a job from a
checkpoint, due to an exception such as the one pasted below[*]. In
this case, it appears that the last checkpoint is somehow corrupt.
Does anyone know why this might happen?


2) When the above happens, I have no choice but to cancel the job, as
it repeatedly attempts to restart and keeps getting the same
exception. Given that no savepoint was taken recently, is it possible
for me to restore the job from an older checkpoint (e.g. the
second-last checkpoint)?


The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.


Thanks,

Josh


[*]The exception when restoring state:

java.lang.Exception: Could not restore checkpointed state to operators
and functions
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error while restoring RocksDB
state from 
/mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
... 3 more
Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
at 
org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
... 7 more

Re: Handling decompression exceptions

2016-10-11 Thread Flavio Pompermaier

I posted a workaround for that at
https://github.com/okkam-it/flink-examples/blob/master/src/main/java/it/okkam/datalinks/batch/flink/datasourcemanager/importers/Csv2RowExample.java

On 11 Oct 2016 1:57 p.m., "Fabian Hueske"  wrote:

> Hi,
>
> Flink's String parser does not support escaped quotes. You data contains a
> double double quote (""). The parser identifies this as the end of the
> string field.
> As a workaround, you can read the file as a regular text file, line by
> line and do the parsing in a MapFunction.
>
> Best, Fabian
>
> 2016-10-11 13:37 GMT+02:00 Yassine MARZOUGUI :
>
>> Forgot to add parseQuotedStrings('"'). After adding it I'm getting the
>> same exception with the second code too.
>>
>> 2016-10-11 13:29 GMT+02:00 Yassine MARZOUGUI :
>>
>>> Hi Fabian,
>>>
>>> I tried to debug the code, and it turns out a line in my csv data is
>>> causing the ArrayIndexOutOfBoundsException, here is the exception
>>> stacktrace:
>>>
>>> java.lang.ArrayIndexOutOfBoundsException: -1
>>> at org.apache.flink.types.parser.StringParser.parseField(String
>>> Parser.java:49)
>>> at org.apache.flink.types.parser.StringParser.parseField(String
>>> Parser.java:28)
>>> at org.apache.flink.types.parser.FieldParser.resetErrorStateAnd
>>> Parse(FieldParser.java:98)
>>> at org.apache.flink.api.common.io.GenericCsvInputFormat.parseRe
>>> cord(GenericCsvInputFormat.java:395)
>>> at org.apache.flink.api.java.io.CsvInputFormat.readRecord(CsvIn
>>> putFormat.java:110)
>>> at org.apache.flink.api.common.io.DelimitedInputFormat.nextReco
>>> rd(DelimitedInputFormat.java:470)
>>> at org.apache.flink.api.java.io.CsvInputFormat.nextRecord(CsvIn
>>> putFormat.java:78)
>>> at org.myorg.quickstart.MyCsvInputFormat.nextRecord(MyCsvInputF
>>> ormat.java:106)
>>> at org.apache.flink.runtime.operators.DataSourceTask.invoke(Dat
>>> aSourceTask.java:162)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> And here is a sample CSV:
>>>
>>> timestamp,url,id
>>> 2016-08-31 12:08:11.223,"https://www.toyota.fr/hybrid-innovation/infogr
>>> aphie.jsontcgcc, ce)_13h00 /""=/-3h00 %=) 1",000
>>>
>>> Using my code, I get the previous exception when parsing the sample CSV.
>>> If I use the following code, I get an incorrect result : (2016-08-31
>>> 12:08:11.223, ce)_13h00 /""=/-3h00 %=) 1") instead of (2016-08-31
>>> 12:08:11.223, 000)
>>>
>>> DataSet> withReadCSV = 
>>> env.readCsvFile("C:\\Users\\yassine\\Desktop\\test.csv")
>>> .ignoreFirstLine()
>>> .fieldDelimiter(",")
>>> .includeFields("101")
>>> .ignoreInvalidLines()
>>> .types(String.class, String.class);
>>> withReadCSV.writeAsText("C:\\Users\\yassine\\Desktop\\withreadcsv.txt", 
>>> FileSystem.WriteMode.OVERWRITE).setParallelism(1);
>>>
>>>
>>> Is it a bug in Flink or is my data not compliant with the csv standards?
>>>
>>> Thanks,
>>> Yassine
>>>
>>>
>>> 2016-10-11 11:21 GMT+02:00 Fabian Hueske :
>>>
 Hi Yassine,

 I ran your code without problems and got the correct result.
 Can you provide the Stacktrace of the Exception?

 Thanks, Fabian

 2016-10-10 10:57 GMT+02:00 Yassine MARZOUGUI :

> Thank you Fabian and Stephan for the suggestions.
> I couldn't override "readLine()" because it's final, so went with
> Fabian's solution, but I'm struggling with csv field masks. Any help is
> appreciated.
> I created an Input Format which is basically TupleCsvInputFormat for
> which I overrode the nextRecord() method to catch the exceptions. But I'm
> having a *java.lang.ArrayIndexOutOfBoundsException*  when I add a
> boolean[]{true, false, true} field mask . If I add a int[]{1,0,1} field
> mask, the job succeeds but outputs the first and second columns. Here is 
> my
> code:
>
> TupleTypeInfo> typeInfo =
> TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
> Path histPath = new Path("hdfs:///shared/file.csv");
>
> CsvInputFormat > myInputFormt = new
> MyCsvInputFormat<>(histPath, typeInfo, new boolean[]{true, false, true});
> myInputFormt.enableQuotedStringParsing('"');
> myInputFormt.setSkipFirstLineAsHeader(true);
> myInputFormt.setLenient(true);
>
> DataSet> test = env.createInput(myInputFormt,t
> ypeInfo).withParameters(parameters);
> test.writeAsText("E:\\data\\test.csv", FileSystem.WriteMode.OVERWRITE
> );
>
> and here is the  custom input format:
>
> public class MyCsvInputFormat extends CsvInputFormat {
> private static final long serialVersionUID = 1L;
> private TupleSerializerBase tupleSerializer;
> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>

Re: Handling decompression exceptions

2016-10-11 Thread Fabian Hueske

Hi,

Flink's String parser does not support escaped quotes. You data contains a
double double quote (""). The parser identifies this as the end of the
string field.
As a workaround, you can read the file as a regular text file, line by line
and do the parsing in a MapFunction.

Best, Fabian

2016-10-11 13:37 GMT+02:00 Yassine MARZOUGUI :

> Forgot to add parseQuotedStrings('"'). After adding it I'm getting the
> same exception with the second code too.
>
> 2016-10-11 13:29 GMT+02:00 Yassine MARZOUGUI :
>
>> Hi Fabian,
>>
>> I tried to debug the code, and it turns out a line in my csv data is
>> causing the ArrayIndexOutOfBoundsException, here is the exception
>> stacktrace:
>>
>> java.lang.ArrayIndexOutOfBoundsException: -1
>> at org.apache.flink.types.parser.StringParser.parseField(String
>> Parser.java:49)
>> at org.apache.flink.types.parser.StringParser.parseField(String
>> Parser.java:28)
>> at org.apache.flink.types.parser.FieldParser.resetErrorStateAnd
>> Parse(FieldParser.java:98)
>> at org.apache.flink.api.common.io.GenericCsvInputFormat.parseRe
>> cord(GenericCsvInputFormat.java:395)
>> at org.apache.flink.api.java.io.CsvInputFormat.readRecord(CsvIn
>> putFormat.java:110)
>> at org.apache.flink.api.common.io.DelimitedInputFormat.nextReco
>> rd(DelimitedInputFormat.java:470)
>> at org.apache.flink.api.java.io.CsvInputFormat.nextRecord(CsvIn
>> putFormat.java:78)
>> at org.myorg.quickstart.MyCsvInputFormat.nextRecord(MyCsvInputF
>> ormat.java:106)
>> at org.apache.flink.runtime.operators.DataSourceTask.invoke(
>> DataSourceTask.java:162)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> And here is a sample CSV:
>>
>> timestamp,url,id
>> 2016-08-31 12:08:11.223,"https://www.toyota.fr/hybrid-innovation/infogr
>> aphie.jsontcgcc, ce)_13h00 /""=/-3h00 %=) 1",000
>>
>> Using my code, I get the previous exception when parsing the sample CSV.
>> If I use the following code, I get an incorrect result : (2016-08-31
>> 12:08:11.223, ce)_13h00 /""=/-3h00 %=) 1") instead of (2016-08-31
>> 12:08:11.223, 000)
>>
>> DataSet> withReadCSV = 
>> env.readCsvFile("C:\\Users\\yassine\\Desktop\\test.csv")
>> .ignoreFirstLine()
>> .fieldDelimiter(",")
>> .includeFields("101")
>> .ignoreInvalidLines()
>> .types(String.class, String.class);
>> withReadCSV.writeAsText("C:\\Users\\yassine\\Desktop\\withreadcsv.txt", 
>> FileSystem.WriteMode.OVERWRITE).setParallelism(1);
>>
>>
>> Is it a bug in Flink or is my data not compliant with the csv standards?
>>
>> Thanks,
>> Yassine
>>
>>
>> 2016-10-11 11:21 GMT+02:00 Fabian Hueske :
>>
>>> Hi Yassine,
>>>
>>> I ran your code without problems and got the correct result.
>>> Can you provide the Stacktrace of the Exception?
>>>
>>> Thanks, Fabian
>>>
>>> 2016-10-10 10:57 GMT+02:00 Yassine MARZOUGUI 
>>> :
>>>
 Thank you Fabian and Stephan for the suggestions.
 I couldn't override "readLine()" because it's final, so went with
 Fabian's solution, but I'm struggling with csv field masks. Any help is
 appreciated.
 I created an Input Format which is basically TupleCsvInputFormat for
 which I overrode the nextRecord() method to catch the exceptions. But I'm
 having a *java.lang.ArrayIndexOutOfBoundsException*  when I add a
 boolean[]{true, false, true} field mask . If I add a int[]{1,0,1} field
 mask, the job succeeds but outputs the first and second columns. Here is my
 code:

 TupleTypeInfo> typeInfo =
 TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
 Path histPath = new Path("hdfs:///shared/file.csv");

 CsvInputFormat > myInputFormt = new
 MyCsvInputFormat<>(histPath, typeInfo, new boolean[]{true, false, true});
 myInputFormt.enableQuotedStringParsing('"');
 myInputFormt.setSkipFirstLineAsHeader(true);
 myInputFormt.setLenient(true);

 DataSet> test = env.createInput(myInputFormt,t
 ypeInfo).withParameters(parameters);
 test.writeAsText("E:\\data\\test.csv", FileSystem.WriteMode.OVERWRITE);

 and here is the  custom input format:

 public class MyCsvInputFormat extends CsvInputFormat {
 private static final long serialVersionUID = 1L;
 private TupleSerializerBase tupleSerializer;
 public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
 tupleTypeInfo) {
 this(filePath, DEFAULT_LINE_DELIMITER, DEFAULT_FIELD_DELIMITER,
 tupleTypeInfo);
 }
 public MyCsvInputFormat(Path filePath, String lineDelimiter, String
 fieldDelimiter, TupleTypeInfoBase tupleTypeInfo) {
 this(filePath, lineDelimiter, fieldDelimiter, tupleTypeInfo,
 createDefaultMask(tupleTypeInfo.getArity()));
 }

CEP and slightly out of order elements

Hi,

If using CEP with event-time I have events which can be slightly out of
order and I want to sort them by timestamp within their time-windows before
applying CEP-

For example, if using 5 second windows and I use the following

ds2 = ds.keyBy.window(TumblingWindow(10 seconds).apply(/*Sort by
Timestamp*/);

Next assign watermarks again on ds2 (because elements in ds2 will all have
the same timestamp of WINDOW_END_TIME-1ms)
ds2.assignTimestampsAndWatermarks()

Finally apply CEP on ds2 with a WITHIN window of 5 seconds (shorter
timestamp than the one I used earlier).

The reasoning is, if I am using the next() operator in CEP, the events
should be in the order of their timestamps.

Is this the right way to handle this problem? I have heard people say that
assigning watermarks twice can lead to wrong results. But don't I need to
assign timestamps once more in this scenario.

Thanks,
Sameer

Re: About Sliding window

2016-10-11 Thread Kostas Kloudas

Hi Zhangrucong,

Sliding windows only support time-based slide. 
So your use-case is not supported out-of-the-box.

But, if you describe a bit more what you want to do, 
we may be able to find a way together to do your job using 
the currently offered functionality.

Kostas

> On Oct 11, 2016, at 1:20 PM, Zhangrucong  wrote:
> 
> Hello everyone:
>   Now, I am want to use DataStream sliding window API. I look at the API and 
> I have a question, dose the sliding time window support sliding by every 
> incoming event?
>  
> Thanks in advance!

Re: Handling decompression exceptions

2016-10-11 Thread Yassine MARZOUGUI

Forgot to add parseQuotedStrings('"'). After adding it I'm getting the same
exception with the second code too.

2016-10-11 13:29 GMT+02:00 Yassine MARZOUGUI :

> Hi Fabian,
>
> I tried to debug the code, and it turns out a line in my csv data is
> causing the ArrayIndexOutOfBoundsException, here is the exception
> stacktrace:
>
> java.lang.ArrayIndexOutOfBoundsException: -1
> at org.apache.flink.types.parser.StringParser.parseField(
> StringParser.java:49)
> at org.apache.flink.types.parser.StringParser.parseField(
> StringParser.java:28)
> at org.apache.flink.types.parser.FieldParser.resetErrorStateAndParse(
> FieldParser.java:98)
> at org.apache.flink.api.common.io.GenericCsvInputFormat.parseRecord(
> GenericCsvInputFormat.java:395)
> at org.apache.flink.api.java.io.CsvInputFormat.readRecord(
> CsvInputFormat.java:110)
> at org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(
> DelimitedInputFormat.java:470)
> at org.apache.flink.api.java.io.CsvInputFormat.nextRecord(
> CsvInputFormat.java:78)
> at org.myorg.quickstart.MyCsvInputFormat.nextRecord(
> MyCsvInputFormat.java:106)
> at org.apache.flink.runtime.operators.DataSourceTask.
> invoke(DataSourceTask.java:162)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
>
> And here is a sample CSV:
>
> timestamp,url,id
> 2016-08-31 12:08:11.223,"https://www.toyota.fr/hybrid-innovation/
> infographie.jsontcgcc, ce)_13h00 /""=/-3h00 %=) 1",000
>
> Using my code, I get the previous exception when parsing the sample CSV.
> If I use the following code, I get an incorrect result : (2016-08-31
> 12:08:11.223, ce)_13h00 /""=/-3h00 %=) 1") instead of (2016-08-31
> 12:08:11.223, 000)
>
> DataSet> withReadCSV = 
> env.readCsvFile("C:\\Users\\yassine\\Desktop\\test.csv")
> .ignoreFirstLine()
> .fieldDelimiter(",")
> .includeFields("101")
> .ignoreInvalidLines()
> .types(String.class, String.class);
> withReadCSV.writeAsText("C:\\Users\\yassine\\Desktop\\withreadcsv.txt", 
> FileSystem.WriteMode.OVERWRITE).setParallelism(1);
>
>
> Is it a bug in Flink or is my data not compliant with the csv standards?
>
> Thanks,
> Yassine
>
>
> 2016-10-11 11:21 GMT+02:00 Fabian Hueske :
>
>> Hi Yassine,
>>
>> I ran your code without problems and got the correct result.
>> Can you provide the Stacktrace of the Exception?
>>
>> Thanks, Fabian
>>
>> 2016-10-10 10:57 GMT+02:00 Yassine MARZOUGUI :
>>
>>> Thank you Fabian and Stephan for the suggestions.
>>> I couldn't override "readLine()" because it's final, so went with
>>> Fabian's solution, but I'm struggling with csv field masks. Any help is
>>> appreciated.
>>> I created an Input Format which is basically TupleCsvInputFormat for
>>> which I overrode the nextRecord() method to catch the exceptions. But I'm
>>> having a *java.lang.ArrayIndexOutOfBoundsException*  when I add a
>>> boolean[]{true, false, true} field mask . If I add a int[]{1,0,1} field
>>> mask, the job succeeds but outputs the first and second columns. Here is my
>>> code:
>>>
>>> TupleTypeInfo> typeInfo =
>>> TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
>>> Path histPath = new Path("hdfs:///shared/file.csv");
>>>
>>> CsvInputFormat > myInputFormt = new
>>> MyCsvInputFormat<>(histPath, typeInfo, new boolean[]{true, false, true});
>>> myInputFormt.enableQuotedStringParsing('"');
>>> myInputFormt.setSkipFirstLineAsHeader(true);
>>> myInputFormt.setLenient(true);
>>>
>>> DataSet> test = env.createInput(myInputFormt,t
>>> ypeInfo).withParameters(parameters);
>>> test.writeAsText("E:\\data\\test.csv", FileSystem.WriteMode.OVERWRITE);
>>>
>>> and here is the  custom input format:
>>>
>>> public class MyCsvInputFormat extends CsvInputFormat {
>>> private static final long serialVersionUID = 1L;
>>> private TupleSerializerBase tupleSerializer;
>>> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>>> tupleTypeInfo) {
>>> this(filePath, DEFAULT_LINE_DELIMITER, DEFAULT_FIELD_DELIMITER,
>>> tupleTypeInfo);
>>> }
>>> public MyCsvInputFormat(Path filePath, String lineDelimiter, String
>>> fieldDelimiter, TupleTypeInfoBase tupleTypeInfo) {
>>> this(filePath, lineDelimiter, fieldDelimiter, tupleTypeInfo,
>>> createDefaultMask(tupleTypeInfo.getArity()));
>>> }
>>> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>>> tupleTypeInfo, int[] includedFieldsMask) {
>>> this(filePath, DEFAULT_LINE_DELIMITER, DEFAULT_FIELD_DELIMITER,
>>> tupleTypeInfo, includedFieldsMask);
>>> }
>>> public MyCsvInputFormat(Path filePath, String lineDelimiter, String
>>> fieldDelimiter, TupleTypeInfoBase tupleTypeInfo, int[]
>>> includedFieldsMask) {
>>> super(filePath);
>>> boolean[] mask = (includedFieldsMask == null)
>>>

Re: Handling decompression exceptions

2016-10-11 Thread Yassine MARZOUGUI

Hi Fabian,

I tried to debug the code, and it turns out a line in my csv data is
causing the ArrayIndexOutOfBoundsException, here is the exception
stacktrace:

java.lang.ArrayIndexOutOfBoundsException: -1
at
org.apache.flink.types.parser.StringParser.parseField(StringParser.java:49)
at
org.apache.flink.types.parser.StringParser.parseField(StringParser.java:28)
at
org.apache.flink.types.parser.FieldParser.resetErrorStateAndParse(FieldParser.java:98)
at
org.apache.flink.api.common.io.GenericCsvInputFormat.parseRecord(GenericCsvInputFormat.java:395)
at
org.apache.flink.api.java.io.CsvInputFormat.readRecord(CsvInputFormat.java:110)
at
org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(DelimitedInputFormat.java:470)
at
org.apache.flink.api.java.io.CsvInputFormat.nextRecord(CsvInputFormat.java:78)
at
org.myorg.quickstart.MyCsvInputFormat.nextRecord(MyCsvInputFormat.java:106)
at
org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:162)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)

And here is a sample CSV:

timestamp,url,id
2016-08-31 12:08:11.223,"
https://www.toyota.fr/hybrid-innovation/infographie.jsontcgcc, ce)_13h00
/""=/-3h00 %=) 1",000

Using my code, I get the previous exception when parsing the sample CSV. If
I use the following code, I get an incorrect result : (2016-08-31
12:08:11.223, ce)_13h00 /""=/-3h00 %=) 1") instead of (2016-08-31
12:08:11.223, 000)

DataSet> withReadCSV =
env.readCsvFile("C:\\Users\\yassine\\Desktop\\test.csv")
.ignoreFirstLine()
.fieldDelimiter(",")
.includeFields("101")
.ignoreInvalidLines()
.types(String.class, String.class);
withReadCSV.writeAsText("C:\\Users\\yassine\\Desktop\\withreadcsv.txt",
FileSystem.WriteMode.OVERWRITE).setParallelism(1);


Is it a bug in Flink or is my data not compliant with the csv standards?

Thanks,
Yassine


2016-10-11 11:21 GMT+02:00 Fabian Hueske :

> Hi Yassine,
>
> I ran your code without problems and got the correct result.
> Can you provide the Stacktrace of the Exception?
>
> Thanks, Fabian
>
> 2016-10-10 10:57 GMT+02:00 Yassine MARZOUGUI :
>
>> Thank you Fabian and Stephan for the suggestions.
>> I couldn't override "readLine()" because it's final, so went with
>> Fabian's solution, but I'm struggling with csv field masks. Any help is
>> appreciated.
>> I created an Input Format which is basically TupleCsvInputFormat for
>> which I overrode the nextRecord() method to catch the exceptions. But I'm
>> having a *java.lang.ArrayIndexOutOfBoundsException*  when I add a
>> boolean[]{true, false, true} field mask . If I add a int[]{1,0,1} field
>> mask, the job succeeds but outputs the first and second columns. Here is my
>> code:
>>
>> TupleTypeInfo> typeInfo =
>> TupleTypeInfo.getBasicTupleTypeInfo(String.class, String.class);
>> Path histPath = new Path("hdfs:///shared/file.csv");
>>
>> CsvInputFormat > myInputFormt = new
>> MyCsvInputFormat<>(histPath, typeInfo, new boolean[]{true, false, true});
>> myInputFormt.enableQuotedStringParsing('"');
>> myInputFormt.setSkipFirstLineAsHeader(true);
>> myInputFormt.setLenient(true);
>>
>> DataSet> test = env.createInput(myInputFormt,t
>> ypeInfo).withParameters(parameters);
>> test.writeAsText("E:\\data\\test.csv", FileSystem.WriteMode.OVERWRITE);
>>
>> and here is the  custom input format:
>>
>> public class MyCsvInputFormat extends CsvInputFormat {
>> private static final long serialVersionUID = 1L;
>> private TupleSerializerBase tupleSerializer;
>> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>> tupleTypeInfo) {
>> this(filePath, DEFAULT_LINE_DELIMITER, DEFAULT_FIELD_DELIMITER,
>> tupleTypeInfo);
>> }
>> public MyCsvInputFormat(Path filePath, String lineDelimiter, String
>> fieldDelimiter, TupleTypeInfoBase tupleTypeInfo) {
>> this(filePath, lineDelimiter, fieldDelimiter, tupleTypeInfo,
>> createDefaultMask(tupleTypeInfo.getArity()));
>> }
>> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>> tupleTypeInfo, int[] includedFieldsMask) {
>> this(filePath, DEFAULT_LINE_DELIMITER, DEFAULT_FIELD_DELIMITER,
>> tupleTypeInfo, includedFieldsMask);
>> }
>> public MyCsvInputFormat(Path filePath, String lineDelimiter, String
>> fieldDelimiter, TupleTypeInfoBase tupleTypeInfo, int[]
>> includedFieldsMask) {
>> super(filePath);
>> boolean[] mask = (includedFieldsMask == null)
>> ? createDefaultMask(tupleTypeInfo.getArity())
>> : toBooleanMask(includedFieldsMask);
>> configure(lineDelimiter, fieldDelimiter, tupleTypeInfo, mask);
>> }
>> public MyCsvInputFormat(Path filePath, TupleTypeInfoBase
>> tupleTypeInfo, boolean[] includedFieldsMask) {
>> this(filePath, DEFAULT_LINE_DELIMITER,

About Sliding window

2016-10-11 Thread Zhangrucong

Hello everyone:
  Now, I am want to use DataStream sliding window API. I look at the API and I 
have a question, dose the sliding time window support sliding by every incoming 
event?

Thanks in advance!

Exception when restoring state from RocksDB - how to recover?

Hi all,


I just have a couple of questions about checkpointing and restoring
state from RocksDB.


1) In some cases, I find that it is impossible to restore a job from a
checkpoint, due to an exception such as the one pasted below[*]. In
this case, it appears that the last checkpoint is somehow corrupt.
Does anyone know why this might happen?


2) When the above happens, I have no choice but to cancel the job, as
it repeatedly attempts to restart and keeps getting the same
exception. Given that no savepoint was taken recently, is it possible
for me to restore the job from an older checkpoint (e.g. the
second-last checkpoint)?


The version of Flink I'm using Flink-1.1-SNAPSHOT, from mid-June.


Thanks,

Josh


[*]The exception when restoring state:

java.lang.Exception: Could not restore checkpointed state to operators
and functions
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:480)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:219)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error while restoring RocksDB
state from 
/mnt/yarn/usercache/hadoop/appcache/application_1476181294189_0001/flink-io-09ad1cb1-8dff-4f9a-9f61-6cae27ee6f1d/d236820a793043bd63360df6f175cae9/StreamFlatMap_9_8/dummy_state/dc5beab1-68fb-48b3-b3d6-272497d15a09/chk-1
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:537)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.injectKeyValueStateSnapshots(RocksDBStateBackend.java:489)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.restoreState(AbstractStreamOperator.java:204)
at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.restoreState(AbstractUdfStreamOperator.java:154)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreState(StreamTask.java:472)
... 3 more
Caused by: org.rocksdb.RocksDBException: NotFound: Backup not found
at org.rocksdb.BackupEngine.restoreDbFromLatestBackup(Native Method)
at 
org.rocksdb.BackupEngine.restoreDbFromLatestBackup(BackupEngine.java:177)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.restoreFromSemiAsyncSnapshot(RocksDBStateBackend.java:535)
... 7 more

Re: Listening to timed-out patterns in Flink CEP

Hi David,

the problem is still that there is no corresponding watermark saying that 4
seconds have now passed. With your code, watermarks will be periodically
emitted but the same watermark will be emitted until a new element arrives
which will reset the watermark. Thus, the system can never know until this
watermark is seen whether there will be an earlier event or not. I fear
that this is a fundamental problem with stream processing.

You're right that the negation operator won't solve the problem. It will
indeed suffer from the same problem.

Cheers,
Till

On Sun, Oct 9, 2016 at 7:37 PM,  wrote:

> >>FLINK-3320  (CEP
> "not" operator) does not address this because again, how would the "not
> match" be triggered if no event at all occurs?
>
> Good question.
>
> I'm not sure whether the following will work:
>
> This could be done by creating a CEP matching pattern that uses both of
> "notNext" (or "notFollowedBy") and "within" constructs. Something like this:
>
> Pattern pattern = Pattern.begin("first")
> .notNext("second")
> .within(Time.seconds(3));
>
> I'm hoping Flink CEP experts (Till?) will comment on this.
>
> Note: I have requested these negation patterns to be implemented in Flink
> CEP, but notNext/notFollowedBy are not yet implemented in Flink..
>
>
> - LF
>
>
>
>
> --
> *From:* David Koch 
> *To:* user@flink.apache.org; lg...@yahoo.com
> *Sent:* Sunday, October 9, 2016 5:51 AM
>
> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>
> Hello,
>
> Thank you for the explanation as well as the link to the other post.
> Interesting to learn about some of the open JIRAs.
>
> Indeed, I was not using event time, but processing time. However, even
> when using event time I only get notified of timeouts upon subsequent
> events.
>
> The link  contains an example where I read
>   from a socket, wrap this in a custom "event" with timestamp,
> key the resultant stream by  and attempt to detect  instances no
> further than 3 seconds apart using CEP.
>
> Apart from the fact that results are only printed when I close the socket
> (normal?) I don't observe any change in behaviour
>
> So event-time/watermarks or not: SOME event has to occur for the timeout
> to be triggered.
>
> FLINK-3320  (CEP "not"
> operator) does not address this because again, how would the "not match" be
> triggered if no event at all occurs?
>
> On Sat, Oct 8, 2016 at 12:50 AM,  wrote:
>
> The following is a better link:
>
> http://mail-archives.apache. org/mod_mbox/flink-user/
> 201609.mbox/%3CCAC27z% 3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g%
> 40mail.gmail.com%3E
> 
>
>
> - LF
>
>
>
>
> --
> *From:* "lg...@yahoo.com" 
> *To:* "user@flink.apache.org" 
> *Sent:* Friday, October 7, 2016 3:36 PM
>
> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>
> Isn't the upcoming CEP negation (absence of an event) feature solve this
> issue?
>
> See this discussion thread:
> http://mail-archives.apache. org/mod_mbox/flink-user/
> 201609.mbox/%3CCAC27z%3DOD% 2BTq8twBw_ 1YKni5sWAU3g1S9WDpJw0DUwgiG9YX
> 9Fg%40mail.gmail.com%3E
> 
>
>
>
> //  Atul
>
>
> --
> *From:* Till Rohrmann 
> *To:* user@flink.apache.org
> *Sent:* Friday, October 7, 2016 12:58 AM
> *Subject:* Re: Listening to timed-out patterns in Flink CEP
>
> Hi David,
>
> in case of event time, the timeout will be detected when the first
> watermark exceeding the timeout value is received. Thus, it depends a
> little bit how you generate watermarks (e.g. periodically, watermark per
> event).
>
> In case of processing time, the time is only updated whenever a new
> element arrives. Thus, if you have an element arriving 4 seconds after
> Event A, it should detect the timeout. If the next event arrives 20 seconds
> later, than you won't see the timeout until then.
>
> In the case of processing time, we could think about registering timeout
> timers for processing time. However, I would highly recommend you to use
> event time, because with processing time, Flink cannot guarantee meaningful
> computations, because the events might arrive out of order.
>
> Cheers,
> Till
>
> On Thu, Oct 6, 2016 at 3:08 PM, David Koch  wrote:
>
> Hello,
>
> With Flink CEP, is there a way to actively listen to pattern matches that
> time out? I am under the impression that this is not possible.
>
> In my case I partition a stream containing

Re: jdbc.JDBCInputFormat

2016-10-11 Thread Alberto Ramón

I will check it this nigth

Thanks

2016-10-11 11:24 GMT+02:00 Timo Walther :

> I have opened a PR (https://github.com/apache/flink/pull/2619). Would be
> great if you could try it and comment if it solves you problem.
>
> Timo
>
> Am 10/10/16 um 17:48 schrieb Timo Walther:
>
> I could reproduce the error locally. I will prepare a fix for it.
>
> Timo
>
> Am 10/10/16 um 11:54 schrieb Alberto Ramón:
>
> It's from Jun and Unassigned   :(
> Is There a Workarround?
>
> I'm will try to contact with the reporter , Martin Scholl )
>
> 2016-10-10 11:04 GMT+02:00 Timo Walther :
>
>> I think you already found the correct issue describing your problem (
>> FLINK-4108). This should get higher priority.
>>
>> Timo
>>
>> Am 09/10/16 um 13:27 schrieb Alberto Ramón:
>>
>>
>> After solved some issues, I connected with Kylin, but I can't read data
>>
>> import org.apache.flink.api.scala._import 
>> org.apache.flink.api.java.io.jdbc.JDBCInputFormatimport 
>> org.apache.flink.api.table.Rowimport 
>> org.apache.flink.api.table.typeutils.RowTypeInfoimport 
>> org.apache.flink.api.common.typeinfo.{BasicTypeInfo, TypeInformation}
>>
>> var stringColum: TypeInformation[Int] = createTypeInformation[Int]val 
>> DB_ROWTYPE = new RowTypeInfo(Seq(stringColum))
>> val inputFormat = JDBCInputFormat.buildJDBCInputFormat()
>>   .setDrivername("org.apache.kylin.jdbc.Driver")
>>   .setDBUrl("jdbc:kylin://172.17.0.2:7070/learn_kylin")
>>   .setUsername("ADMIN")
>>   .setPassword("KYLIN")
>>   .setQuery("select count(distinct seller_id) as sellers from kylin_sales 
>> group by part_dt order by part_dt")
>>   .setRowTypeInfo(DB_ROWTYPE)
>>   .finish()
>>
>>   val dataset =env.createInput(inputFormat)
>> dataset.print()
>>
>>
>> The error is:[image: Imágenes integradas 1]
>>
>>
>> (I checked that queries and  config are correct with SQuirriel)
>>
>> (Isn't a connection problem, Because if I turn off database the error is 
>> different "Reused Connection")
>>
>>
>>
>> Can you see a problem in my code? (I found  Flink 4108 unsolved issue,I 
>> don't know if is related)
>>
>>
>> BR, Alberto
>>
>> 2016-10-07 21:46 GMT+02:00 Fabian Hueske :
>>>
>>> As the exception says the class 
>>> org.apache.flink.api.scala.io.jdbc.JDBCInputFormat
>>> does not exist. You have to do: import 
>>> org.apache.flink.api.java.io.jdbc.JDBCInputFormat
>>>
>>> There is no Scala implementation of this class but you can also use Java
>>> classes in Scala.
>>> 2016-10-07 21:38 GMT+02:00 Alberto Ramón :

 I want use CreateInput + buildJDBCInputFormat to acces to database on
 SCALA
 PB1:

 import org.apache.flink.api.scala.io.jdbc.JDBCInputFormatError:(25, 37) 
 object jdbc is not a member of package org.apache.flink.api.java.io
 import org.apache.flink.api.java.io.jdbc.JDBCInputFormat

 Then, I can't use:[image: Imágenes integradas 1]

 I tried to download code from git and recompile, also

 --
>> Freundliche Grüße / Kind Regards
>>
>> Timo Walther
>>
>> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>>
>> --
> Freundliche Grüße / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>
> --
> Freundliche Grüße / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>
>

Re: Create stream from multiple files using "env.readFile(format, input1, FileProcessingMode.PROCESS_CONTINUOUSLY, 1000, FilePathFilter.createDefaultFilter())" ?

2016-10-11 Thread Aljoscha Krettek

Hi,
how does "doesn't work" manifest?

Cheers,
Aljoscha

On Wed, 28 Sep 2016 at 22:54 Anchit Jatana 
wrote:

> Hi All,
>
> I have a use case where in need to create multiple source streams from
> multiple files and monitor the files for any changes using the "
> FileProcessingMode.PROCESS_CONTINUOUSLY"
>
> Intention is to achieve something like this(have a monitored stream for
> each of the multiple files), something like this:
>
> DataStream stream1 = env.readFile(format, input1,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> DataStream stream2 = env.readFile(format, input2,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> DataStream stream3= env.readFile(format, input3,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> .
>
> .
>
> .
>
> .
>
> DataStream streamN = env.readFile(format, inputN,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> Since, this implementation doesn't work, can someone suggest a way how
> this thing can be achieved?
>
>
> PS: Main intention is to '*monitor'* all the files and stream the updated
> content if any change has been made to it.
>
>
> Thank you!
>
> Regards,
>
> Anchit
>

Re: more complex patterns for CEP (was: CEP two transitions to the same state)