Ynt: Flink 1.2-SNAPSHOT Bucketing Sink problem
Hi Fabian, Thank you for quick response. We will check the versions of the flink in cluster and maven. By the way, the BucketingSink function is really good at sinking to hdfs (Especially for event time file naming). However, currently in version 1.1.3, the RollingSink function is used. Is there any plan to release the BucketingSink function in stable version? Gönderen: Fabian Hueske <fhue...@gmail.com> Gönderildi: 26 Ekim 2016 Çarşamba 20:38:56 Kime: dev@flink.apache.org Konu: Re: Flink 1.2-SNAPSHOT Bucketing Sink problem Hi Ozan, a NoSuchMethodError indicates a version mismatch. Since you are using a SNAPSHOT build it is likely that dependencies changed when you recompiled Flink or your job. You should make sure that you are using the same version (on SNAPSHOT the same commit) for jobs and cluster. Best, Fabian 2016-10-26 18:30 GMT+02:00 Ozan DENİZ <ozande...@outlook.com>: > Hi everyone, > > > We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for > sink operation. The BucketingSink function was working correctly. However > today, when I put the job on cluster I got this error; > > > java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators. > StreamingRuntimeContext.getProcessingTimeService() > Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService; > at org.apache.flink.streaming.connectors.fs.bucketing. > BucketingSink.open(BucketingSink.java:327) > at org.apache.flink.api.common.functions.util.FunctionUtils. > openFunction(FunctionUtils.java:36) > at org.apache.flink.streaming.api.operators. > AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > openAllOperators(StreamTask.java:372) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > invoke(StreamTask.java:261) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609) > at java.lang.Thread.run(Thread.java:745) > > > Is there any way to solve this problem? > > > (The rolling sink function is working based on the processing time and we > need event time for hdfs. Therefore we have choosen bucketing sink > function.) > > > Thank you for your helps. > > > Ozan. >
Flink 1.2-SNAPSHOT Bucketing Sink problem
Hi everyone, We are currently using BucketingSink (Flink 1.2-SNAPSHOT) function for sink operation. The BucketingSink function was working correctly. However today, when I put the job on cluster I got this error; java.lang.NoSuchMethodError: org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getProcessingTimeService()Lorg/apache/flink/streaming/runtime/tasks/ProcessingTimeService; at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.open(BucketingSink.java:327) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:372) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:261) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609) at java.lang.Thread.run(Thread.java:745) Is there any way to solve this problem? (The rolling sink function is working based on the processing time and we need event time for hdfs. Therefore we have choosen bucketing sink function.) Thank you for your helps. Ozan.
Ynt: No support for request PutMappingRequest
Hi Aljoscha,
We are trying to add sortable feature for elasticsearch. To do this, we need to
add mapping to index.
We try to sort some fields in elasticsearch. To make it our json format should
like this;
"tweet": {
"type": "string",
"analyzer": "english",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
We should add "not_analyzed" to map.
private XContentBuilder buildMapping( String typeName )
{
XContentBuilder mapping = null;
try
{
mapping = jsonBuilder()
.startObject()
.startObject(typeName)
.startObject("properties")
.startObject(LogFields.MESSAGE)
.field("type","string")
.field("analyzer", "standard")
.startObject("fields")
.startObject("raw")
.field("type","string")
.field("index", "not_analyzed")
.endObject()
.endObject()
.endObject()
.endObject()
.endObject()
.endObject();
}
catch ( IOException e )
{
e.printStackTrace();
}
return mapping;
}
Gönderen: Aljoscha Krettek <aljos...@apache.org>
Gönderildi: 26 Eylül 2016 Pazartesi 16:27:12
Kime: dev@flink.apache.org
Konu: Re: No support for request PutMappingRequest
Hi,
I think PutMappingRequest is a request that can only be sent using
IndicesAdminClient. In my understanding this is an administrative command
that isn't related to actually storing data in an index.
What are you trying to store with the PutMappingRequest?
Cheers,
Aljoscha
On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <ozande...@outlook.com> wrote:
> Hi,
>
>
> I am sending the error message below;
>
>
> java.lang.IllegalArgumentException: No support for request
> [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b
> ]
> at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at sink.ESSink.process(ESSink.java:77) ~[classes/:?]
> at sink.ESSink.process(ESSink.java:25) ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> [flink-runtime_2.10-1.1.1.jar:1.1.1]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED
>
>
> We try to add mapping request to elastic search. We cannot access to
> client attribute (it is private) in elasticsearch class.
>
>
> Is there any way to overcome this problem.
>
>
> Thanks,
>
>
> Ozan
>
>
>
>
> Gönderen: Till Rohrmann <trohrm...@apache.org>
> Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34
> Kime: dev@flink.apache.org
> Bilgi: Aljoscha Krettek
> Konu: Re: No support for request PutMappingRequest
>
> Hi Ozan,
>
> I'm not super experienced with Flink's elasticsearch connector, but could
> you post the complete stack trace to figure out where the problem comes
> from?
>
> I've also pulled in Aljoscha, the original author of the elas
Ynt: No support for request PutMappingRequest
Hi, I am sending the error message below; java.lang.IllegalArgumentException: No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b] at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) ~[elasticsearch-2.3.5.jar:2.3.5] at org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] at sink.ESSink.process(ESSink.java:77) ~[classes/:?] at sink.ESSink.process(ESSink.java:25) ~[classes/:?] at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) [flink-runtime_2.10-1.1.1.jar:1.1.1] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED We try to add mapping request to elastic search. We cannot access to client attribute (it is private) in elasticsearch class. Is there any way to overcome this problem. Thanks, Ozan Gönderen: Till Rohrmann <trohrm...@apache.org> Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34 Kime: dev@flink.apache.org Bilgi: Aljoscha Krettek Konu: Re: No support for request PutMappingRequest Hi Ozan, I'm not super experienced with Flink's elasticsearch connector, but could you post the complete stack trace to figure out where the problem comes from? I've also pulled in Aljoscha, the original author of the elasticsearch sink. Maybe he can give you a detailed answer. Cheers, Till On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <ozande...@outlook.com> wrote: > Hi everyone, > > > We are trying to use elasticsearch (2.x) connector for Flink application. > However, we encounter a problem when we try to add mapping to elasticsearch > index. > > > The error message is below when we run the Flink application. > > > No support for request [org.elasticsearch.action. > admin.indices.mapping.put.PutMappingRequest] > > > We are using "putMappingRequest" function for mapping. In Flink, is there > any way to add mapping to indexes in elasticsearch? > > > If not, can we contribute for adding mapping feature to elasticsearch > connector? >
No support for request PutMappingRequest
Hi everyone, We are trying to use elasticsearch (2.x) connector for Flink application. However, we encounter a problem when we try to add mapping to elasticsearch index. The error message is below when we run the Flink application. No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest] We are using "putMappingRequest" function for mapping. In Flink, is there any way to add mapping to indexes in elasticsearch? If not, can we contribute for adding mapping feature to elasticsearch connector?
RE: Apache Flink Web Monitor Api
Hi Ufuk, Thank you for your response. We've seen the cancel and run operations in code.However, it would be very nice to add restart operation. We're now implementing custom api to restart flink app. > From: u...@apache.org > Date: Mon, 6 Jun 2016 15:57:48 +0200 > Subject: Re: Apache Flink Web Monitor Api > To: dev@flink.apache.org > > Hey Ozan! For cancel and submit, yes: > > - cancel: /jobs/:jobid/cancel > - submit: /jars/upload and /jars/:jarid/run > > You can look into WebRuntimeMonitor class for more details about the > submission. Cancellation should be straight forward. > > Restart is currently not supported via the REST API and I'm not aware > of any plans to add this. > > – Ufuk > > > On Mon, Jun 6, 2016 at 9:57 AM, Ozan DENİZ <ozande...@outlook.com> wrote: > > Hi everyone, > > We need to implement api functions to cancel, restart and start the flink > > job. > > Is there any way to add these functions to WebMonitor api?Is there any plan > > to implement these functions to WebMonitor api? > > Thanks. > > Ozan
Apache Flink Web Monitor Api
Hi everyone, We need to implement api functions to cancel, restart and start the flink job. Is there any way to add these functions to WebMonitor api?Is there any plan to implement these functions to WebMonitor api? Thanks. Ozan
RE: Savepoint for time windows
Hi Stephan and Ufuk, Thank you for your reply. I have assigned uid to the "assignTimestampsAndWatermarks", "addSource", "apply" operators. However, I couldn't assign uid to the time window. Therefore the time window doesn't hold any state regarding timestamp. For example, I implemented a custom window trigger. Trigger condition: There must be 4 logs or 1 day (By the way, we are using event time.) to trigger. After I send 3 logs, I snapshot the running application, I canceled the job then I changed the topology. To change the topology, I've just split data stream into two separate data streams. I re-run the application with the new topology by using snapshot. Then I didn't send any log to the new topology. In this case window shouldn't trigger and shouldn't call apply function. But when I checked the output file, I saw that window has been triggered just after I re-run the application with new topology. I think it flushes the old window. Is there any way to hold old window state and continue with coming log? > Date: Mon, 18 Apr 2016 18:04:50 +0200 > Subject: Re: Savepoint for time windows > From: se...@apache.org > To: dev@flink.apache.org > > Hi! > > Yes, window contents is part of savepoints. If you change the topology, it > is crucial that the new topology matches the old window contents to the new > operator. > > If you change the structure of the program, you probably need to assign > persistent names to the operators. See > https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/savepoints.html#changes-to-your-program > > Stephan > > > On Mon, Apr 18, 2016 at 5:43 PM, Ufuk Celebi <u...@apache.org> wrote: > > > Can you please share the program before and after the savepoint? > > > > – Ufuk > > > > On Mon, Apr 18, 2016 at 3:11 PM, Ozan DENİZ <ozande...@outlook.com> wrote: > > > Hi everyone, > > > > > > I am trying to implement savepoint mechanism for my Flink project. > > > > > > Here is the scenario: > > > > > > I got the snapshot of Flink application by using "flink savepoint > > " command while the application is running. > > > > > > After saving snapshot of application, I canceled the job from web ui > > than I changed the topology of Flink application. > > > (To change the topology, I split the keyedstream into two seperate > > keyedstream) > > > > > > After changing the topology, I run the new application by using the > > snapshot which I took in first step. > > > > > > But after running the application, the window which has been snapshot, > > triggers without the new log. > > > > > > Question is: Is there any way to save old window state to continue after > > starting new topology with snapshot? > > > > >
Savepoint for time windows
Hi everyone, I am trying to implement savepoint mechanism for my Flink project. Here is the scenario: I got the snapshot of Flink application by using "flink savepoint " command while the application is running. After saving snapshot of application, I canceled the job from web ui than I changed the topology of Flink application. (To change the topology, I split the keyedstream into two seperate keyedstream) After changing the topology, I run the new application by using the snapshot which I took in first step. But after running the application, the window which has been snapshot, triggers without the new log. Question is: Is there any way to save old window state to continue after starting new topology with snapshot?
Apache Flink: aligning watermark among parallel tasks
We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App. During the streaming process, all the 4 tasks' watermark values must be close to trigger window event. For example; Task 1 watermark value = 8 Task 2 watermark value = 1 Task 3 watermark value = 8 Task 4 watermark value = 8 Task 2 is waiting for log to update its watermark. However, the condition can occur before Task 2's update and we want to fire the window event before it. Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?
Apache Flink: aligning watermark among parallel tasks
We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App. During the streaming process, all the 4 tasks' watermark values must be close to trigger window event. For example; Task 1 watermark value = 8 Task 2 watermark value = 1 Task 3 watermark value = 8 Task 4 watermark value = 8 Task 2 is waiting for log to update its watermark. However, the condition can occur before Task 2's update and we want to fire the window event before it. Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?
