[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[
https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514715#comment-17514715
]
Martijn Visser commented on FLINK-26281:
{{Remove deprecated 'type', explain exactly-once semantic.}}
Fixed in master: 9a3524008c4a2d955ccafb6e2ece39db37c2e3df
Fixed in release-1.15: ba63354de3d9a8837bbfb23cb87cf6c4af33e735
> Test Elasticsearch connector End2End
>
>
> Key: FLINK-26281
> URL: https://issues.apache.org/jira/browse/FLINK-26281
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ElasticSearch
>Affects Versions: 1.15.0
>Reporter: Alexander Preuss
>Assignee: Konstantin Knauf
>Priority: Critical
> Labels: pull-request-available, release-testing
> Fix For: 1.15.0
>
>
> Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323
> Documentation for [datastream
> api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/]
> Documentation for [table
> api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/]
> As 1.15 deprecated the SinkFunction-based Elasticsearch connector and
> introduces the new connector based on the Sink interface we should test it
> behaves correctly and as the user expects.
>
> Some suggestions what to test:
> * Test delivery guarantees (none, at-least-once) (exactly-once should not
> run)
> * Write a simple job that is inserting/upserting data into Elasticsearch
> * Write a simple job that is inserting/upserting data into Elasticsearch and
> use a non-default parallelism
> * Write a simple job in both datastream api and table api
> * Test restarting jobs and scaling up/down
> * Test failure of a simple job that is inserting data with exactly-once
> delivery guarantee by terminating and restarting Elasticsearch
> * Test against Elasticsearch 6.X and 7.X with the respective connectors
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511052#comment-17511052 ] Yangze Guo commented on FLINK-26281: master: 4e5a34d35e45154a85ad893cfbacf7d7202db411 > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Critical > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511037#comment-17511037 ] Yun Gao commented on FLINK-26281: - I'll first change the priority to critical since all issues related to 1.15 should have been fixed. > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511028#comment-17511028 ] Yangze Guo commented on FLINK-26281: 1.15: d480fec487ce2b3d86338b9a09e88bfcfccd49e2 > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510505#comment-17510505 ] Alexander Preuss commented on FLINK-26281: -- [~guoyangze] I just checked and saw what you meant, good catch thanks! I just opened PRs for removing it against master and 1.15 [https://github.com/apache/flink/pull/19202] [https://github.com/apache/flink/pull/19203] > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510489#comment-17510489 ] Alexander Preuss commented on FLINK-26281: -- [~knaufk] Thank you very much for the testing! I opened a [PR|https://github.com/apache/flink/pull/19200] for the docs changes but unfortunately I am not at all familiar with ES datastreams. Maybe you or Kostas can add the information for that bullet point there? > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510401#comment-17510401 ] Yangze Guo commented on FLINK-26281: [~alexanderpreuss] I know it is an old option. Maybe it is out of the scope of this PR. But it seems it is not used anywhere in the latest codebase. If so, we can remove it :) > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510376#comment-17510376 ] Alexander Preuss commented on FLINK-26281: -- Hi [~guoyangze], yes you can use all of the old connector options as we reverted the table API part to how it was previously. For datastream you can set connection request timeout, connection timeout and socket timeout instead :) > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510372#comment-17510372 ] Yangze Guo commented on FLINK-26281: [~fpaul] Thanks, and very much looking forward to the new es connector and general failure handler! Maybe one question about the document. Does the `connection.max-retry-timeout` still take effect anywhere? > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510356#comment-17510356 ] Konstantin Knauf commented on FLINK-26281: -- I've done a couple of tests with ES7, none with ES6: DataStream & Table API, NONE/AT_LEAST_ONCE, Elastic Search Index, Elastic Search Data Stream [1]. Overall, no blockers. Here are some suggestions for improvements to the documentation: * explain how end-to-end exactly-once can be achieved with the ES connector (AT_LEAST_ONCE + upserts based on a deterministic id). In my understanding it is not recommended anymore in Elastic to set the id manually, so this might be worth highlighting. * remove the `type` from the IndexRequest, because it is deprected * Add documentation for usage of es datastreams (e.g. requires a @timestamp field, only OpType.CREATE, what it means for end-to-end delivery guarantees) [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510327#comment-17510327 ] Fabian Paul commented on FLINK-26281: - [~guoyangze] I think we fixed now all issues. The failure handler is reintroduced in the Table API and the delivery guarantee defaults to at-least once. > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509778#comment-17509778 ] Fabian Paul commented on FLINK-26281: - Merged in release-1.15: e893b700eaf99ef8b278cfa75478098b7674b762 > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17507616#comment-17507616 ] Fabian Paul commented on FLINK-26281: - Merged in master: 7c4c470ec3e8ff8e06b7298f0f0a9952e3f90248 > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504154#comment-17504154 ] Yangze Guo commented on FLINK-26281: [~alexanderpreuss] Most of them use the sink in Table API. AFAIK, some of them implement their custom `ActionRequestFailureHandler`, the others use the provided ones. > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504138#comment-17504138 ] Alexander Preuss commented on FLINK-26281: -- Hi [~guoyangze], for the failure handler, can you confirm wether they are using the sink in datastream or table API? Are they implementing a custom ActionRequestFailureHandler or using the provided ones (IngoringFailure, NoOp, RetryRejectedExecution)? > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: pull-request-available, release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502173#comment-17502173 ] Alexander Preuss commented on FLINK-26281: -- [~knaufk] you can also try killing the taskmanager :) > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502106#comment-17502106 ] Yangze Guo commented on FLINK-26281: I found two behavior changes atm: - In 1.14, we turn on the `flush-on-checkpoint` by default. While in 1.15, the default value of `sink.delivery-guarantee` is `NONE`. - We remove the failure handling mechanism(instead of deprecation), which AFAIK some of our users need it. > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26281) Test Elasticsearch connector End2End
[ https://issues.apache.org/jira/browse/FLINK-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501330#comment-17501330 ] Konstantin Knauf commented on FLINK-26281: -- [~alexanderpreuss] Is it important that the failure for exactly-once delivery guarantees is introduced via terminating ElasticSearch or could it also be introduced by killing a Taskmanager? > Test Elasticsearch connector End2End > > > Key: FLINK-26281 > URL: https://issues.apache.org/jira/browse/FLINK-26281 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.15.0 >Reporter: Alexander Preuss >Assignee: Konstantin Knauf >Priority: Blocker > Labels: release-testing > Fix For: 1.15.0 > > > Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323 > Documentation for [datastream > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/] > Documentation for [table > api|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/] > As 1.15 deprecated the SinkFunction-based Elasticsearch connector and > introduces the new connector based on the Sink interface we should test it > behaves correctly and as the user expects. > > Some suggestions what to test: > * Test delivery guarantees (none, at-least-once) (exactly-once should not > run) > * Write a simple job that is inserting/upserting data into Elasticsearch > * Write a simple job that is inserting/upserting data into Elasticsearch and > use a non-default parallelism > * Write a simple job in both datastream api and table api > * Test restarting jobs and scaling up/down > * Test failure of a simple job that is inserting data with exactly-once > delivery guarantee by terminating and restarting Elasticsearch > * Test against Elasticsearch 6.X and 7.X with the respective connectors -- This message was sent by Atlassian Jira (v8.20.1#820001)
