FYI in case it’s relevant for this discussion > I'm not sure what is the ` Avro JSON` means
Avro supports two encoding mechanisms – binary encoding, and JSON encoding. [1] In other words, an Avro record, although normally represented as binary data, can instead be represented as a JSON document. I contributed an improvement to the Flink Avro connector to add support for JSON-encoded Avro as part of FLINK-33058 [2], although this is not yet merged. I must admit, I’m not familiar with OpenSearch or what the Flink OpenSearch connector offers, so my apologies if I’m just adding noise to the conversation here! Kind regards Dale [1] - https://avro.apache.org/docs/1.11.1/specification/#encodings [2] - https://issues.apache.org/jira/browse/FLINK-33058 From: Xuyang <xyzhong...@163.com> Date: Tuesday, 21 November 2023 at 15:17 To: Praveen Chandna <praveen.chan...@ericsson.com> Cc: user@flink.apache.org <user@flink.apache.org> Subject: [EXTERNAL] Re:Flink OpenSearch Connector - Avro JSON to JSON Hi, Praveen. `OpenSearch supports writing data in the JSON format, but in Flink its default data format is Avro JSON. ` I'm not sure what is the ` Avro JSON` means. In Opensearch connector, there are multi formats it supports sucn as "avro", ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2Q-hCD4fShO6MOZ7kqFFTQcREK765Iajk9p4JulKED1SoHa3HN_0airYOK74p2NVaOJi29ksZPjde_iw7kIq1eSKh9l8HkqhLgo2XShwug$> ZjQcmQRYFpfptBannerEnd Hi, Praveen. `OpenSearch supports writing data in the JSON format, but in Flink its default data format is Avro JSON.` I'm not sure what is the ` Avro JSON` means. In Opensearch connector, there are multi formats it supports sucn as "avro", "format" and etc by adding corresponding jars, and the default format[1] is "json" you want. `OpenSearch Java Rest Client support writing the Java object directly to the openSearch using the below code.` Currently, the Opensearch connector connector always write the data serialized[2]. You can modify here directly in Opensearch connector, re-package it and try to use it. [1] https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/OpensearchConnectorOptions.java#L136<https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/OpensearchConnectorOptions.java#L136> [2] https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/RowOpensearchEmitter.java#L110C32-L110C45<https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/RowOpensearchEmitter.java#L110C32-L110C45> -- Best！ Xuyang At 2023-11-21 13:32:35, "Praveen Chandna via user" <user@flink.apache.org> wrote: Hello Team Please helps to answer the below query. 1. OpenSearch supports writing data in the JSON format, but in Flink its default data format is Avro JSON. What is the best practice to write data to Open Search using Flink OpenSearch Connector? Do we need to manually convert Avro JSON to JSON format or is there any in-built support in Flink to write JSON to openSearch. 2. How can I write the existing complex nested JSON data into OpenSearch using the Flink connector rather than writing each key-value(or filed/value). 3. OpenSearch Java Rest Client support writing the Java object directly to the openSearch using the below code. IndexRequest<UserDataObject> indexRequest =new IndexRequest.Builder<UserDataObject>().index(index).document(indexData).build(); https://opensearch.org/docs/latest/clients/java/<https://opensearch.org/docs/latest/clients/java/> Thanks !! // Regards Praveen Chandna Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
