[
https://issues.apache.org/jira/browse/FLINK-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468056#comment-17468056
]
Xue Wang commented on FLINK-25021:
----------------------------------
Hi [~fpaul], happy new year!
One potential scenario would be to use Flink as a streaming ETL solution and
then ingest results into ADX for final time-series or user behavior analytics,
supporting both interactive queries and data visualizations at low latency.
Some common scenarios as depicted in [ADX
solutions|https://docs.microsoft.com/en-us/azure/data-explorer/solution-architectures]
often suggest using ADX to directly consume events from message queues such as
Kafka or Azure EventHub. However in actual development this is usually a
sub-optimal solution due to various reasons.
Furthermore, you may also find some docs describing other scenarios like [Azure
Data Explorer and Stream Analytics for anomaly
detection|https://azure.microsoft.com/en-us/blog/azure-data-explorer-and-stream-analytics-for-anomaly-detection/]
and [When to use Azure Data Explorer or/and Azure Stream
Analytics|https://docs.microsoft.com/en-us/azure/stream-analytics/azure-database-explorer-output#when-to-use-azure-data-explorer-orand-azure-stream-analytics]
(Azure Stream Analytics is an Azure native offering for streaming processing).
I personally often treat ADX as an close alternative to ElasticSearch, since
feature-wise they are close enough (though by design they are essentially
different in many ways). I hope this can also help you understand why I'd like
to use Flink with ADX. I believe AWS should have similar offering via Kinesis,
however I cannot comment on this since I'm not familiar with AWS .
> Source/Sink for Azure Data Explorer (ADX)
> -----------------------------------------
>
> Key: FLINK-25021
> URL: https://issues.apache.org/jira/browse/FLINK-25021
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Xue Wang
> Assignee: Xue Wang
> Priority: Major
>
> Hi all,
> I'm considering implementing source/sink for Azure Data Explorer (ADX). But
> first I'd like to check with the community if this is already a supported
> scenario. If not, will you consider adding it to the list of official
> connectors?
> FWIW, Azure Data Explorer is a widely used analytic service on Azure well
> suited for ad-hoc and time series analysis over large volume of structured,
> semi-structured, and unstructured data. And it can integrate with a wide
> range of data sources and visualization tools.
> References:
> [General Intro by ADX
> PM|https://vincentlauzon.com/2020/02/19/azure-data-explorer-kusto]
> [ADX
> Documentation|https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview]
> [Ingest data using the Azure Data Explorer Java
> SDK|https://docs.microsoft.com/en-us/azure/data-explorer/java-ingest-data]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
