Nikolay Sokolov created GRIFFIN-213:
---------------------------------------
Summary: Support pluggable datasource connectors
Key: GRIFFIN-213
URL: https://issues.apache.org/jira/browse/GRIFFIN-213
Project: Griffin (Incubating)
Issue Type: Improvement
Reporter: Nikolay Sokolov
As of Griffin 0.3, code modification is required, in order to add new data
connectors.
Proposal is to add new data connector type, CUSTOM, that would allow to specify
class name of data connector implementation to use. Additional jars with custom
connector implementations would be provided in spark configuration template.
Class name would be specified in "class" config of data connector. For example:
{code:json}
"connectors": [
{
"type": "CUSTOM",
"config": {
"class": "org.example.griffin.JDBCConnector"
// extra connector-specific parameters
}
}
]
{code}
Proposed contract for implementations is based on current convention:
- for batch
** class should be a subclass of BatchDataConnector
** if should have method with signature:
{code:java}
public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
{code}
- for streaming
** class should be a subclass of StreamingDataConnector
** it should have method with signature:
{code:java}
public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
{code}
Signatures of context objects:
{code:scala}
case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
dcParam: DataConnectorParam,
timestampStorage: TimestampStorage)
case class StreamingDataConnectorContext(@transient sparkSession: SparkSession,
@transient ssc: StreamingContext,
dcParam: DataConnectorParam,
timestampStorage: TimestampStorage,
streamingCacheClientOpt:
Option[StreamingCacheClient])
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
