Hi Evgeny,
I can share pseudo code, not all developed code and python modules.
Syslog messages are inseted into cache via Flume.
Python daemons process messages ( read from CACHE ) and make
insert/update/delete in fiew Ignite tables via SQL.
Number of syslog messages and python code is same on prod and test.
##############################################
# scan CACHE, fetch k/v from CACHE, SQL insert/update/delete ...
while True:
try:
ig_connection = new_ignite_connection(config_params,
logger)
stream_cache = ig_connection.get_cache("XY_CACHE_NAME")
# cache is
created by code with get_or_create_cache
CACHE_SIZE = stream_cache.get_size()
if CACHE_SIZE > 0:
# returns a generator, that yields two-tuples
of key and value
cache_list = stream_cache.scan()
for k, v in cache_list:
# parse k, v and make some
INSERT/UPDATE in TABLES
# TABLE ALERTS ~ 50 colums
SQL_EVENT_INSERT_QUERY =
"INSERT INTO ALERTS (%s) VALUES (%s)" %
(alerts_insert_column_names(), alerts_insert_parameters() )
ig_connection.sql(SQL_EVENT_INSERT_QUERY, query_args =
var_EVENT_INSERT_QUERY)
# remove
stream_cache.remove_key(k)
ig_connection.close()
time.sleep(POLL_INTERVAL)
except (OSError, SocketError) as e:
logger.error('ERROR in client connection: %s , client
status: %s ' %(e,
ig_connection))
##############################################
<http://apache-ignite-users.70518.x6.nabble.com/file/t2557/Ignite_prod_test_HLD.png>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
