Hi Kafka people, I am very new to kafka, so perhaps my question is naive. I spent some time searching around at resources of kafka but only became more confused.
What are common ways to pull info from a web site and send to kafka to become a log entry? There is a web site that I want to pull a piece of data from once/month and have that data written to Kafka log. Consumers will be listening for that message to do processing on it. I am not sure about common ways to do this. I am thinking I could have some scheduler (e.g. cron) wake up once per month and trigger the pull of the data from the web site and then send it to a kafka stream. Does kafka have ability to trigger events once/month or is using cron a better idea? What is the scheduler triggering a stand-alone batch job or the running of a some service like a kafka producer? Should I worry about a service running all the time when it is likely to only do a few seconds of work each month? Many thanks, James.
