eamonford commented on pull request #12: URL: https://github.com/apache/incubator-sdap-ingester/pull/12#issuecomment-669474653
> A general comment, initially I found the usage of async functions pretty smart but the more I see them the more I am afraid the code will be difficult to maintain. The purpose of having the async call is to make the components robust to the unavailability of the other components ? Right ? I guess this become common in the microservices architectures... I am interested by your thoughts on that. I agree asyncio can be confusing to read sometimes. One of the main advantages of using asyncio instead of multithreading is that it does *not* spawn threads (unless you explicitly ask it to) -- rather it allows you to control where the processor switches between executing different parts of the code. Not spawning threads is an advantage because 1) sometimes libraries or your own code are not thread-safe and 2) the CPU can only handle as many threads simultaneously as the number of cores it has. So, asyncio is good for cases where there is a lot of waiting on I/O, as when the Granule Ingester saves tiles to Solr and Cassandra. I want to point out though that the granule ingester was already using asyncio for writing to Solr, but it was using the aiohttp library instead of pysolr. In this commit, I've switched out aiohttp for pysolr because it makes it easier to connect using zookeeper -- unfortunately pysolr doesn't natively use asyncio so I added the `run_in_executor` wrapper so that it'll work with asyncio. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
