I am excited to introduce a new feature releted to Tika's external module - the ability to inject protobufs via grpc from an external parser. To demonstrate this, as a StromCrawler committer, I have made an integration from StromCrawler to achieve this.
Some development details: Currently Tika runs within the worker. I've been working on the other side for a while now: a bolt that sends the page it already fetched to a separate tika server over grpc, no second fetch, tika never touches the network. You should check it out - here's a demo running it end to end with StormCrawler, one bolt talking to tika, a checker confirming nothing gets fetched twice: https://github.com/dpol1/tika-demo The real bolt for stormcrawler: waits for a tika release with the grpc classes. before writing it, I'd love to get feedback on how you would want it shaped, a bolt next to the parser or a drop-in replacement for it. Cheers, Davide
