matthieucham commented on pull request #7293: URL: https://github.com/apache/beam/pull/7293#issuecomment-770363477
Hi Ismael I was able to work around the «replace» issue by simply adding an empty BsonDocument in the pipeline when it consists in more than one stage. That way, I could retrieve documents from a MongoDb view. But this only work on small collections: on huge collections the $bucketAuto stage make the sorting overflow the 100MB ram limit on the MongoDb side, and there was no way around it, unless forking MongoDbIo completely, which I did ultimately. I'll give more details in a blog post soon Regards Le ven. 29 janv. 2021 17:22, Ismaël Mejía <[email protected]> a écrit : > Sorry for late answer. I am also not that closely familiar with MongoDB > APIs so you might be right about the overwrite. > > If I remember correctly I think the hardcoded cast to FindQuery was in > place to support projection too, it is ugly indeed and hopefully we can > correct it. Did you find a workaround or find some possible fix to propose? > > Don't hesitate to create a JIRA issue for this and tell me if you feel > motivated to work on it. In case of more questions you can ask me here or > on TheASF slack > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/apache/beam/pull/7293#issuecomment-769904071>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABIHEQXCT5KV6TAM5ZEDGHTS4LODNANCNFSM4GKVQLCA> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
