[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595059#comment-16595059 ] Pascal Gula commented on BEAM-5148: --- This might be useful? https://github.com/uqfoundation/dill/issues/207 > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Pascal Gula >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595050#comment-16595050 ] Pascal Gula commented on BEAM-5148: --- [~chamikara], I am facing an issue with the Sink part of the connector, and described it in this SO post: [https://stackoverflow.com/questions/52040923/error-trying-to-implement-a-mongodb-io-connector-sink] Any help would be very useful! > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Pascal Gula >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589895#comment-16589895 ] Chamikara Jayalath commented on BEAM-5148: -- Checkout existing source/sink tests (for example, textio_test) for the set of tests that you should consider developing. > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Chamikara Jayalath >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589894#comment-16589894 ] Chamikara Jayalath commented on BEAM-5148: -- Checkout existing source/sink tests (for example, textio_test) for the set of tests that you should consider developing. > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Chamikara Jayalath >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589879#comment-16589879 ] Pascal Gula commented on BEAM-5148: --- Thanks [~chamikara] for your feedback. I sent the request for the role. Before submitting the IO connector, it might be wise to have some basic test ready to run. Can you point me to the most complete one so that I can follow the coding/test style? > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Chamikara Jayalath >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589202#comment-16589202 ] Chamikara Jayalath commented on BEAM-5148: -- Thanks. Looks good in general. Could you send this for review in the form of a Beam pull request so that I can provide detailed comments. Also, I tried to assign this Jira to you but seems like you currently don't have the Beam contributor role assigned to your Jira account. Could you send a request through dev list or Slack for this ? (a PMC member can add you). > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Chamikara Jayalath >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582809#comment-16582809 ] Pascal Gula commented on BEAM-5148: --- Thanks [~altay] for your feedback! I would be more than happy to refactor the implementation using the latest framework as soon as I could get some minimal guidance. Waiting for the feedback of [~chamikara] to move forward. > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Ahmet Altay >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582783#comment-16582783 ] Ahmet Altay commented on BEAM-5148: --- Thank you [~GeoloeG_IsT]. [~chamikara] could provide feedback on your implementation. And if you like you could move it to Beam in the future. One major feedback I have is, this IO is using the old source framework. Beam is transitioning towards building sources using the SDF framework and the previous IO framework will be eventually deprecated. > Implement MongoDB IO for Python SDK > --- > > Key: BEAM-5148 > URL: https://issues.apache.org/jira/browse/BEAM-5148 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 3.0.0 >Reporter: Pascal Gula >Assignee: Ahmet Altay >Priority: Major > Fix For: Not applicable > > > Currently Java SDK has MongoDB support but Python SDK does not. With current > portability efforts other runners may soon be able to use Python SDK. Having > mongoDB support will allow these runners to execute large scale jobs using it. > Since we need this IO components @ Peat, we started working on a PyPi package > available at this repository: [https://github.com/PEAT-AI/beam-extended] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
