I have created PR https://github.com/apache/beam/pull/15434 for JIRA https://issues.apache.org/jira/browse/BEAM-12824 : Beam Connector for Reading Data from Delta Lake
Now build using maven, I do not know how to create build.gradle for building java & scala code in one project. Could someone help to create build.gradle? I put examples for using Delta Lake connector into the new directory "beam/sdks/java/examples/deltalake", not sure it is the right place for the examples, could someone suggest another place? On Tue, Aug 31, 2021 at 8:48 AM Luke Cwik <[email protected]> wrote: > I added you as a contributor to JIRA and assigned BEAM-12824 to you. > > Typically you would create your own clone of the repo and make pull > requests from your clone to the apache/beam repo. See > https://beam.apache.org/contribute/#contributing-code for more details. > > Branches on the main repo are usually reserved for other purposes (e.g. > release branches). > > On Mon, Aug 30, 2021 at 8:56 PM Michael Benenson <[email protected]> > wrote: > >> Hi, Cham >> Right, it would be fine to extract common functionality from FileIO. >> >> On Mon, Aug 30, 2021 at 1:14 PM Chamikara Jayalath <[email protected]> >> wrote: >> >>> +1 to what Kenn said. >>> >>> We should probably think about how to not to copy a large amount of code >>> though. Seems like the following class currently repeats a large amount of >>> code in FileIO.java ? >>> >>> https://github.com/mbenenso/beam-deltalake/blob/main/src/main/java/org/apache/beam/sdk/io/DeltaFileIO.java >>> >>> I suspect most of this logic can be shared for code health. >>> >>> Thanks, >>> Cham >>> >>> >>> On Mon, Aug 30, 2021 at 1:07 PM Kenneth Knowles <[email protected]> wrote: >>> >>>> By all means, open a pull request to the main Beam repository (for this >>>> or any other change you would like to propose) with the connector and >>>> example. We can continue any detailed code-level conversation on the pull >>>> request. A quick 1-pager (we tend to use Google Docs) can be helpful for >>>> describing it at a high level so the community generally knows it exists >>>> and how it works. >>>> >>>> Could you also file a JIRA ticket so it is automatically included in >>>> our release notes once it is released? See >>>> https://beam.apache.org/community/contact-us/ and let us know your >>>> JIRA account. >>>> >>>> Kenn >>>> >>>> On Mon, Aug 30, 2021 at 12:33 PM Michael Benenson <[email protected]> >>>> wrote: >>>> >>>>> Hi, folks >>>>> >>>>> I have put into the open source Beam connector for reading data from >>>>> Delta Lake. Please, see https://github.com/mbenenso/beam-deltalake >>>>> for the connector & https://github.com/mbenenso/beam-deltalake-example >>>>> for the usage example. >>>>> >>>>> Would you like to consider adding this connector into the Beam >>>>> repository? >>>>> >>>>> -- >>>>> Thanks >>>>> >>>>> ------------------------------ >>>>> Michael Benenson >>>>> >>>> >> >> -- >> Thanks >> >> ------------------------------ >> Michael Benenson >> > -- Thanks ------------------------------ Michael Benenson
