Re: [DISCUSS] Proposal for Ignite Extensions as a separate Apache Bahir module
Hi Luciano, Thank you for your email. As discussed in Ignite community we are planning to migrate the Apache Ignite integrations in ignite-extentions git repository. https://github.com/apache/ignite-extensions Warm regards, Saikat On Sun, Nov 3, 2019 at 7:09 PM Luciano Resende wrote: > Seems like this has died down without any issues, what should be our > next steps here? > > On Wed, Oct 23, 2019 at 1:12 PM Dmitry Pavlov wrote: > > > > Hi Bahir Community, > > > > I would also help with these extensions migration, release, patch > review. My involvement is highly dependent on the current workload. > > > > I would like to wait a little bit to be sure everybody agrees in Ignite. > > > > Sincerely, > > Dmitriy Pavlov > > PMC and Committer at Apache Ignite > > PPMC and Committer at Apache Training (incubating). > > > > On 2019/10/22 01:23:43, Luciano Resende wrote: > > > On Mon, Oct 21, 2019 at 5:48 PM Saikat Maitra > wrote: > > > > > > > > Hello, > > > > > > > > I am Saikat and I am committer in Apache Ignite project. I am > interested in > > > > joining the Apache Bahir community and contribute to following Apache > > > > Ignite Extensions. > > > > > > > > https://apacheignite.readme.io/docs/integrations > > > > > > > > The reason we wanted to contribute our Apache Ignite integration as > > > > separate Extensions is this will help us to manage and maintain > separate > > > > lifecycle for Apache Ignite integrations. > > > > > > > > All the integrations will continue to be part of ASF and we will > keep > > > > supporting and developing in accordance with ASF vision and > practices. Our > > > > inspiration for the move is very similar to as mentioned in Apache > Flink. > > > > > > > > https://flink.apache.org/ecosystem.html#third-party-projects > > > > > > > > I would be very grateful if you please review and share your > thoughts on > > > > the proposal. > > > > > > > > Warm Regards, > > > > Saikat > > > > > > What are your thoughts around involvement on the maintenance tasks > > > such as pr reviews and releases around these extensions? > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
Re: [DISCUSS] Proposal for Ignite Extensions as a separate Apache Bahir module
Seems like this has died down without any issues, what should be our next steps here? On Wed, Oct 23, 2019 at 1:12 PM Dmitry Pavlov wrote: > > Hi Bahir Community, > > I would also help with these extensions migration, release, patch review. My > involvement is highly dependent on the current workload. > > I would like to wait a little bit to be sure everybody agrees in Ignite. > > Sincerely, > Dmitriy Pavlov > PMC and Committer at Apache Ignite > PPMC and Committer at Apache Training (incubating). > > On 2019/10/22 01:23:43, Luciano Resende wrote: > > On Mon, Oct 21, 2019 at 5:48 PM Saikat Maitra > > wrote: > > > > > > Hello, > > > > > > I am Saikat and I am committer in Apache Ignite project. I am interested > > > in > > > joining the Apache Bahir community and contribute to following Apache > > > Ignite Extensions. > > > > > > https://apacheignite.readme.io/docs/integrations > > > > > > The reason we wanted to contribute our Apache Ignite integration as > > > separate Extensions is this will help us to manage and maintain separate > > > lifecycle for Apache Ignite integrations. > > > > > > All the integrations will continue to be part of ASF and we will keep > > > supporting and developing in accordance with ASF vision and practices. Our > > > inspiration for the move is very similar to as mentioned in Apache Flink. > > > > > > https://flink.apache.org/ecosystem.html#third-party-projects > > > > > > I would be very grateful if you please review and share your thoughts on > > > the proposal. > > > > > > Warm Regards, > > > Saikat > > > > What are your thoughts around involvement on the maintenance tasks > > such as pr reviews and releases around these extensions? > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Commented] (BAHIR-213) Faster S3 file Source for Structured Streaming with SQS
[ https://issues.apache.org/jira/browse/BAHIR-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966315#comment-16966315 ] ASF GitHub Bot commented on BAHIR-213: -- lresende commented on issue #91: [BAHIR-213] Faster S3 file Source for Structured Streaming with SQS URL: https://github.com/apache/bahir/pull/91#issuecomment-549194053 ping @steveloughran This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Faster S3 file Source for Structured Streaming with SQS > --- > > Key: BAHIR-213 > URL: https://issues.apache.org/jira/browse/BAHIR-213 > Project: Bahir > Issue Type: New Feature > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.4.0 >Reporter: Abhishek Dixit >Priority: Major > > Using FileStreamSource to read files from a S3 bucket has problems both in > terms of costs and latency: > * *Latency:* Listing all the files in S3 buckets every microbatch can be > both slow and resource intensive. > * *Costs:* Making List API requests to S3 every microbatch can be costly. > The solution is to use Amazon Simple Queue Service (SQS) which lets you find > new files written to S3 bucket without the need to list all the files every > microbatch. > S3 buckets can be configured to send notification to an Amazon SQS Queue on > Object Create / Object Delete events. For details see AWS documentation here > [Configuring S3 Event > Notifications|https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html] > > Spark can leverage this to find new files written to S3 bucket by reading > notifications from SQS queue instead of listing files every microbatch. > I hope to contribute changes proposed in [this pull > request|https://github.com/apache/spark/pull/24934] to Apache Bahir as > suggested by [gaborgsomogyi|https://github.com/gaborgsomogyi] > [here|https://github.com/apache/spark/pull/24934#issuecomment-511389130] -- This message was sent by Atlassian Jira (v8.3.4#803005)
