[jira] [Commented] (BAHIR-213) Faster S3 file Source for Structured Streaming with SQS
[ https://issues.apache.org/jira/browse/BAHIR-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919860#comment-16919860 ] Abhishek Dixit commented on BAHIR-213: -- [~lresende] [~ste...@apache.org] This PR for new extension has been open for quite some time with no reviews or reviewers assigned. Can you please review or assign a reviewer? Do I need to send an email to Bhair Mailing List for that? > Faster S3 file Source for Structured Streaming with SQS > --- > > Key: BAHIR-213 > URL: https://issues.apache.org/jira/browse/BAHIR-213 > Project: Bahir > Issue Type: New Feature > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.4.0 >Reporter: Abhishek Dixit >Priority: Major > > Using FileStreamSource to read files from a S3 bucket has problems both in > terms of costs and latency: > * *Latency:* Listing all the files in S3 buckets every microbatch can be > both slow and resource intensive. > * *Costs:* Making List API requests to S3 every microbatch can be costly. > The solution is to use Amazon Simple Queue Service (SQS) which lets you find > new files written to S3 bucket without the need to list all the files every > microbatch. > S3 buckets can be configured to send notification to an Amazon SQS Queue on > Object Create / Object Delete events. For details see AWS documentation here > [Configuring S3 Event > Notifications|https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html] > > Spark can leverage this to find new files written to S3 bucket by reading > notifications from SQS queue instead of listing files every microbatch. > I hope to contribute changes proposed in [this pull > request|https://github.com/apache/spark/pull/24934] to Apache Bahir as > suggested by [gaborgsomogyi|https://github.com/gaborgsomogyi] > [here|https://github.com/apache/spark/pull/24934#issuecomment-511389130] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende reassigned BAHIR-217:
-
Assignee: Abhishek Dixit
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Assignee: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
> Fix For: Spark-2.4.0
>
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved BAHIR-217.
---
Fix Version/s: Spark-2.4.0
Resolution: Fixed
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
> Fix For: Spark-2.4.0
>
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919824#comment-16919824
]
ASF subversion and git services commented on BAHIR-217:
---
Commit 549c50be02f98b93c5f79890332e8de97332e8f5 in bahir's branch
refs/heads/master from abhishekd0907
[ https://gitbox.apache.org/repos/asf?p=bahir.git;h=549c50b ]
[BAHIR-217] Installation of Oracle JDK8 is Failing in Travis CI (#93)
Install of Oracle JDK 8 Failing in Travis CI and as a result,
build is failing for new pull requests.
We just need to add `dist: trusty` in the .travis.yml file
as mentioned in the issue below:
https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919823#comment-16919823
]
ASF GitHub Bot commented on BAHIR-217:
--
lresende commented on pull request #93: [BAHIR-217] Install of Oracle JDK 8
Failing in Travis CI
URL: https://github.com/apache/bahir/pull/93
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919821#comment-16919821
]
ASF GitHub Bot commented on BAHIR-217:
--
lresende commented on issue #93: [BAHIR-217] Install of Oracle JDK 8 Failing in
Travis CI
URL: https://github.com/apache/bahir/pull/93#issuecomment-526706426
Thanks for the clarification. LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919811#comment-16919811
]
ASF GitHub Bot commented on BAHIR-217:
--
abhishekd0907 commented on issue #93: [BAHIR-217] Install of Oracle JDK 8
Failing in Travis CI
URL: https://github.com/apache/bahir/pull/93#issuecomment-526698979
> Just out of curiosity, I [build master on
travis](https://travis-ci.org/apache/bahir/builds/544930279) without a problem.
How are you seeing the issue requiring trusty distro for travis builds.
@lresende I had raised another PR in BHAIR regarding sql-streaming SQS
connector #91
and its travis build is failing due to install jdk error. You can see the
details below:
[Travis Job log](https://travis-ci.org/apache/bahir/jobs/573750572)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919736#comment-16919736
]
ASF GitHub Bot commented on BAHIR-217:
--
lresende commented on issue #93: [BAHIR-217] Install of Oracle JDK 8 Failing in
Travis CI
URL: https://github.com/apache/bahir/pull/93#issuecomment-526678080
Just out of curiosity, I [build master on
travis](https://travis-ci.org/apache/bahir/builds/544930279) without a problem.
How are you seeing the issue requiring trusty distro for travis builds.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BAHIR-217) Install of Oracle JDK 8 Failing in Travis CI
[
https://issues.apache.org/jira/browse/BAHIR-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919359#comment-16919359
]
ASF GitHub Bot commented on BAHIR-217:
--
abhishekd0907 commented on pull request #93: [BAHIR-217] Install of Oracle JDK
8 Failing in Travis CI
URL: https://github.com/apache/bahir/pull/93
## What changes were proposed in this pull request?
Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
for new pull requests.
We need to make a small fix in _.travis.yml_ file as mentioned in the
issue here:
https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
We just need to add
`dist: trusty`
in the _.travis.yml_ file as mentioned in the issue above.
## How will these changes be tested?
Travis Build should successfully pass for this PR, thus testing the changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Install of Oracle JDK 8 Failing in Travis CI
>
>
> Key: BAHIR-217
> URL: https://issues.apache.org/jira/browse/BAHIR-217
> Project: Bahir
> Issue Type: Bug
> Components: Build
>Reporter: Abhishek Dixit
>Priority: Major
> Labels: build, easyfix
>
> Install of Oracle JDK 8 Failing in Travis CI. As a result, build is failing
> for new pull requests.
> We need to make a small fix in _ __ .travis.yml_ file as mentioned in the
> issue here:
> https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
> We just need to add
> {code:java}
> dist: trusty{code}
> in the .travis.yml file as mentioned in the issue above.
> I can raise a PR for this fix if required.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
