[
https://issues.apache.org/jira/browse/SQOOP-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056011#comment-15056011
]
ASF GitHub Bot commented on SQOOP-1532:
---------------------------------------
GitHub user eruizgar opened a pull request:
https://github.com/apache/sqoop/pull/11
[SQOOP-1532] Support Sqoop on Spark Execution Engine
We have implemented the Issue SQOOP-1532 to support sqoop jobs to run on
spark environment. You can run on the standalone spark cluster, using sqoop
client.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Stratio/sqoop SQOOP-1532
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/sqoop/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 6fe1428afd5d98ff97a4e26ad19f34467fe6ddfa
Author: [email protected] <[email protected]>
Date: 2015-11-17T16:50:33Z
Added spark dependencies
commit eff93a0cc9428dcf401894649de0f679cf932fc6
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-17T16:56:16Z
SQOOP-2684: Sqoop2: Upgrade groovy to 2.4.0
(Dian Fu via Jarek Jarcec Cecho)
commit dc011e12b128853ea45a3c6f8a9704b7285ac16f
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-17T17:04:01Z
SQOOP-2682: Sqoop2: Add test cases for the object name with special char
(Colin Ma via Jarek Jarcec Cecho)
commit 370ea29c2aa1d1fb0814e56f0afd16d4f66e2e41
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-17T17:08:35Z
SQOOP-2680: Sqoop2: Remove the id from public interface for connection
(Colin Ma via Jarek Jarcec Cecho)
commit dfe984c14aef83338e1ac68972ba7503fcaa6d0f
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-18T16:06:49Z
SQOOP-2396: Sqoop2: Race condition in purge/update threads on Server
shutdown
(Dian Fu via Jarek Jarcec Cecho)
commit 68ca8bc2e1f2ceb8ad0ab14763a146fb0d2682db
Author: Kate Ting <[email protected]>
Date: 2015-11-18T22:34:17Z
SQOOP-2688: Sqoop2: Provide utility method to safely retrieve value from
JSONObject
(Jarek Jarcec Cecho via Kate Ting)
commit cc3e77b89e653a5f33996d53d3fe4fb7839c16a3
Author: Kate Ting <[email protected]>
Date: 2015-11-18T22:58:57Z
SQOOP-2694: Sqoop2: Doc: Register structure in sphinx for our docs
(Jarek Jarcec Cecho via Kate Ting)
commit 4f6ea567ffd9b5f43614c2c2b632789e9c752422
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-19T15:07:07Z
SQOOP-2700: Sqoop2: Tests in shell module are in infinite loop
(Dian Fu via Jarek Jarcec Cecho)
commit ee64ec6e2fb856a48f18685daf8459b9ad1da083
Author: Jarek Jarcec Cecho <[email protected]>
Date: 2015-11-20T16:19:01Z
SQOOP-2699: Sqoop2: Oraoop: Improve Oracle parameters
(David Robson via Jarek Jarcec Cecho)
commit bad653c995df426189c67775748ed83321b6ad54
Author: Kate Ting <[email protected]>
Date: 2015-11-20T22:26:21Z
SQOOP-2698: Sqoop2: RESTiliency: Split the InvalidRESTCallsTest into
independent test cases
(Jarek Jarcec Cecho via Kate Ting)
commit 2c58a54ec871dc05198023ab6e8a3e2afa1d9343
Author: Enrique ruiz <[email protected]>
Date: 2015-11-17T16:50:33Z
[SQOOP-1532] Support Sqoop on Spark Execution Engine
----
> Sqoop2: Support Sqoop on Spark Execution Engine
> -----------------------------------------------
>
> Key: SQOOP-1532
> URL: https://issues.apache.org/jira/browse/SQOOP-1532
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 2.0.0
>
>
> The current execution engine supported in sqoop is MR.
> The goal if this ticket is to support sqoop jobs ( map only and map+reduce )
> to run on spark environment.
> It should at the minimum support running on the standalone spark cluster and
> then subsequently work with YARN/mesos.
> High level goals
> 1. Hook up with the connector apis to provide the basic load/ extract to the
> spark RDD.
> 2. Implementation of the Sqoop RDD to support extraction from different data
> sources . The design proposal will discuss the alternatives on how this can
> be achieved.
> 3. Optimizing the loading/writing ( re-use/ refactor the consumer thread code
> to be agnostic of the hadoop output format)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
