ASF GitHub Bot commented on BAHIR-75:

GitHub user sourav-mazumder opened a pull request:


    [BAHIR-75] [WIP] Remote HDFS connector for Apache Spark using webhdfs 
protocol with support for Apache Knox

    This component implements Hadoop File System 
(org.apache,hadoop.fs.FileSystem) to provide an alternate mechanism (instead of 
using 'webhdfs or swebhdfs' file uri) for Spark to access (read/write) files 
from/to a remote Hadoop cluster using webhdfs protocol. 
    This component takes care of the following requirements related to 
accessing files (read/write) from/to a remote enterprise Hadoop cluster from a 
remote Spark cluster-
    1. Support for Apache Knox
    2. Support for passing user id/password different from the user who has 
started the spark-shell/spark-submit process.
    3. Support for SSL in three modes - Ignoring certificate validation, 
certificate validation through user supplied trust store path and password, and 
automatic creation of certificate using openssl and keytool.
    4. Optimized way of getting data from remote HDFS where each connection 
will get only its part of data.
    This component is not a full fledged implementation of Hadoop File System. 
It implements only those interfaces those are needed by Spark for reading data 
form remote HDFS and writing back the data to remote HDFS.
    **Example Usage -**
    Step 1: Set Hadoop configuration to define a custom uri of your choice and 
specify the class name BahirWebHdfsFileSystem. For example - 
    You can use any name (apart form the standard uris like hdfs, webhdfs, file 
etc. already used by Spark) instead of 'remoteHdfs'. However subsequently while 
loading the file (or writing a file) the same should be used.
    Step 2: Set the user name and password as below -
    `val userid = "biadmin"`
    `val password = "password"`
    `val userCred = userid + ":" + password`
    Step 3 : Now you are ready to load any file from the remote Hadoop cluster 
using Spark's standard Dataframe/DataSet APIs. For example -
    `val filePath = 
    `val srvr = 
    `val knoxPath = "gateway/default"`
    `val webhdfsPath = "webhdfs/v1"`
    `val prtcl = "remoteHdfs"`
    `val fullPath = s"$prtcl://$srvr/$knoxPath/$webhdfsPath/$filePath"`
    `val df = spark.read.format("csv").option("header", "true").load(fullPath)`
    Please not the use of 'gateway/default' and 'webhdfs/v1' used for 
specifying the server specific information in the path. The first one is 
specific to Apache Knox and the second one is specific for  webhdfs protocol.
    Step 4; To write data back to remote HDFS following steps can be used 
(using standard Dataframe writer of spark)
    `val filePathWrite = 
    `val srvr = "ehaasp-577-mastermanager.bi.services.bluemix.net:8443"`
    `val knoxPath = "gateway/default"`
    `val webhdfsPath = "webhdfs/v1"`
    `val prtcl = "remoteHdfs"`
    `val fullPath = s"$prtcl://$srvr/$knoxPath/$webhdfsPath/$filePathWrite"`
    `df.write.format("csv").option("header", "true").save(filePathw)`
    **We are still working on followings -**
    - Unit Testing
    - Code cleanup
    - Examples showcasing various configuration parameters
    - API documentation

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sourav-mazumder/bahir 

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #28
commit eee70dc6dac8aa1b3b9387aaa136250d340f2b46
Author: Sourav Mazumder <sourav.mazumde...@gmail.com>
Date:   2016-11-15T23:34:14Z

    [BAHIR-75] Initital code delivery for WebHDFS data source

commit c2d53fdd55eee69120cf00dc17869786945ed93a
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T01:00:39Z

    [BAHIR-75] - fix RAT excludes for DataSourceRegister

commit af805e3226cbc31b2a9993aa635795a3d1fdd8c7
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T01:27:08Z

    [BAHIR-75] - minor README fixes

commit 78ff29c8885534935d43a911af9c27f667725989
Author: Christian Kadner <ckad...@users.noreply.github.com>
Date:   2016-11-16T01:38:04Z

    [BAHIR-75] - minor README fixes (2)

commit 24e79c9743f46de296941a423dd94772174495bd
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T03:16:49Z

    [BAHIR-75] - include DataSourceRegister in Maven build

commit 365ee1f702dc42e50c2e922e9c069f80f2d016a2
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T18:37:46Z

    [BAHIR-75] - fix package declaration in webhdfs package object

commit d4c6e56db57a9cd13b0a25dc77cc3c4179c9a7b7
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T22:46:39Z

    [BAHIR-75] - fix 798 Scalastyle violations

commit d7b3bf7ed849ef5f39ffb935127fa049b3537603
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-16T23:32:16Z

    [BAHIR-75] - use "${scala.binary.version}"" instead of "2.11"

commit a77e3725693c9d4034ffa4dd6a4b9521824370eb
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-11-17T03:20:56Z

    [BAHIR-75] - add "spark-" prefix to artifactId consistent with other Bahir 

commit 6936bd8a47919db8093a596f30761c7b46c98547
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-01T00:38:53Z

    [BAHIR-75][WIP] - rudimentary extension of WebHdfsFileSystem

commit a9ef90745d498a5243ca96e4615d859733fcba60
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-01T02:10:55Z

    [BAHIR-75][WIP] - rudimentary extension of WebHdfsFileSystem (use original 
hadoop namespace for package private field/method access)

commit f791f1cb330518e56d968ac04605cb91b9fbfefa
Author: Sourav Mazumder <sourav.mazumde...@gmail.com>
Date:   2016-12-07T21:21:59Z

    WebHdfsConnector prototype

commit 2932f99244b23732ff1a6306f7e88e42c72a3360
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-07T22:01:50Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - code style fixes, remove 
unused files

commit 2497971264299fc8dbc5475b4787f15008b671e8
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-07T22:31:52Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - more code style fixes, 
remove more unused files

commit 29718806923a64d1b56490978516b8bd95479f8c
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-07T22:39:34Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - more and more code style 
fixes, remove more unused files

commit a9bbe31d17706d04bb44214b33479980a24a7d8e
Author: Sourav Mazumder <smazum...@sourav1.fyre.ibm.com>
Date:   2016-12-08T00:41:51Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - add printouts for debugging

commit f6429c9be053f996fbfe6c7bf794843375aebf27
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-08T00:50:12Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - add printouts for debugging

commit 39f59851de1c462545866ea17ee547493fc71c5b
Author: Sourav Mazumder <smazum...@us.ibm.com>
Date:   2016-12-09T18:56:01Z

    [BAHIR-75][WIP] - write to remote via webhdfs

commit 183b1ec88b26c3de578bab4d2b8eb9ebb1c80e69
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-09T19:11:15Z

    [BAHIR-75][WIP] - override WebHdfsFileSystem - fix code style errors, 
temporarily disable style checks for println

commit 59cad8ea9fbbcc4951b2a8a121e736e4121a31a7
Author: Sourav Mazumder <smazum...@us.ibm.com>
Date:   2016-12-12T22:38:49Z

    [BAHIR-75][WIP] - write files via webhdfs

commit d047318a899636242ac83471f58c044dc7be9f45
Author: Sourav Mazumder <smazum...@us.ibm.com>
Date:   2016-12-22T20:22:00Z

    [BAHIR-75][WIP] - write files via webhdfs continued

commit 8beedf99579c35cc47ea3d65cf45828f41d9969b
Author: Christian Kadner <ckad...@apache.org>
Date:   2016-12-22T20:38:54Z

    [BAHIR-75][WIP] - custom WebHdfsFileSystem - minor scalastyle fixes

commit b63a202a431db5261de9ce1ec19dfa943f565807
Author: Christian Kadner <ckad...@us.ibm.com>
Date:   2016-12-22T21:14:03Z

    Merge branch 'master' into BAHIR-75-WebHdfsFileSystem

commit b467c5250ef51db4fec642d455740acd56a7caae
Author: Christian Kadner <ckad...@apache.org>
Date:   2016-12-22T23:19:51Z

    [BAHIR-75][WIP] - remove unnecessary dependencies from pom.xml

commit d3de3a725f0dfcab194a02cd3279594aa988292b
Author: Sourav Mazumder <smazum...@us.ibm.com>
Date:   2016-12-23T23:09:15Z

    [BAHIR-75][WIP] - minor fixes

commit b103aa965d9e48d20986d094699ed3cfec352f4c
Author: Sourav Mazumder <sourav.mazumde...@gmail.com>
Date:   2016-12-23T23:10:56Z

    Merge branch 'BAHIR-75-WebHdfsFileSystem' of 
https://github.com/sourav-mazumder/bahir into BAHIR-75-WebHdfsFileSystem


> WebHDFS: Initial Code Delivery
> ------------------------------
>                 Key: BAHIR-75
>                 URL: https://issues.apache.org/jira/browse/BAHIR-75
>             Project: Bahir
>          Issue Type: Sub-task
>          Components: Spark SQL Data Sources
>            Reporter: Sourav Mazumder
>   Original Estimate: 504h
>  Remaining Estimate: 504h

This message was sent by Atlassian JIRA

Reply via email to