Hi Gokul,
When allowing a user to connect to a remote HDFS namenode, are we assuming
that it will always access the root ("/") ? or do we have a way to specify
which folder the data source should point to?
On Thu, May 21, 2015 at 12:05 PM, Gokul Balakrishnan <[email protected]> wrote:
> Hi Srinath/Anjana,
>
> Sorry for the delay. Yes, we're changing from Hadoop as part of the
> improvements, where we will have the same underlying implementation for the
> config but with two DatasourceReader implementations, one each for HBase
> and HDFS respectively.
>
> We are also modifying the format of the expected configuration to better
> match the properties specified in other Carbon datasource configurations,
> so the updated config would look something like:
>
> <datasource>
> <name>WSO2_ANALYTICS_FS_DB_HDFS</name>
> <description>The datasource used for analytics file system</
> description>
> <jndiConfig>
> <name>jdbc/WSO2HDFSDB</name>
> </jndiConfig>
> <definition type="HDFS">
> <configuration>
> <property name="fs.default.name" value="hdfs://localhost:9000"
> />
> <property name="dfs.data.dir" value="/dfs/data" />
> <property name="fs.hdfs.impl" value
> ="org.apache.hadoop.hdfs.DistributedFileSystem" />
> <property name="fs.file.impl" value
> ="org.apache.hadoop.fs.LocalFileSystem" />
> </configuration>
> </definition>
> </datasource>
>
> Thanks,
>
> On 14 May 2015 at 00:52, Anjana Fernando <[email protected]> wrote:
>
>> Hi Srinath,
>>
>> Yeah, I'd a chat with Gokul yesterday, we are changing this to HDFS and
>> also having another HBase one as well, I think he has already done the
>> changes. @Gokul, please send the updated information.
>>
>> Cheers,
>> Anjana.
>>
>> On Thu, May 14, 2015 at 1:10 PM, Srinath Perera <[email protected]> wrote:
>>
>>> Can we call type HDFS instead of Hadoop? ( if we can change that without
>>> much trouble)
>>>
>>> On Tue, May 12, 2015 at 8:38 PM, Gokul Balakrishnan <[email protected]>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> As part of the HBase analytics datasource implementation for DAS 3.0,
>>>> we have come up with $subject which is envisioned to offer a standardised
>>>> way to specify connectivity parameters for a remote Hadoop-based instance
>>>> in a Carbon datasource configuration.
>>>>
>>>> The datasource reader will expect the configuration to be specified in
>>>> a similar format which is used for standard Apache Commons Configuration
>>>> [1], as used by both HDFS and HBase. An example datasource definition would
>>>> look like:
>>>>
>>>> <datasource>
>>>> <name>WSO2_ANALYTICS_FS_DB_HDFS</name>
>>>> <description>The datasource used for analytics file system</
>>>> description>
>>>> <jndiConfig>
>>>> <name>jdbc/WSO2HDFSDB</name>
>>>> </jndiConfig>
>>>> <definition type="HADOOP">
>>>> <configuration>
>>>> <property>
>>>> <name>fs.default.name</name>
>>>> <value>hdfs://localhost:9000</value>
>>>> </property>
>>>> <property>
>>>> <name>dfs.data.dir</name>
>>>> <value>/dfs/data</value>
>>>> </property>
>>>> <property>
>>>> <name>fs.hdfs.impl</name>
>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</
>>>> value>
>>>> </property>
>>>> <property>
>>>> <name>fs.file.impl</name>
>>>> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>> </property>
>>>> </configuration>
>>>> </definition>
>>>> </datasource>
>>>>
>>>> The definition type for the above is set as "HADOOP". The datasource
>>>> reader implementation is currently hosted at [2], and would be merged with
>>>> the carbon-data git repo once reviewed.
>>>>
>>>> Appreciate your thought and suggestions.
>>>>
>>>> Thanks,
>>>> Gokul.
>>>>
>>>> [1] http://commons.apache.org/proper/commons-configuration/
>>>>
>>>> [2]
>>>> https://github.com/gokulbs/carbon-data/tree/master/components/data-sources/org.wso2.carbon.datasource.reader.hadoop
>>>>
>>>> --
>>>> Balakrishnan Gokulakrishnan
>>>> Senior Software Engineer,
>>>> WSO2, Inc. http://wso2.com
>>>> Mob: +94 77 593 5789 | +1 650 272 9927
>>>>
>>>> _______________________________________________
>>>> Architecture mailing list
>>>> [email protected]
>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>>
>>>>
>>>
>>>
>>> --
>>> ============================
>>> Srinath Perera, Ph.D.
>>> http://people.apache.org/~hemapani/
>>> http://srinathsview.blogspot.com/
>>>
>>> _______________________________________________
>>> Architecture mailing list
>>> [email protected]
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> *Anjana Fernando*
>> Senior Technical Lead
>> WSO2 Inc. | http://wso2.com
>> lean . enterprise . middleware
>>
>
>
>
> --
> Balakrishnan Gokulakrishnan
> Senior Software Engineer,
> WSO2, Inc. http://wso2.com
> Mob: +94 77 593 5789 | +1 650 272 9927
>
> _______________________________________________
> Architecture mailing list
> [email protected]
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>
--
Thanks and Regards
*,Shani Ranasinghe*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 77 2273555
Blog: http://waysandmeans.blogspot.com/
linked in: lk.linkedin.com/pub/shani-ranasinghe/34/111/ab
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
