How to build a auditing process in data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Hi All,
Thanks a lot for all your previous suggestions.
As you suggested I am using Spark SQL for data migration i.e oracle ->
cassandra.

But I have two challenges
1) How should I handle failures , how that logic should be build?
2) How to do/handle any incremental migration?
3) How to do auditing ?

Please give me some ideas around these.

Regards,
Shyam

On Fri, Sep 14, 2018 at 9:42 PM sha p  wrote:

> Hi All,
>  Is there any free Cassandra IDEs available? If so please share?
>
> what is advantages of enterprise edition IDEs?
>
> Regards,
> Shyam
>
> On Wed, Sep 5, 2018 at 6:51 PM sha p  wrote:
>
>>
>> Hi all ,
>>> Me new to Cassandra , i was asked to migrate data from Oracle to
>>> Cassandra.
>>> Please help me giving your valuable guidance.
>>> 1) Can it be done using open source Cassandra.
>>> 2) Where should I start data model from?
>>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>>> 4) How I decide the size of cluster , please provide some sample
>>> guidelines.
>>> 5) this should be in production , so what kind of things i should take
>>> care for better support or debugging tomorrow?
>>> 6) Please provide some good books /links which can help me in this task.
>>>
>>>
>>> Thanks in advance.
>>> Highly appreciated your every amal help.
>>>
>>> Regards,
>>> Shyam
>>>
>>

Re: Error during truncate: Cannot achieve consistency level ALL , how to fix it

[sha p]

RF is replication factor. Sorry for confusing

On 19 Sep 2018 5:45 p.m., "Jonathan Baynes" 
wrote:

What RF is your system_auth keyspace?



If its one, match it to the user keyspace, and restart the node.



*From:* sha p [mailto:shatestt...@gmail.com]
*Sent:* 19 September 2018 11:49
*To:* user@cassandra.apache.org
*Subject:* Error during truncate: Cannot achieve consistency level ALL ,
how to fix it



Hi All,

 I am new to Cassandra. Following below link



https://grokonez.com/spring-framework/spring-data/start-spring-data-cassandra-springboot#III_Sourcecode
<https://urldefense.proofpoint.com/v2/url?u=https-3A__grokonez.com_spring-2Dframework_spring-2Ddata_start-2Dspring-2Ddata-2Dcassandra-2Dspringboot-23III-5FSourcecode=DwMFaQ=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8=j2h9K99DL76u4mWIOBsOk5lr41V4bADBYHYktY6e154=LzDHllUpsVHGyHiYwEkCXDyapbFOatkyPSmPsiFaCFo=>





I have three node cluster , keyspace set with RF = 2 , but when I run this
application from above source code bellow error is thrown

 Caused by: com.datastax.driver.core.exceptions.TruncateException: Error
during truncate: Cannot achieve consistency level ALL """ 





What wrong i am doing here ..How to fix it ? Plz help me.



Regards,

Shyam



This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy it. Any unauthorized
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden. Tradeweb reserves the right to monitor all e-mail
communications through its networks. If you do not wish to receive
marketing emails about our products / services, please let us know by
contacting us, either by email at contac...@tradeweb.com or by writing to
us at the registered office of Tradeweb in the UK, which is: Tradeweb
Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y
9DT. To see our privacy policy, visit our website @ www.tradeweb.com.

Error during truncate: Cannot achieve consistency level ALL , how to fix it

[sha p]

>
> Hi All,
>>>  I am new to Cassandra. Following below link
>>>
>>>
>>> https://grokonez.com/spring-framework/spring-data/start-spring-data-cassandra-springboot#III_Sourcecode
>>>
>>>
>>> I have three node cluster , keyspace set with RF = 2 , but when I run
>>> this application from above source code bellow error is thrown
>>>  Caused by: com.datastax.driver.core.exceptions.TruncateException: Error
>>> during truncate: Cannot achieve consistency level ALL """ 
>>>
>>>
>>> What wrong i am doing here ..How to fix it ? Plz help me.
>>>
>>> Regards,
>>> Shyam
>>>
>>

Re: Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Hi All,
 Is there any free Cassandra IDEs available? If so please share?

what is advantages of enterprise edition IDEs?

Regards,
Shyam

On Wed, Sep 5, 2018 at 6:51 PM sha p  wrote:

>
> Hi all ,
>> Me new to Cassandra , i was asked to migrate data from Oracle to
>> Cassandra.
>> Please help me giving your valuable guidance.
>> 1) Can it be done using open source Cassandra.
>> 2) Where should I start data model from?
>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>> 4) How I decide the size of cluster , please provide some sample
>> guidelines.
>> 5) this should be in production , so what kind of things i should take
>> care for better support or debugging tomorrow?
>> 6) Please provide some good books /links which can help me in this task.
>>
>>
>> Thanks in advance.
>> Highly appreciated your every amal help.
>>
>> Regards,
>> Shyam
>>
>

Re: Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Thanks a lot Rahul sir,
Where can I find more information ? any blog or links on that ?

On Mon, Sep 10, 2018 at 6:52 PM Rahul Singh 
wrote:

> Look into Kafka Connect. It does tracking internally in a topic. Works
> better going from relational to Cassandra.
>
> Still won’t fix your potential data model issue related to skew and wide
> partitions.
>
> Rahul Singh
> Chief Executive Officer
> m 202.905.2818
>
> Anant Corporation
> 1010 Wisconsin Ave NW, Suite 250
> Washington, D.C. 20007
>
> We build and manage digital business technology platforms.
> On Sep 6, 2018, 9:20 AM -0400, sha p , wrote:
>
> Thank you Jeff.
> While migration , how can test/validate against Cassandra particularly i
> am going for "parallel run". Any sample strategy?
>
>
> Regards,
> Shyam
>
> On Thu, 6 Sep 2018, 09:48 Jeff Jirsa,  wrote:
>
>> It very much depends on your application. You'll PROBABLY want to double
>> write for some period of time -  start writes to both Cassandra and Oracle,
>> and then ensure they're both in sync. Once you're sure they're both in
>> sync, move your reads from Oracle to Cassandra.
>>
>>
>>
>> On Wed, Sep 5, 2018 at 8:58 PM sha p  wrote:
>>
>>> Hi all,
>>> Sir how should I keep track of the data which is moved to Cassandra ,
>>> what are the best strategies available?
>>>
>>> Regards,
>>> Shyam
>>>
>>> On Wed, 5 Sep 2018, 18:51 sha p,  wrote:
>>>
>>>>
>>>> Hi all ,
>>>>> Me new to Cassandra , i was asked to migrate data from Oracle to
>>>>> Cassandra.
>>>>> Please help me giving your valuable guidance.
>>>>> 1) Can it be done using open source Cassandra.
>>>>> 2) Where should I start data model from?
>>>>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>>>>> 4) How I decide the size of cluster , please provide some sample
>>>>> guidelines.
>>>>> 5) this should be in production , so what kind of things i should take
>>>>> care for better support or debugging tomorrow?
>>>>> 6) Please provide some good books /links which can help me in this
>>>>> task.
>>>>>
>>>>>
>>>>> Thanks in advance.
>>>>> Highly appreciated your every amal help.
>>>>>
>>>>> Regards,
>>>>> Shyam
>>>>>
>>>>

Re: Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Thank you Jeff.
While migration , how can test/validate against Cassandra particularly i am
going for "parallel run". Any sample strategy?


Regards,
Shyam

On Thu, 6 Sep 2018, 09:48 Jeff Jirsa,  wrote:

> It very much depends on your application. You'll PROBABLY want to double
> write for some period of time -  start writes to both Cassandra and Oracle,
> and then ensure they're both in sync. Once you're sure they're both in
> sync, move your reads from Oracle to Cassandra.
>
>
>
> On Wed, Sep 5, 2018 at 8:58 PM sha p  wrote:
>
>> Hi all,
>> Sir how should I keep track of the data which is moved to Cassandra ,
>> what are the best strategies available?
>>
>> Regards,
>> Shyam
>>
>> On Wed, 5 Sep 2018, 18:51 sha p,  wrote:
>>
>>>
>>> Hi all ,
>>>> Me new to Cassandra , i was asked to migrate data from Oracle to
>>>> Cassandra.
>>>> Please help me giving your valuable guidance.
>>>> 1) Can it be done using open source Cassandra.
>>>> 2) Where should I start data model from?
>>>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>>>> 4) How I decide the size of cluster , please provide some sample
>>>> guidelines.
>>>> 5) this should be in production , so what kind of things i should take
>>>> care for better support or debugging tomorrow?
>>>> 6) Please provide some good books /links which can help me in this task.
>>>>
>>>>
>>>> Thanks in advance.
>>>> Highly appreciated your every amal help.
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>

Re:

[sha p]

Thank you so much Andrew.  I will start reading it.

On Thu, 6 Sep 2018, 10:26 Andrew Baker,  wrote:

> Hi Shyam,
>
>   Those are big questions! The book *Cassandra: The Definitive Guide *is
> a good place to start, it will walk you through a little bit of each of
> those questions. It should be a challenging project. Look around at
> http://cassandra.apache.org/ and Datastax has some good tutorials and
> videos too as I recall.
>
> -Andrew
>
> On Wed, Sep 5, 2018 at 6:19 AM sha p  wrote:
>
>> Hi all ,
>> Me new to Cassandra , i was asked to migrate data from Oracle to
>> Cassandra.
>> Please help me giving your valuable guidance.
>> 1) Can it be done using open source Cassandra.
>> 2) Where should I start data model from?
>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>> 4) How I decide the size of cluster , please provide some sample
>> guidelines.
>> 5) this should be in production , so what kind of things i should take
>> care for better support or debugging tomorrow?
>> 6) Please provide some good books /links which can help me in this task.
>>
>>
>> Thanks in advance.
>> Highly appreciated your every amal help.
>>
>> Regards,
>> Shyam
>>
>

Re: Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Hi all,
Sir how should I keep track of the data which is moved to Cassandra , what
are the best strategies available?

Regards,
Shyam

On Wed, 5 Sep 2018, 18:51 sha p,  wrote:

>
> Hi all ,
>> Me new to Cassandra , i was asked to migrate data from Oracle to
>> Cassandra.
>> Please help me giving your valuable guidance.
>> 1) Can it be done using open source Cassandra.
>> 2) Where should I start data model from?
>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>> 4) How I decide the size of cluster , please provide some sample
>> guidelines.
>> 5) this should be in production , so what kind of things i should take
>> care for better support or debugging tomorrow?
>> 6) Please provide some good books /links which can help me in this task.
>>
>>
>> Thanks in advance.
>> Highly appreciated your every amal help.
>>
>> Regards,
>> Shyam
>>
>

Re: [EXTERNAL] Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

Thank you all very much.
Migration is due to Oracle not scaling as expected.

Sure I will be posting my queries and doubts time to time for your guidance.


Thank you.
Shyam

On Wed, 5 Sep 2018, 21:28 Rahul Singh,  wrote:

> Look here for some “migration” or data modeling articles.
>
> *https://anant.github.io/awesome-cassandra/*
> <https://anant.github.io/awesome-cassandra/>
>
> Rahul Singh
> Chief Executive Officer
> m 202.905.2818
>
> Anant Corporation
> 1010 Wisconsin Ave NW, Suite 250
> Washington, D.C. 20007
>
> We build and manage digital business technology platforms.
> On Sep 5, 2018, 10:47 AM -0500, Jeff Jirsa , wrote:
>
> All of  Sean's points are good, a few more:
> - Apache Cassandra (free, open source, official) is usually sufficient.
> DSE may be faster, but really it's about whether or not you're willing to
> pay for support. If you're trying to stop paying Oracle, I suspect you'd
> probably not want to start paying someone else - try the free version
> first, and you can look for proprietary options after that.
> - http://shop.oreilly.com/product/0636920043041.do is relatively recent
> and mostly pretty good
> - Ask a lot of questions, use this list, but try things out first so
> people have a way to point you in the right direction.
>
>
>
> On Wed, Sep 5, 2018 at 7:58 AM Durity, Sean R 
> wrote:
>
>> 3 starting points:
>>
>> -  DO NOT migrate your tables as they are in Oracle to
>> Cassandra. In most cases, you need a different model for Cassandra
>>
>> -  DO take the (free) DataStax Academy courses to learn much
>> more about Cassandra as you dive in. It is a systematic and bite-size
>> approach to learning all things Cassandra (and eventually, DataStax
>> Enterprise, should you go that way). However, open source Cassandra is fine
>> as a data platform. DSE gives you more options for data models, better
>> administration and monitoring tools, support, etc. It all depends on what
>> you need/want to build/can afford
>>
>> -  Cluster sizing depends on your goals for the data platform.
>> Do you need lots of storage, lots of throughput, high availability, low
>> latency, workload separation, etc.? A couple guidelines – use at least 3
>> nodes per data center (DC) and at least 2 DCs for availability. Use SSDs
>> for storage and keep node size 3 TB or less for reasonable administration.
>> If six nodes are too many – you probably don’t need Cassandra. If you can
>> define what you need your data platform to deliver, then you can start a
>> sizing discussion. The good thing is, you can always scale (as long as the
>> data model is good).
>>
>>
>>
>>
>>
>> Sean Durity
>>
>>
>>
>> *From:* sha p 
>> *Sent:* Wednesday, September 05, 2018 9:21 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Regarding migrating data from Oracle to
>> Cassandra.migrate data from Oracle to Cassandra.
>>
>>
>>
>>
>>
>> Hi all ,
>>
>> Me new to Cassandra , i was asked to migrate data from Oracle to
>> Cassandra.
>>
>> Please help me giving your valuable guidance.
>>
>> 1) Can it be done using open source Cassandra.
>>
>> 2) Where should I start data model from?
>>
>> 3) I should use java, what kind of  jar/libs/tools I need use ?
>>
>> 4) How I decide the size of cluster , please provide some sample
>> guidelines.
>>
>> 5) this should be in production , so what kind of things i should take
>> care for better support or debugging tomorrow?
>>
>> 6) Please provide some good books /links which can help me in this task.
>>
>>
>>
>>
>>
>> Thanks in advance.
>>
>> Highly appreciated your every amal help.
>>
>>
>>
>> Regards,
>>
>> Shyam
>>
>>
>> --
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>

Regarding migrating data from Oracle to Cassandra.migrate data from Oracle to Cassandra.

[sha p]

> Hi all ,
> Me new to Cassandra , i was asked to migrate data from Oracle to Cassandra.
> Please help me giving your valuable guidance.
> 1) Can it be done using open source Cassandra.
> 2) Where should I start data model from?
> 3) I should use java, what kind of  jar/libs/tools I need use ?
> 4) How I decide the size of cluster , please provide some sample
> guidelines.
> 5) this should be in production , so what kind of things i should take
> care for better support or debugging tomorrow?
> 6) Please provide some good books /links which can help me in this task.
>
>
> Thanks in advance.
> Highly appreciated your every amal help.
>
> Regards,
> Shyam
>

[no subject]

[sha p]

Hi all ,
Me new to Cassandra , i was asked to migrate data from Oracle to Cassandra.
Please help me giving your valuable guidance.
1) Can it be done using open source Cassandra.
2) Where should I start data model from?
3) I should use java, what kind of  jar/libs/tools I need use ?
4) How I decide the size of cluster , please provide some sample guidelines.
5) this should be in production , so what kind of things i should take care
for better support or debugging tomorrow?
6) Please provide some good books /links which can help me in this task.


Thanks in advance.
Highly appreciated your every amal help.

Regards,
Shyam

Re: Sorl/DSE Spark

[sha p]

Got it.
Thank you so for your detailed explanation.

Regards,
Shyam

On Thu, 12 Apr 2018, 17:37 Evelyn Smith, <u5015...@gmail.com> wrote:

> Cassandra tends to be used in a lot of web applications. It’s loads are
> more natural and evenly distributed. Like people logging on throughout the
> day. And people operating it tend to be latency sensitive.
>
> Spark on the other hand will try and complete it’s tasks as quickly as
> possible. This might mean bulk reading from the Cassandra at 10 times the
> usual operations load, but for only say 5 minutes every half hour (however
> long it takes to read in the data for a job and whenever that job is run).
> In this case during that 5 minutes your normal operations work (customers)
> are going to experience a lot of latency.
>
> This even happens with streaming jobs, every time spark goes to interact
> with Cassandra it does so very quickly, hammers it for reads and then does
> it’s own stuff until it needs to write things out. This might equate to
> intermittent latency spikes.
>
> In theory, you can throttle your reads and writes but I don’t know much
> about this and don’t see people actually doing it.
>
> Regards,
> Evelyn.
>
> On 12 Apr 2018, at 4:30 pm, sha p <shatestt...@gmail.com> wrote:
>
> Evelyn,
> Can you please elaborate on below
> Spark is notorious for causing latency spikes in Cassandra which is not
> great if you are are sensitive to that.
>
>
> On Thu, 12 Apr 2018, 10:46 Evelyn Smith, <u5015...@gmail.com> wrote:
>
>> Are you building a search engine -> Solr
>> Are you building an analytics function -> Spark
>>
>> I feel they are used in significantly different use cases, what are you
>> trying to build?
>>
>> If it’s an analytics functionality that’s seperate from your operations
>> functionality I’d build it in it’s own DC. Spark is notorious for causing
>> latency spikes in Cassandra which is not great if you are are sensitive to
>> that.
>>
>> Regards,
>> Evelyn.
>>
>> On 12 Apr 2018, at 6:55 am, kooljava2 <koolja...@yahoo.com.INVALID>
>> wrote:
>>
>> Hello,
>>
>> We are exploring on configuring Sorl/Spark. Wanted to get input on this.
>> 1) How do we decide which one to use?
>> 2) Do we run this on a DC where there is less workload?
>>
>> Any other suggestion or comments are appreciated.
>>
>> Thank you.
>>
>>
>>
>

Re: Sorl/DSE Spark

[sha p]

Evelyn,
Can you please elaborate on below
Spark is notorious for causing latency spikes in Cassandra which is not
great if you are are sensitive to that.


On Thu, 12 Apr 2018, 10:46 Evelyn Smith,  wrote:

> Are you building a search engine -> Solr
> Are you building an analytics function -> Spark
>
> I feel they are used in significantly different use cases, what are you
> trying to build?
>
> If it’s an analytics functionality that’s seperate from your operations
> functionality I’d build it in it’s own DC. Spark is notorious for causing
> latency spikes in Cassandra which is not great if you are are sensitive to
> that.
>
> Regards,
> Evelyn.
>
> On 12 Apr 2018, at 6:55 am, kooljava2  wrote:
>
> Hello,
>
> We are exploring on configuring Sorl/Spark. Wanted to get input on this.
> 1) How do we decide which one to use?
> 2) Do we run this on a DC where there is less workload?
>
> Any other suggestion or comments are appreciated.
>
> Thank you.
>
>
>

Re: No columns are defined for Materialized View other than primary key

[sha p]

There is one more column "data" here in MView?

On 7 Sep 2017 7:49 p.m., "DuyHai Doan"  wrote:

> The answer of your question is in the error message. For once it's very
> clear. The primary key of your materialized view is EXACTLY the same as for
> your base table.
>
> So the question is what's the point creating this materialized view ...
>
>
>
> On Thu, Sep 7, 2017 at 4:01 PM, Alex Kotelnikov <
> alex.kotelni...@diginetica.com> wrote:
>
>> Hey. I have a problem creating a materialized view.
>>
>> My case is quite similar to
>> https://issues.apache.org/jira/browse/CASSANDRA-13564
>> but discussion in comments there faded, let me describe by case.
>>
>> I have a table like
>> CREATE TABLE users (
>>   site_id int,
>>   user_id text,
>>   n int,
>>   data set,
>>   PRIMARY KEY ((site_id, user_id), n));
>>
>> user data is updated and read by PK and sometimes I have to fetch all
>> user for some specific site_id. It appeared that full scan by
>> token(site_id,user_id) filtered by WHERE site_id =  works much
>> slower than unfiltered full scan on
>> CREATE MATERIALIZED VIEW users_1 AS
>> SELECT site_id, user_id, n, data
>> FROM users
>> WHERE site_id = 1 AND user_id IS NOT NULL AND n IS NOT NULL
>> PRIMARY KEY ((site_id, user_id), n);
>>
>> yes, you have to do so for each site_id, but it makes such bulk fetches
>> much faster. (When I do so, I am always puzzled, why I have to put NOT NULL
>> for a part of a primary key).
>> And just in case, I tried secondary indices on site_id. For such use they
>> improve nothing.
>>
>>
>> But things are changing and we realized that we want to get rid of
>> clustering key, n.
>>
>> DROP MATERIALIZED VIEW users_1;
>> DROP TABLE users;
>>
>> CREATE TABLE users (
>> site_id int,
>> user_id text,
>> data set,
>> PRIMARY KEY ((site_id, user_id)));
>>
>> CREATE MATERIALIZED VIEW users_1 AS
>> SELECT site_id, user_id, data
>> FROM users
>> WHERE site_id = 1 AND user_id IS NOT NULL
>> PRIMARY KEY ((site_id, user_id));
>>
>> And here I get the error I listed in the subject.
>> InvalidRequest: Error from server: code=2200 [Invalid query] message="No
>> columns are defined for Materialized View other than primary key"
>>
>> But why? I still expect scans to be faster with MV. It appears to be
>> possible to create a dummy column and using as a clustering key. That's
>> ugly.
>> --
>>
>> Best Regards,
>>
>>
>> *Alexander Kotelnikov*
>>
>> *Team Lead*
>>
>> DIGINETICA
>> Retail Technology Company
>>
>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>
>> *www.diginetica.com *
>>
>
>