For erlang proplist object, extractor is needed?

[Hao]

If I want to save Erlang proplist into Riak and use Riak Search 2.0, do I need 
to create a Erlang Proplist extractor for solr to be able to index it?


--
Hao

  ___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

How to fix an stale index situation after a bitcask data restore

[Hao]

Single node with search=on, I previously indexed 2 map objects. Then I did a 
restore of my previous bitcask folder. I can see newly created bucket type 
still there. And when I search by index, the index data are also there. So I 
removed the index folder (under /var/lib/riak/yz/todoriak_main_movie_idx/data ) 
, riak should start to index anew. But it turned out, no.The index folder was 
re-created by Riak though.


When I search by index, I still see the old 2 map objects. And when I add new 
objects, they are not indexed at all. Always the old 2 map objects only.

I thought AAE will fix index. No? Is the AAE only for cluster?
What can I do now? Delete index and re-create? What's the business of re-attach?

Thank you
-Hao
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Number of buckets...

[Dmitri Zagidulin]

Hi Joe,

For most use cases, there would be no limit to the number of buckets you
can have on a level db cluster. (Aside from obvious limits of, eventually
you'd run out of disk space for all the objects).

Riak essentially treats the bucket as merely a prefix for the key. (It
basically concatenates the bucket type, the bucket, and the key, and then
hashes /that/ to determine where to place the object on the ring). So,
there isn't a data structure anywhere on disk or in memory that holds a
list of buckets on the cluster, to impose a limit.

The exception to that is when you set non-standard properties on specific
buckets. For example, if you have siblings turned on for all of your
buckets, but you have a 'logs' bucket that will only have immutable data,
it makes sense to set allow_mult=false for just the logs bucket, using the
Set Bucket Properties api call (for example,
http://docs.basho.com/riak/latest/dev/references/http/set-bucket-props/ ).

Setting custom properties for a handful of buckets is not a problem. But if
you decide to do that for hundreds of buckets, it becomes an issue. Whereas
before, buckets were just a "stateless" prefix to the key, now the custom
properties do actually live in a data structure on the cluster, and have to
be gossiped around the ring. And as you add more custom buckets, that
gossiped structure becomes larger and larger.

If you do need to set custom properties on hundreds of buckets (such as
specifying a Search index on them, for example), consider using Bucket
Types instead. Bucket Type custom settings and metadata use a different
persistence mechanism than bucket settings, and there's a lot more room for
custom types.  In addition, once you set up a custom bucket type, all of
the buckets /within/ that type get those settings essentially for free. See
http://docs.basho.com/riak/latest/dev/advanced/bucket-types/ for more
discussion.

The other exception that the docs mention (about the file handles etc),
comes into play when you use a custom Multi backend (see
http://docs.basho.com/riak/latest/ops/advanced/backends/multi/ ) for
individual buckets.
You can, for example, say in the config that you want your 'users' bucket
to live in a custom LevelDB backend, separate from the rest of your levelDB
data. And it'll have a separate data directory (which you can house on a
different disk, etc). Which is where the file handle limit comes in. If you
intend to spin up multiple custom backends on your cluster, you want to be
sure the file system can handle opening that many files. The cases where
you'd want to do that are extremely rare, though.

So, to summarize: In most cases, there is no theoretical limit on the
number of buckets you can have.
If you must have custom settings on individual buckets, think tens and not
hundreds or thousands. Better yet, use custom settings on Bucket Types,
instead.

Does that answer your question?

Dmitri

On Fri, Aug 7, 2015 at 10:56 AM, Joe Olson  wrote:

> From http://docs.basho.com/riak/latest/theory/concepts/:
>
> "In general, large numbers of buckets within a Riak cluster is not a
> problem. In practice, there are two potential restrictions on the maximum
> number of buckets a Riak cluster can handle"
>
> Is there any further, more specific documentation on this that I could
> read?
>
> I am interested in understanding the upper limits of the number of buckets
> for say, a 10 node, ring_size = 64 (or 128) LevelDB backed Riak cluster.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Re: Riak shuts down itself after couple of minutes

[Cuyler Jones]

Hi Hao --

As Alex said, the error you're receiving is generally related to open file
limits set by your OS configuration. Riak requires a large number of
available open files handles. You can find information on how to up your
limits here: http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/

Hope that helps!

Thanks!

Cuyler

On Fri, Aug 7, 2015 at 9:13 AM, Hao  wrote:

> Which one? I am bit of lost.
>
> [root@localhost riak_data]# cat /proc/sys/fs/file-max
> 786810
> [root@localhost riak_data]# ulimit -Hn
> 4096
> [root@localhost riak_data]# ulimit -Sn
> 1024
>
> --Hao
>
>
>
> 在 2015-08-07 23:58:56，"Alex Moore"  写道：
>
> Hi Hao,
>
> Looks like you might be running into an EMFILE error:
> http://docs.basho.com/riak/latest/community/faqs/logs/#riak-logs-contain-error-emfile-in-the-message
>
> What’s the open files limit on your nodes?
>
> Thanks,
> Alex
>
> On Aug 7, 2015, at 11:55 AM, 王昊  wrote:
>
> I am running it on a single node.
>
> Using Erlang riak client. I added a map bucket type, create a new index,
> set the index on a bucket, saved a few map data into the bucket. Then I did
> some search. All good. Then I can remember I tried a invalid search query
> like "title_s_register:[New movie one]" which shuts down Riak. But it may
> or may not be the first time Riak starts to shut down. I can't remember.
>
> Now Riak itself shut down after running a few minutes.  I must have done
> something really wrong. Any idea what I can do? I have tired to restore
> bitcask data folder from a backup from days ago. It didn't help. It still
> crashes.
>
> The console.log is here : http://www.pastebin.ca/3092510
> The error begins at Line 172
>
> Anyone can help? Much appreciated.
>
> -Hao
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re:Re: Riak shuts down itself after couple of minutes

[Hao]

Which one? I am bit of lost.

[root@localhost riak_data]# cat /proc/sys/fs/file-max
786810
[root@localhost riak_data]# ulimit -Hn
4096
[root@localhost riak_data]# ulimit -Sn
1024



--Hao

  

在 2015-08-07 23:58:56，"Alex Moore"  写道：
Hi Hao,


Looks like you might be running into an EMFILE error:  
http://docs.basho.com/riak/latest/community/faqs/logs/#riak-logs-contain-error-emfile-in-the-message


What’s the open files limit on your nodes?


Thanks,
Alex


On Aug 7, 2015, at 11:55 AM, 王昊  wrote:


I am running it on a single node.

Using Erlang riak client. I added a map bucket type, create a new index, set 
the index on a bucket, saved a few map data into the bucket. Then I did some 
search. All good. Then I can remember I tried a invalid search query like 
"title_s_register:[New movie one]" which shuts down Riak. But it may or may not 
be the first time Riak starts to shut down. I can't remember.

Now Riak itself shut down after running a few minutes.  I must have done 
something really wrong. Any idea what I can do? I have tired to restore bitcask 
data folder from a backup from days ago. It didn't help. It still crashes.

The console.log is here : http://www.pastebin.ca/3092510 
The error begins at Line 172

Anyone can help? Much appreciated.

-Hao
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak shuts down itself after couple of minutes

[Alex Moore]

Hi Hao,

Looks like you might be running into an EMFILE error:  
http://docs.basho.com/riak/latest/community/faqs/logs/#riak-logs-contain-error-emfile-in-the-message
 


What’s the open files limit on your nodes?

Thanks,
Alex

> On Aug 7, 2015, at 11:55 AM, 王昊  wrote:
> 
> I am running it on a single node.
> 
> Using Erlang riak client. I added a map bucket type, create a new index, set 
> the index on a bucket, saved a few map data into the bucket. Then I did some 
> search. All good. Then I can remember I tried a invalid search query like 
> "title_s_register:[New movie one]" which shuts down Riak. But it may or may 
> not be the first time Riak starts to shut down. I can't remember.
> 
> Now Riak itself shut down after running a few minutes.  I must have done 
> something really wrong. Any idea what I can do? I have tired to restore 
> bitcask data folder from a backup from days ago. It didn't help. It still 
> crashes.
> 
> The console.log is here : http://www.pastebin.ca/3092510 
> 
> The error begins at Line 172
> 
> Anyone can help? Much appreciated.
> 
> -Hao
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak shuts down itself after couple of minutes

[王昊]

I am running it on a single node.

Using Erlang riak client. I added a map bucket type, create a new index, set 
the index on a bucket, saved a few map data into the bucket. Then I did some 
search. All good. Then I can remember I tried a invalid search query like 
"title_s_register:[New movie one]" which shuts down Riak. But it may or may not 
be the first time Riak starts to shut down. I can't remember.

Now Riak itself shut down after running a few minutes.  I must have done 
something really wrong. Any idea what I can do? I have tired to restore bitcask 
data folder from a backup from days ago. It didn't help. It still crashes.

The console.log is here : http://www.pastebin.ca/3092510 
The error begins at Line 172

Anyone can help? Much appreciated.

-Hao___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Number of buckets...

[Joe Olson]

>From http://docs.basho.com/riak/latest/theory/concepts/: 

"In general, large numbers of buckets within a Riak cluster is not a problem. 
In practice, there are two potential restrictions on the maximum number of 
buckets a Riak cluster can handle" 

Is there any further, more specific documentation on this that I could read? 

I am interested in understanding the upper limits of the number of buckets for 
say, a 10 node, ring_size = 64 (or 128) LevelDB backed Riak cluster. 
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com