[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[eiri]

Github user eiri commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209616524
  
@rnewson View compaction files handled in `couch_file:deleted/3` function 
which does what the first PR comment describes and you are commenting on 
`couch_file:delete_dir/2` which is called on database's views directories when 
database deleted while parameter `rename_on_delete` set to false 
[[ref]](https://github.com/cloudant/couchdb-couch/blob/61028-fix-rename-on-delete/src/couch_file.erl#L258)

The scope of PR is to extend exiting `rename_on_delete` behaviour on 
compaction and view files. I'm doing this by making views directories to be 
renamed on deletion of a database in the same manner the database file renamed 
right now. For compaction files I'm making them not to be deleted in `.deleted` 
directory after renaming, the same way as Cloudant does. I'm renaming them 
differently, though, so they'd be more easily identified if necessary.

The renaming configuration parameters or radically diverging from how file 
preservation works right now is not in scope of this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[kxepal]

Github user kxepal commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209598178
  
Hm...reading the links, references and quick googling shows that this 
problem is extX specific and other FS may handle delete right without IOpain. 
Feels like this algorithm better to make configurable and notify admins about 
such problem and the option, but I would like to make some benchmarks for FS I 
uses for CouchDB: XFS and ZFS. Though for XFS this [shouldn't be a 
problem](http://www.bullopensource.org/ext4/20070404/FileDeletion.html) already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (COUCHDB-2988) Allow query selector as changes and replication filter

[Nick Vatamaniuc (JIRA)]

Nick Vatamaniuc created COUCHDB-2988:


 Summary: Allow query selector as changes and replication filter
 Key: COUCHDB-2988
 URL: https://issues.apache.org/jira/browse/COUCHDB-2988
 Project: CouchDB
  Issue Type: Improvement
  Components: Database Core, Mango
Reporter: Nick Vatamaniuc






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[kxepal]

Github user kxepal commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209592830
  
@rnewson @davisp Thanks a lot! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[davisp]

Github user davisp commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209554817
  
I should've made the point that ionice doesn't work if you're using noop io 
scheduling.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[davisp]

Github user davisp commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209554642
  
@kxepal We actually stole the truncate-to-zero approach from a talk at one 
of the devops conferences some years ago. Its purely based on the observation 
that @rnewson made, when you delete a file, the kernel will consume a large 
amount of disk resources trying to remove that file. Seeing it in the talk made 
us realize why we ended up having such a performance impact when a compaction 
finished. Here's a couple quick links that describe similar solutions:

http://www.awasu.com/weblog/deleting-large-files-from-an-ext3-file-system/

http://serverfault.com/questions/480526/deleting-very-large-file-without-webserver-freezing

The ionice suggestion in the second link I'm pretty sure depends on having 
a kernel io scheduler configured and we've seen that have a performance impact 
with anything other than the noop scheduler.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Fix normalize_dbname to work with slas...

[davisp]

Github user davisp commented on the pull request:

https://github.com/apache/couchdb-couch/pull/158#issuecomment-209546028
  
This looks odd to me. So I have this right, an example:

Path1 == <<"shards/-/username/dbname.234234234">>
mem3:dbname(Path1) == <<"username/dbname">>
couch_db:normalize_dbname(Path1) == <<"dbname">>

We used the original normalize_dbname (though never by that name) to check 
if a db was a _replicator or _users db. The missing detail/assumption here was 
that we never cared if someone named a database like <<"foo/_replicator">> in 
so much as it'd just get picked up as a _replicator database. This patch and 
related changes appear to be wanting to avoid that when we've never cared 
before.

However I'd go about this fairly differently. normalize_dbname used to be 
this:

lists:last(binary:split(mem3:dbname(Path), <<"/">>, [global])).

I'd just remove the global option so we only split once.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[rnewson]

Github user rnewson commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209543364
  
@kxepal it was an observation from production but I'm sure you can 
trivially reproduce it. make a 200gb file on ext4, delete it while running any 
kind of benchmark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[kxepal]

Github user kxepal commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209526371
  
@rnewson Interesting. What would you recommend to read about? (sorry for 
offtopic)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[rnewson]

Github user rnewson commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209525568
  
@eiri the first PR comment says 'This change makes compaction files to stay 
in .deleted directory in case config parameter rename_on_delete is set and 
renames them in fashion, that makes it easier to restore them if necessary. ' 
but what you've done is rename-then-delete which does not achieve the goal.

I think rename_on_delete might be a misleading name. 
rename_instead_of_delete is the meaning. Wherever we delete a file, whether 
that's a database or view compaction that completes or a user deletes a 
database, we want one of two behaviours: 1) we delete the file immediately 
(yes, we'll mv it to the .delete directory first so that we can clean up easily 
in the event of a crash, but that's normal couchdb behaviour) or 2) we rename 
the file (appending a unique timestamp or similar) and do not delete it, so 
that it can be recovered. Anyone that sets this config flag is then required to 
find and delete these files later if they don't intend to restore them (so a 
useful pattern like *.deleted.couch / *.deleted.view is warranted).

Have I misunderstood the scope of this pull request?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-couch pull request: Extend rename_on_delete behaviour on c...

[rnewson]

Github user rnewson commented on the pull request:

https://github.com/apache/couchdb-couch/pull/161#issuecomment-209524096
  
@kxepal: because the kernel/filesystem will devote itself to freeing the 
blocks, it causes all other reads/writes to suffer hugely. It _is_ more 
expensive to repeatedly truncate vs simple delete, but it's spaced out over 
time so affects other operations considerably less.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: Improve Less error display

[benkeen]

Github user benkeen closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/680


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (COUCHDB-2987) Mango Python tests failure

[Nick Vatamaniuc (JIRA)]

Nick Vatamaniuc created COUCHDB-2987:


 Summary: Mango Python tests failure
 Key: COUCHDB-2987
 URL: https://issues.apache.org/jira/browse/COUCHDB-2987
 Project: CouchDB
  Issue Type: Bug
  Components: Mango
Reporter: Nick Vatamaniuc


Saw this tests failure running mango's test suit:

{code}
$ nosetests
S...SF..SSS.S.S...SSS
==
FAIL: test_empty_subsel_match (02-basic-find-test.BasicFindTests)
--
Traceback (most recent call last):
  File "/Users/nvatama/asf/couchdb/src/mango/test/02-basic-find-test.py", line 
256, in test_empty_subsel_match
assert len(docs) == 1
AssertionError:
 >> begin captured logging << 
requests.packages.urllib3.connectionpool: DEBUG: "POST 
/mango_test_b7fb2baf897741a288e8174971ef388c/_bulk_docs HTTP/1.1" 201 97
requests.packages.urllib3.connectionpool: DEBUG: "POST 
/mango_test_b7fb2baf897741a288e8174971ef388c/_find HTTP/1.1" 200 None
- >> end captured logging << -

--
Ran 137 tests in 51.613s

FAILED (SKIP=90, failures=1)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb-fauxton pull request: Joy of testing

[benkeen]

Github user benkeen closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/684


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: fork mocha-phantomjs for now

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb-fauxton/pull/701


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: fork mocha-phantomjs for now

[robertkowalski]

Github user robertkowalski commented on the pull request:

https://github.com/apache/couchdb-fauxton/pull/701#issuecomment-209477465
  
+1 when green


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-fauxton pull request: fork mocha-phantomjs for now

[garrensmith]

GitHub user garrensmith opened a pull request:

https://github.com/apache/couchdb-fauxton/pull/701

fork mocha-phantomjs for now



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/garrensmith/couchdb-fauxton 
fork-mocha-phantomjs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-fauxton/pull/701.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #701


commit 33538c0b4518eadf5271970fcca7d9fb663e8bf9
Author: Garren Smith 
Date:   2016-04-13T14:06:31Z

fork mocha-phantomjs for now




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] couchdb-mem3 pull request: Improve mem3_sync event listener perfor...

[kocolosk]

Github user kocolosk commented on a diff in the pull request:

https://github.com/apache/couchdb-mem3/pull/19#discussion_r59537173
  
--- Diff: src/mem3_sync_event_listener.erl ---
@@ -51,35 +68,136 @@ handle_event(NodesDb, updated, #state{nodes = NodesDb} 
= St) ->
 Nodes = mem3:nodes(),
 Live = nodes(),
 [mem3_sync:push(NodesDb, N) || N <- Nodes, lists:member(N, Live)],
-{ok, St};
+maybe_push_shards(St);
 handle_event(ShardsDb, updated, #state{shards = ShardsDb} = St) ->
 mem3_sync:push(ShardsDb, mem3_sync:find_next_node()),
-{ok, St};
+maybe_push_shards(St);
 handle_event(UsersDb, updated, #state{users = UsersDb} = St) ->
 mem3_sync:push(UsersDb, mem3_sync:find_next_node()),
-{ok, St};
+maybe_push_shards(St);
 handle_event(<<"shards/", _/binary>> = ShardName, updated, St) ->
-try mem3:shards(mem3:dbname(ShardName)) of
-Shards ->
-Targets = [S || #shard{node=N, name=Name} = S <- Shards,
-N =/= node(), Name =:= ShardName],
-Live = nodes(),
-[mem3_sync:push(ShardName,N) || #shard{node=N} <- Targets,
-lists:member(N, Live)]
-catch error:database_does_not_exist ->
-ok
-end,
-{ok, St};
+Buckets = bucket_shard(ShardName, St#state.buckets),
+maybe_push_shards(St#state{buckets=Buckets});
 handle_event(<<"shards/", _:18/binary, _/binary>> = ShardName, deleted, 
St) ->
 mem3_sync:remove_shard(ShardName),
-{ok, St};
+maybe_push_shards(St);
 handle_event(_DbName, _Event, St) ->
-{ok, St}.
+maybe_push_shards(St).
 
+handle_cast({set_frequency, Frequency}, St) ->
+#state{delay = Delay, buckets = Buckets0} = St,
+Buckets1 = rebucket_shards(Delay, Frequency, Buckets0),
+maybe_push_shards(St#state{frequency=Frequency, buckets=Buckets1});
+handle_cast({set_delay, Delay}, St) ->
+#state{frequency = Frequency, buckets = Buckets0} = St,
+Buckets1 = rebucket_shards(Delay, Frequency, Buckets0),
+maybe_push_shards(St#state{delay=Delay, buckets=Buckets1});
 handle_cast(Msg, St) ->
 couch_log:notice("unexpected cast to mem3_sync_event_listener: ~p", 
[Msg]),
-{ok, St}.
+maybe_push_shards(St).
 
+handle_info(timeout, St) ->
+maybe_push_shards(St);
 handle_info(Msg, St) ->
 couch_log:notice("unexpected info to mem3_sync_event_listener: ~p", 
[Msg]),
+maybe_push_shards(St).
+
+handle_config_change("mem3", "sync_delay", Delay0, _, St) ->
+try list_to_integer(Delay0) of
+Delay1 ->
+couch_event_listener:cast(
+?MODULE,
+{set_delay, Delay1}
+)
+catch error:badarg ->
+couch_log:warning(
+"ignoring bad value for mem3.sync_delay: ~p",
+[Delay0]
+)
+end,
+{ok, St};
+handle_config_change("mem3", "sync_frequency", Frequency0, _, St) ->
+try list_to_integer(Frequency0) of
+Frequency1 ->
+couch_event_listener:cast(
+?MODULE,
+{set_frequency, Frequency1}
+)
+catch error:badarg ->
+couch_log:warning(
+"ignoring bad value for mem3.sync_frequency: ~p",
+[Frequency0]
+)
+end,
+{ok, St};
+handle_config_change(_, _, _, _, St) ->
 {ok, St}.
+
+handle_config_terminate(_, stop, _) -> ok;
+handle_config_terminate(_Server, _Reason, St) ->
+Fun = fun() ->
+timer:sleep(5000),
+config:listen_for_changes(?MODULE, St)
+end,
+spawn(Fun).
+
+bucket_shard(ShardName, [B|Bs]=Buckets0) ->
+case waiting(ShardName, Buckets0) of
+true -> Buckets0;
+false -> [sets:add_element(ShardName, B)|Bs]
+end.
--- End diff --

OK, with fresh eyes this morning I agree with you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (COUCHDB-2977) JavaScript and CSS files for fauxton can not be found

[Garren Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239017#comment-15239017
 ] 

Garren Smith commented on COUCHDB-2977:
---

Thanks for reporting Patrick. I will take a look at this and see whats 
happening. We have just moved over to webpack which I think should resolve this 
issue. But I will confirm first.

> JavaScript and CSS files for fauxton can not be found
> -
>
> Key: COUCHDB-2977
> URL: https://issues.apache.org/jira/browse/COUCHDB-2977
> Project: CouchDB
>  Issue Type: Bug
>  Components: Fauxton
>Reporter: Patrick Palacin
>Assignee: Garren Smith
>
> Hi folks,
> i followed these guide:
> https://docs.google.com/document/d/1BtndYr-0KDQTqBSLVdJoR_8C5ObYjT1RBo_Qyh5ykdQ/edit#
> installing as single node setup on latest alpha (15.3) . When i try now to 
> open _utils I get a blank page. Firefox devtools are showing:
> "NetworkError: 404 Object Not Found - 
> http://159.122.91.165:5984/_utils/dashboard.assets/js/require-c9def4126d9d299af5b771713151534f.js;
> require...534f.js
> "NetworkError: 404 Object Not Found - 
> http://159.122.91.165:5984/_utils/dashboard.assets/css/index-6772bc2fbb800cb971b29ad7f561.css;
> I am running on Ubuntu 16.04 LTS and installed all dependencies from the 
> standard repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] couchdb pull request: Update Fauxton to v1.1.1

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb/pull/399


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---