I am not sure why you want to have a BigTable PageStore since it would have quite negative performance implications (imho). you may consider also the presentation of Guido van Rossum

https://sites.google.com/site/appengineappstats/AppStats_Meetup09.pdf?attredirects=0


I attach an implementation of a Memcache-based PageStore to this mail. It is very much based on the code posted by Richard Wilkinson some time before. If you find any bugs or are able to improve it, please post it to the list.

regards,
andr


On 24.12.2009 18:43, sudhir543-...@yahoo.com wrote:


  Last comment on that page is from my self only :)
  It says.. I dont want to keep every thing in session.. and no one would want 
to.
I want 'GAE big table based PageStore'



Sudhir NimavatSenior software engineer.
Quick start global PVT LTD.
Baroda - 390007
Gujarat, India

Personally I'm always ready to learn, although I do not always like being taught





________________________________
From: Ilja Pavkovic<ilja.pavko...@binaere-bauten.de>
To: users@wicket.apache.org
Sent: Thu, 24 December, 2009 10:55:27 PM
Subject: Re: GAE big table PageStore

Hi,

http://lmgtfy.com/?q=google+app+engine+wicket

first link :)

Best Regards,
     Ilja Pavkovic


Am Donnerstag, 24. Dezember 2009 10:56:51 schrieb sudhir543-...@yahoo.com:
I need to write app specifically for GAE, I know Disk based page store
  wouldnt work. Is there any existing solution for this? I searched mailing
  lists and did googling, seems that it has been discussed earlier too, but
  I don't find any implementation.

If I can get a prebuilt solution, I want to avoid my self from writing.

SN





Sudhir NimavatSenior software engineer.
Quick start global PVT LTD.
Baroda - 390007
Gujarat, India

Personally I'm always ready to learn, although I do not always like being
  taught


       The INTERNET now has a personality. YOURS! See your Yahoo! Homepage.
  http://in.yahoo.com/



/*

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

*/


import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.Map.Entry;
import java.util.concurrent.locks.ReentrantReadWriteLock;

import org.apache.wicket.IClusterable;
import org.apache.wicket.Page;
import org.apache.wicket.protocol.http.pagestore.AbstractPageStore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.google.appengine.api.memcache.MemcacheService;
import com.google.appengine.api.memcache.MemcacheServiceFactory;

public class MemcachePageStore extends AbstractPageStore {

        private static final Logger logger = LoggerFactory
                        .getLogger(MemcachePageStore.class);
        private static final String PAGESTORE_MEMCACHE_KEY = 
"PAGESTORE_MEMCACHE_KEY";
        private final int MAX_PAGES_PER_MAP;

        private static final int NO_MAX_PAGES_PER_MAP = -99;

        private MemcacheService memcache;

        public MemcachePageStore() {
                logger.debug("New Memcache Page Store, MAX_PAGES_PER_MAP is 
Unlimited");
                MAX_PAGES_PER_MAP = MemcachePageStore.NO_MAX_PAGES_PER_MAP;
                this.initMemcache();
        }

        public MemcachePageStore(final int maxPagesPerMap) {
                if (logger.isDebugEnabled()) {
                        logger.debug("New Memcache Page Store, 
MAX_PAGES_PER_MAP is "
                                        + maxPagesPerMap);
                }
                MAX_PAGES_PER_MAP = maxPagesPerMap;
                this.initMemcache();
        }

        public boolean containsPage(final String sessionId,
                        final String pageMapName, final int pageId, final int 
pageVersion) {
                return getPageMapStore(sessionId).getPageStore(pageMapName)
                                .containsPage(pageId, pageVersion);
        }

        public void destroy() {
                // nothing to do - PageStores will be destroyed with their 
sessions
        }

        public Page getPage(final String sessionId, final String pagemap,
                        final int id, final int versionNumber, final int 
ajaxVersionNumber) {
                final SerializedPage sPage = 
getPageMapStore(sessionId).getPageStore(
                                pagemap).getPage(id, versionNumber, 
ajaxVersionNumber);
                return sPage != null ? deserializePage(sPage.getData(), 
versionNumber)
                                : null;
        }

        public void pageAccessed(final String sessionId, final Page page) {
                // do nothing
        }

        public void removePage(final String sessionId, final String pagemap,
                        final int id) {
                PageMapStore pms = getPageMapStore(sessionId);
                if (id == -1) {
                        if (logger.isDebugEnabled()) {
                                logger.debug("Remove page map: " + pagemap);
                        }
                        pms.removePageMap(pagemap);
                } else {
                        if (logger.isDebugEnabled()) {
                                logger
                                                .debug("Remove page: " + id + " 
from page map "
                                                                + pagemap);
                        }
                        pms.getPageStore(pagemap).removePage(id);
                }
                putInMemcache(sessionId, pms);
        }

        public void storePage(final String sessionId, final Page page) {
                List<SerializedPage> list = serializePage(page);
                PageMapStore pms = getPageMapStore(sessionId);
                PageStore ps = pms.getPageStore(page.getPageMapName());
                ps.storePages(list);
                putInMemcache(sessionId, pms);

                if (logger.isDebugEnabled()) {
                        logger.debug("Store page: " + page.toString());
                        logger.debug(getPageMapStore(sessionId).getPageStore(
                                        page.getPageMapName()).toString());

                }
        }

        public void unbind(final String sessionId) {
                memcache.delete(getPagestorePerSessionMemcacheKey(sessionId));
        }

        protected PageMapStore getPageMapStore(final String sessionId) {
                PageMapStore store = (PageMapStore) memcache
                                
.get(getPagestorePerSessionMemcacheKey(sessionId));
                if (store == null) {
                        store = new PageMapStore(MAX_PAGES_PER_MAP);
                        putInMemcache(sessionId, store);
                        if (logger.isDebugEnabled())
                                logger.debug("No Pagestore for sessionId " + 
sessionId
                                                + " found. Created a new one.");
                }
                return store;
        }

        private void initMemcache() {
                if (logger.isDebugEnabled())
                        logger.debug("Initializing Memcache");
                try {
                        memcache = MemcacheServiceFactory.getMemcacheService();
                } catch (Exception e) {
                        logger.error(
                                        "Exception occured when trying to 
initialize Memcache", e);
                        memcache = null;
                }
        }

        private String getPagestorePerSessionMemcacheKey(final String 
sessionId) {
                return PAGESTORE_MEMCACHE_KEY + sessionId;
        }

        private void putInMemcache(final String sessionId, PageMapStore pms) {
                memcache.put(getPagestorePerSessionMemcacheKey(sessionId), pms);
        }

        protected static class PageMapStore implements IClusterable {
                private static final long serialVersionUID = 1L;

                private final Map<String, PageStore> _pageMaps = new 
HashMap<String, PageStore>();
                private final ReentrantReadWriteLock _pageMapsLock = new 
ReentrantReadWriteLock();
                private final int MAX_PAGES_PER_MAP;

                public PageMapStore(final int maxNumPagesPerMap) {
                        MAX_PAGES_PER_MAP = maxNumPagesPerMap;
                }

                public PageStore getPageStore(final String pageMapName) {
                        _pageMapsLock.readLock().lock();
                        PageStore toReturn;
                        try {
                                toReturn = _pageMaps.get(pageMapName);
                        } finally {
                                _pageMapsLock.readLock().unlock();
                        }

                        if (toReturn == null) {
                                /*
                                 * create a new PageStore, note that another 
thread might have
                                 * added one while no lock was held
                                 */
                                _pageMapsLock.writeLock().lock();
                                try {
                                        final PageStore old = 
_pageMaps.put(pageMapName,
                                                        toReturn = new 
PageStore(MAX_PAGES_PER_MAP));
                                        if (old != null) {
                                                // already exists, revert and 
use existing
                                                toReturn = old;
                                                _pageMaps.put(pageMapName, 
toReturn);
                                        }
                                } finally {
                                        _pageMapsLock.writeLock().unlock();
                                }
                        }
                        return toReturn;
                }

                public void removePageMap(final String pagemap) {
                        _pageMapsLock.writeLock().lock();
                        try {
                                _pageMaps.remove(pagemap);
                        } finally {
                                _pageMapsLock.writeLock().unlock();
                        }
                }

                @Override
                public String toString() {
                        final StringBuilder sb = new StringBuilder();
                        for (final Entry<String, PageStore> entry : 
_pageMaps.entrySet()) {
                                sb.append("PageMap: 
").append(entry.getKey()).append("\n");
                                sb.append(entry.getValue().toString());
                        }
                        return sb.toString();
                }
        }

        protected static class PageStore implements IClusterable {
                private static final long serialVersionUID = 1L;

                private final ReentrantReadWriteLock _pagesLock = new 
ReentrantReadWriteLock();
                private final LinkedHashMap<PageKey, SerializedPage> _pages = 
new LinkedHashMap<PageKey, SerializedPage>();
                private final TreeMap<PageKey, Integer> _pageKeys = new 
TreeMap<PageKey, Integer>();
                // if we have an overflow, we probably had a 100.000 years 
uptime,
                // hooray! :)
                private Integer _id = Integer.MIN_VALUE;

                private final int MAX_SIZE;

                public PageStore(final int maxSize) {
                        MAX_SIZE = maxSize;
                }

                public void storePages(final List<SerializedPage> pagesToAdd) {
                        _pagesLock.writeLock().lock();
                        try {
                                // reduce size of page store to within set size 
if required
                                if (MAX_SIZE != NO_MAX_PAGES_PER_MAP) {
                                        int numToRemove = _pages.size() + 
pagesToAdd.size()
                                                        - MAX_SIZE;
                                        if (numToRemove > 0) {
                                                final Iterator<Entry<PageKey, 
SerializedPage>> iter = _pages
                                                                
.entrySet().iterator();
                                                while (iter.hasNext() && 
numToRemove > 0) {
                                                        final Entry<PageKey, 
SerializedPage> entry = iter
                                                                        .next();
                                                        iter.remove();
                                                        
_pageKeys.remove(entry.getKey());
                                                        numToRemove--;
                                                }
                                        }
                                }
                                for (final SerializedPage sPage : pagesToAdd) {
                                        final PageKey pageKey = new 
PageKey(sPage.getPageId(),
                                                        
sPage.getVersionNumber(), sPage
                                                                        
.getAjaxVersionNumber());
                                        // remove to preserve access order
                                        _pages.remove(pageKey);
                                        _pages.put(pageKey, sPage);
                                        _pageKeys.put(pageKey, _id++);
                                }

                        } finally {
                                _pagesLock.writeLock().unlock();
                        }

                }

                public boolean containsPage(final int pageId, final int 
pageVersion) {
                        _pagesLock.readLock().lock();
                        try {
                                // make PageKeys for below and above this 
version
                                // this id and version but -1 for ajax
                                final PageKey below = new PageKey(pageId, 
pageVersion, -1);
                                // this id and version +1, -1 for ajax
                                final PageKey above = new PageKey(pageId, 
pageVersion + 1, -1);

                                final SortedMap<PageKey, Integer> 
thisPageAndVersion = _pageKeys
                                                .subMap(below, above);

                                return !thisPageAndVersion.isEmpty();

                        } finally {
                                _pagesLock.readLock().unlock();
                        }
                }

                public SerializedPage getPage(final int id, final int 
versionNumber,
                                final int ajaxVersionNumber) {
                        _pagesLock.readLock().lock();
                        try {
                                SerializedPage sPage = null;
                                // just find the exact page version
                                if (versionNumber != -1 && ajaxVersionNumber != 
-1) {
                                        sPage = _pages.get(new PageKey(id, 
versionNumber,
                                                        ajaxVersionNumber));
                                }
                                // we need to find last recently stored page 
window - that is
                                // page
                                // at the end of the list
                                else if (versionNumber == -1) {
                                        final PageKey fromKey = new PageKey(id, 
-1, -1);
                                        final PageKey toKey = new PageKey(id + 
1, -1, -1);

                                        final Iterator<Entry<PageKey, Integer>> 
iter = _pageKeys
                                                        .subMap(fromKey, 
toKey).entrySet().iterator();
                                        int max = -1;
                                        PageKey maxPageKey = null;
                                        while (iter.hasNext()) {
                                                final Entry<PageKey, Integer> 
entry = iter.next();
                                                if (entry.getValue() > max) {
                                                        max = entry.getValue();
                                                        maxPageKey = 
entry.getKey();
                                                }
                                        }
                                        if (maxPageKey != null) {
                                                sPage = _pages.get(maxPageKey);
                                        }
                                }
                                // we need to find index with highest ajax 
version
                                else if (ajaxVersionNumber == -1) {
                                        // make a page key which will be 
straight after wanted
                                        // PageKey, ie, version number is one 
after this one, ajax
                                        // version
                                        // number is -1, page id is the same
                                        final PageKey toElement = new 
PageKey(id,
                                                        versionNumber + 1, -1);
                                        final SortedMap<PageKey, Integer> 
posiblePageKeys = _pageKeys
                                                        .headMap(toElement);
                                        if (posiblePageKeys.size() > 0) {
                                                sPage = 
_pages.get(posiblePageKeys.lastKey());
                                        }

                                }
                                return sPage;
                        } finally {
                                _pagesLock.readLock().unlock();
                        }
                }

                public void removePage(final int id) {
                        _pagesLock.writeLock().lock();
                        try {
                                final Iterator<Entry<PageKey, SerializedPage>> 
iter = _pages
                                                .entrySet().iterator();
                                while (iter.hasNext()) {
                                        final PageKey pKey = 
iter.next().getKey();
                                        if (id == pKey.getId()) {
                                                iter.remove();
                                                _pageKeys.remove(pKey);
                                        }
                                }
                        } finally {
                                _pagesLock.writeLock().unlock();
                        }
                }

                @Override
                public String toString() {
                        final StringBuilder sb = new StringBuilder();
                        final Iterator<Entry<PageKey, SerializedPage>> iter = 
_pages
                                        .entrySet().iterator();
                        while (iter.hasNext()) {
                                final Entry<PageKey, SerializedPage> entry = 
iter.next();
                                
sb.append("\t").append(entry.getKey().toString()).append("\n");
                        }
                        if (logger.isTraceEnabled()) {
                                sb.append("\tPageKeys TreeSet: 
").append(_pageKeys.toString());
                        }
                        return sb.toString();
                }

        }

        protected static class PageKey implements IClusterable, 
Comparable<PageKey> {

                private static final long serialVersionUID = 1L;

                private final int _id;
                private final int _versionNumber;
                private final int _ajaxVersionNumber;

                public PageKey(final int id, final int versionNumber,
                                final int ajaxVersionNumber) {
                        _id = id;
                        _versionNumber = versionNumber;
                        _ajaxVersionNumber = ajaxVersionNumber;
                }

                public int getId() {
                        return _id;
                }

                public int getVersionNumber() {
                        return _versionNumber;
                }

                public int getAjaxVersionNumber() {
                        return _ajaxVersionNumber;
                }

                @Override
                public int hashCode() {
                        final int prime = 31;
                        int result = prime + _ajaxVersionNumber;
                        result = prime * result + _id;
                        result = prime * result + _versionNumber;
                        return result;
                }

                @Override
                public boolean equals(final Object obj) {
                        if (this == obj) {
                                return true;
                        }
                        if (obj == null) {
                                return false;
                        }
                        if (obj instanceof PageKey == false) {
                                return false;
                        }
                        final PageKey other = (PageKey) obj;
                        if (_ajaxVersionNumber != other._ajaxVersionNumber) {
                                return false;
                        }
                        if (_id != other._id) {
                                return false;
                        }
                        if (_versionNumber != other._versionNumber) {
                                return false;
                        }
                        return true;
                }

                @Override
                public String toString() {
                        return "PageID: " + _id + " \tVersion: " + 
_versionNumber
                                        + " \tAjax: " + _ajaxVersionNumber;
                }

                public int compareTo(final PageKey o) {
                        if (_id != o.getId()) {
                                return _id - o.getId();
                        } else if (_versionNumber != o.getVersionNumber()) {
                                return _versionNumber - o.getVersionNumber();
                        } else if (_ajaxVersionNumber != 
o.getAjaxVersionNumber()) {
                                return _ajaxVersionNumber - 
o.getAjaxVersionNumber();
                        }
                        return 0;
                }

        }
}

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org

Reply via email to