Ashish,

I've separated unit tests into two categories. One is a unit test set that
don't need mini yarn cluster, and you can do these unit tests though 'mvn
test'. Another is a unit test set running on mini yarn cluster, and you can
do them though 'mvn verify'.

If you execute 'mvn install', mvn internally includes 'verify phase'. So,
you cannot pass 'mvn install' even though you passed 'mvn test'.

I'm suspecting some bugs caused by TaskRunner. If I fix this bug, I'll let
you know the resolution.

Thanks,
Hyunsik



On Thu, Mar 28, 2013 at 9:16 PM, Ashish <[email protected]> wrote:

> Thanks ! Which Integration test is this?
>
>
> On Thu, Mar 28, 2013 at 5:44 PM, Hyunsik Choi <[email protected]
> >wrote:
>
> > Thank you for your reporting. Actually, I haven't tested the integration
> > test on Mac. I have just tried to do 'mvn verify' on Mac. The same
> problem
> > is happen. It is a bug. I've created the jira issue about this problem.
> >
> > https://issues.apache.org/jira/browse/TAJO-15
> >
> > Thanks you,
> > Hyunsik
> >
> >
> > On Thu, Mar 28, 2013 at 8:39 PM, Ashish <[email protected]> wrote:
> >
> > > I am using Mac 10.8.3, and its happening on both JDK 6 and 7
> > >
> > > java version "1.6.0_43"
> > > Java(TM) SE Runtime Environment (build 1.6.0_43-b01-447-11M4203)
> > > Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-447, mixed mode)
> > >
> > > java version "1.7.0_10"
> > > Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
> > > Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
> > >
> > > Adding a little more logs. Would be great if you tell me the starting
> > point
> > > for this. I shall try to hunt down more details
> > >
> > > 2013-03-28 16:42:39,039 INFO  capacity.CapacityScheduler
> > > (CapacityScheduler.java:completedContainer(776)) - Application
> > > appattempt_1364469093530_0002_000001 released container
> > > container_1364469093530_0002_01_000007 on node: host: a.b.c.d:60941
> > > #containers=0 available=4096 used=0 with event: FINISHED
> > > 2013-03-28 16:42:39,235 INFO  rmcontainer.RMContainerImpl
> > > (RMContainerImpl.java:handle(220)) -
> > container_1364469093530_0002_01_000008
> > > Container Transitioned from ALLOCATED to ACQUIRED
> > > 2013-03-28 16:42:39,236 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 1
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(175)) -
> > > ================================================================
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(177)) - > Container Id:
> > > container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(178)) - > Node Id:
> > > a.b.c.d:60945
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(179)) - > Resource (Mem):
> > 3072
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(180)) - > State : NEW
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(181)) - > Priority: 92
> > > 2013-03-28 16:42:39,237 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(183)) -
> > > ================================================================
> > > 2013-03-28 16:42:39,238 INFO  master.SubQuery
> > > (SubQuery.java:transition(713)) - SubQuery
> > > (sq_1364469093530_0002_000001_27) has 1 containers!
> > > 2013-03-28 16:42:39,238 INFO  master.TaskRunnerLauncherImpl
> > > (TaskRunnerLauncherImpl.java:launch(393)) - Launching Container with
> Id:
> > > container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:39,239 INFO  master.TaskRunnerLauncherImpl
> > > (TaskRunnerLauncherImpl.java:createContainerLaunchContext(301)) -
> > Completed
> > > setting up taskrunner command ${JAVA_HOME}/bin/java -Xmx2000m
> > > tajo.worker.TaskRunner a.b.c.d 58243 sq_1364469093530_0002_000001_27
> > > a.b.c.d:60945 container_1364469093530_0002_01_000008 1><LOG_DIR>/stdout
> > > 2><LOG_DIR>/stderr
> > > 2013-03-28 16:42:39,244 INFO  containermanager.ContainerManagerImpl
> > > (ContainerManagerImpl.java:startContainer(402)) - Start request for
> > > container_1364469093530_0002_01_000008 by user xxxxxxx
> > > 2013-03-28 16:42:39,245 INFO  nodemanager.NMAuditLogger
> > > (NMAuditLogger.java:logSuccess(89)) - USER=xxxxxxx IP=a.b.c.d
> > > OPERATION=Start
> > > Container Request TARGET=ContainerManageImpl RESULT=SUCCESS
> > > APPID=application_1364469093530_0002
> > > CONTAINERID=container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:39,245 INFO  application.Application
> > > (ApplicationImpl.java:transition(255)) - Adding
> > > container_1364469093530_0002_01_000008 to application
> > > application_1364469093530_0002
> > > 2013-03-28 16:42:39,246 INFO  container.Container
> > > (ContainerImpl.java:handle(835)) - Container
> > > container_1364469093530_0002_01_000008 transitioned from NEW to
> > LOCALIZING
> > > 2013-03-28 16:42:39,246 INFO  master.TaskRunnerLauncherImpl
> > > (TaskRunnerLauncherImpl.java:launch(424)) - PullServer port returned by
> > > ContainerManager for container_1364469093530_0002_01_000008 : 60947
> > > 2013-03-28 16:42:39,246 INFO  containermanager.AuxServices
> > > (AuxServices.java:handle(160)) - Got event APPLICATION_INIT for appId
> > > application_1364469093530_0002
> > > 2013-03-28 16:42:39,246 INFO  containermanager.AuxServices
> > > (AuxServices.java:handle(164)) - Got APPLICATION_INIT for service
> > > tajo.pullserver
> > > 2013-03-28 16:42:39,246 INFO  master.Query (Query.java:handle(514)) -
> > > Processing q_1364469093530_0002_000001 of type INIT_COMPLETED
> > > 2013-03-28 16:42:39,246 INFO  container.Container
> > > (ContainerImpl.java:handle(835)) - Container
> > > container_1364469093530_0002_01_000008 transitioned from LOCALIZING to
> > > LOCALIZED
> > > 2013-03-28 16:42:39,247 INFO  util.RackResolver
> > > (RackResolver.java:coreResolve(100)) - Resolved L-IDC77TDV7M-M.local to
> > > /default-rack
> > > 2013-03-28 16:42:39,339 INFO  container.Container
> > > (ContainerImpl.java:handle(835)) - Container
> > > container_1364469093530_0002_01_000008 transitioned from LOCALIZED to
> > > RUNNING
> > > 2013-03-28 16:42:39,340 INFO  monitor.ContainersMonitorImpl
> > > (ContainersMonitorImpl.java:isEnabled(168)) - ResourceCalculatorPlugin
> is
> > > unavailable on this system.
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> > > is disabled.
> > > 2013-03-28 16:42:39,535 INFO  nodemanager.DefaultContainerExecutor
> > > (DefaultContainerExecutor.java:launchContainer(175)) - launchContainer:
> > > [bash,
> > >
> > >
> >
> /Users/xxxxxxx/opensource/tajo/incubator-tajo/tajo-core/tajo-core-backend/target/tajo.TajoTestingCluster/tajo.TajoTestingCluster-localDir-nm-1_0/usercache/xxxxxxx/appcache/application_1364469093530_0002/container_1364469093530_0002_01_000008/default_container_executor.sh]
> > > 2013-03-28 16:42:39,903 INFO  nodemanager.NodeStatusUpdaterImpl
> > > (NodeStatusUpdaterImpl.java:getNodeStatus(265)) - Sending out status
> for
> > > container: container_id {, app_attempt_id {, application_id {, id: 2,
> > > cluster_timestamp: 1364469093530, }, attemptId: 1, }, id: 8, }, state:
> > > C_RUNNING, diagnostics: "", exit_status: -1000,
> > > 2013-03-28 16:42:39,904 INFO  rmcontainer.RMContainerImpl
> > > (RMContainerImpl.java:handle(220)) -
> > container_1364469093530_0002_01_000008
> > > Container Transitioned from ACQUIRED to RUNNING
> > > 2013-03-28 16:42:40,020 WARN  nodemanager.DefaultContainerExecutor
> > > (DefaultContainerExecutor.java:launchContainer(193)) - Exit code from
> > task
> > > is : 1
> > > 2013-03-28 16:42:40,021 INFO  nodemanager.ContainerExecutor
> > > (ContainerExecutor.java:logOutput(167)) -
> > > 2013-03-28 16:42:40,021 WARN  launcher.ContainerLaunch
> > > (ContainerLaunch.java:call(274)) - Container exited with a non-zero
> exit
> > > code 1
> > > 2013-03-28 16:42:40,021 INFO  container.Container
> > > (ContainerImpl.java:handle(835)) - Container
> > > container_1364469093530_0002_01_000008 transitioned from RUNNING to
> > > EXITED_WITH_FAILURE
> > > 2013-03-28 16:42:40,021 INFO  launcher.ContainerLaunch
> > > (ContainerLaunch.java:cleanupContainer(300)) - Cleaning up container
> > > container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:40,040 INFO  nodemanager.DefaultContainerExecutor
> > > (DefaultContainerExecutor.java:deleteAsUser(273)) - Deleting absolute
> > path
> > > :
> > >
> > >
> >
> /Users/xxxxxxx/opensource/tajo/incubator-tajo/tajo-core/tajo-core-backend/target/tajo.TajoTestingCluster/tajo.TajoTestingCluster-localDir-nm-1_0/usercache/xxxxxxx/appcache/application_1364469093530_0002/container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:40,040 WARN  nodemanager.NMAuditLogger
> > > (NMAuditLogger.java:logFailure(150)) - USER=xxxxxxx OPERATION=Container
> > > Finished - Failed TARGET=ContainerImpl RESULT=FAILURE
> > DESCRIPTION=Container
> > > failed with state: EXITED_WITH_FAILURE
> > APPID=application_1364469093530_0002
> > > CONTAINERID=container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:40,041 INFO  container.Container
> > > (ContainerImpl.java:handle(835)) - Container
> > > container_1364469093530_0002_01_000008 transitioned from
> > > EXITED_WITH_FAILURE to DONE
> > > 2013-03-28 16:42:40,041 INFO  application.Application
> > > (ApplicationImpl.java:transition(298)) - Removing
> > > container_1364469093530_0002_01_000008 from application
> > > application_1364469093530_0002
> > > 2013-03-28 16:42:40,041 INFO  monitor.ContainersMonitorImpl
> > > (ContainersMonitorImpl.java:isEnabled(168)) - ResourceCalculatorPlugin
> is
> > > unavailable on this system.
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> > > is disabled.
> > > 2013-03-28 16:42:40,241 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:40,241 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:40,905 INFO  nodemanager.NodeStatusUpdaterImpl
> > > (NodeStatusUpdaterImpl.java:getNodeStatus(265)) - Sending out status
> for
> > > container: container_id {, app_attempt_id {, application_id {, id: 2,
> > > cluster_timestamp: 1364469093530, }, attemptId: 1, }, id: 8, }, state:
> > > C_COMPLETE, diagnostics: "\n", exit_status: 1,
> > > 2013-03-28 16:42:40,905 INFO  nodemanager.NodeStatusUpdaterImpl
> > > (NodeStatusUpdaterImpl.java:getNodeStatus(271)) - Removed completed
> > > container container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:40,906 INFO  rmcontainer.RMContainerImpl
> > > (RMContainerImpl.java:handle(220)) -
> > container_1364469093530_0002_01_000008
> > > Container Transitioned from RUNNING to COMPLETED
> > > 2013-03-28 16:42:40,906 INFO  fica.FiCaSchedulerApp
> > > (FiCaSchedulerApp.java:containerCompleted(219)) - Completed container:
> > > container_1364469093530_0002_01_000008 in state: COMPLETED
> event:FINISHED
> > > 2013-03-28 16:42:40,906 INFO  resourcemanager.RMAuditLogger
> > > (RMAuditLogger.java:logSuccess(98)) - USER=xxxxxxx OPERATION=AM
> Released
> > > Container TARGET=SchedulerApp RESULT=SUCCESS
> > > APPID=application_1364469093530_0002
> > > CONTAINERID=container_1364469093530_0002_01_000008
> > > 2013-03-28 16:42:40,906 INFO  fica.FiCaSchedulerNode
> > > (FiCaSchedulerNode.java:releaseContainer(150)) - Released container
> > > container_1364469093530_0002_01_000008 of capacity <memory:3072,
> > vCores:1>
> > > on host a.b.c.d:60945, which currently has 0 containers, <memory:0,
> > > vCores:0> used and <memory:4096, vCores:16> available, release
> > > resources=true
> > > 2013-03-28 16:42:40,906 INFO  capacity.LeafQueue
> > > (LeafQueue.java:releaseResource(1441)) - default used=<memory:0,
> > vCores:0>
> > > numContainers=0 user=xxxxxxx user-resources=<memory:0, vCores:0>
> > > 2013-03-28 16:42:40,907 INFO  capacity.LeafQueue
> > > (LeafQueue.java:completedContainer(1385)) - completedContainer
> > > container=Container: [ContainerId:
> > container_1364469093530_0002_01_000008,
> > > NodeId: a.b.c.d:60945, NodeHttpAddress: a.b.c.d:60948, Resource:
> > > <memory:3072, vCores:1>, Priority: 92, State: NEW, Token: null, Status:
> > > container_id {, app_attempt_id {, application_id {, id: 2,
> > > cluster_timestamp: 1364469093530, }, attemptId: 1, }, id: 8, }, state:
> > > C_COMPLETE, diagnostics: "\n", exit_status: 1, ] resource=<memory:3072,
> > > vCores:1> queue=default: capacity=1.0, absoluteCapacity=1.0,
> > > usedResources=<memory:0, vCores:0>usedCapacity=0.0,
> > > absoluteUsedCapacity=0.0, numApps=1, numContainers=0 usedCapacity=0.0
> > > absoluteUsedCapacity=0.0 used=<memory:0, vCores:0>
> cluster=<memory:12288,
> > > vCores:48>
> > > 2013-03-28 16:42:40,907 INFO  capacity.ParentQueue
> > > (ParentQueue.java:completedContainer(696)) - completedContainer
> > queue=root
> > > usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0>
> > > cluster=<memory:12288, vCores:48>
> > > 2013-03-28 16:42:40,907 INFO  capacity.CapacityScheduler
> > > (CapacityScheduler.java:completedContainer(776)) - Application
> > > appattempt_1364469093530_0002_000001 released container
> > > container_1364469093530_0002_01_000008 on node: host: a.b.c.d:60945
> > > #containers=0 available=4096 used=0 with event: FINISHED
> > > 2013-03-28 16:42:41,242 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:41,242 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:42,245 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:42,246 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:43,248 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:43,249 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:44,251 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:44,252 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:45,255 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:45,256 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:46,259 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:46,260 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:47,263 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > > 2013-03-28 16:42:47,264 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of Allocated
> > > Containers: 0
> > > 2013-03-28 16:42:48,267 INFO  rm.RMContainerAllocator
> > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> Resource:
> > > <memory:6144, vCores:-1>
> > >
> > >
> > >
> > >
> > > On Thu, Mar 28, 2013 at 4:56 PM, Hyunsik Choi <[email protected]
> > > >wrote:
> > >
> > > > Hi Ashish,
> > > >
> > > > The hanging unit tests are integration tests. You can skip the
> > > integration
> > > > test with this command 'mvn install -DskipTests'.
> > > >
> > > > However, I cannot figure out the cause of the hanging unit tests from
> > > your
> > > > log message.  Could you let me know more information?
> > > >
> > > > 1) What are your linux and procps versions? Recently, I had
> experienced
> > > the
> > > > same situation due to this bug (
> > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688731).
> > > >
> > > > 2) What is the first exception of unit tests?
> > > >
> > > > Thanks,
> > > > Hyunsik
> > > >
> > > >
> > > > On Thu, Mar 28, 2013 at 8:07 PM, Ashish <[email protected]>
> > wrote:
> > > >
> > > > > Build is working fine at my end on both JDK 6 and 7.
> > > > >
> > > > > mvn clean;mvn compile;mvn test run fine.
> > > > >
> > > > > Facing a strange issue while running "mvn clean install", the build
> > > never
> > > > > completes and get stuck in this loop
> > > > >
> > > > > 2013-03-28 16:35:48,581 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of
> Allocated
> > > > > Containers: 0
> > > > > 2013-03-28 16:35:49,584 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> > > Resource:
> > > > > <memory:6144, vCores:-1>
> > > > > 2013-03-28 16:35:49,584 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of
> Allocated
> > > > > Containers: 0
> > > > > 2013-03-28 16:35:50,588 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> > > Resource:
> > > > > <memory:6144, vCores:-1>
> > > > > 2013-03-28 16:35:50,588 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of
> Allocated
> > > > > Containers: 0
> > > > > 2013-03-28 16:35:51,590 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(172)) - Available
> > > Resource:
> > > > > <memory:6144, vCores:-1>
> > > > > 2013-03-28 16:35:51,590 INFO  rm.RMContainerAllocator
> > > > > (RMContainerAllocator.java:makeRemoteRequest(173)) - Num of
> Allocated
> > > > > Containers: 0
> > > > >
> > > > > Had to kill the build after observing in this state for more than
> 10
> > > min.
> > > > > Any suggestions on what might be holding it?
> > > > >
> > > > > thanks
> > > > > ashish
> > > > >
> > > > >
> > > > > On Thu, Mar 28, 2013 at 7:52 AM, Hyunsik Choi <
> > [email protected]
> > > > > >wrote:
> > > > >
> > > > > > Thank you for letting me know. JDK 1.6 is right. I'll fix the jdk
> > > > version
> > > > > > of the getting started.
> > > > > >
> > > > > > - hyunsik
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 28, 2013 at 11:15 AM, Tanujit Ghosh <
> > > > [email protected]
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > The Tajo wiki is saying to set JAVA_HOME as
> > > > > > >
> > > > > > > export JAVA_HOME=/usr/lib/jvm/openjdk-1.7.x
> > > > > > >
> > > > > > >
> > > > > > > But as per the Requirement i guess this should be jdk1.6.
> > > > > > >
> > > > > > >
> > > > > > > Also compiling with jdk1.7 given a issue with ResultSetImpl
> > class.
> > > > > > > My environment is gone wrong at the moment, but will send out
> the
> > > > error
> > > > > > > asap.
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Tanujit
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > thanks
> > > > > ashish
> > > > >
> > > > > Blog: http://www.ashishpaliwal.com/blog
> > > > > My Photo Galleries: http://www.pbase.com/ashishpaliwal
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > thanks
> > > ashish
> > >
> > > Blog: http://www.ashishpaliwal.com/blog
> > > My Photo Galleries: http://www.pbase.com/ashishpaliwal
> > >
> >
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Reply via email to