Re: Spark stand-alone mode

[Ilango]

Hi all,

Thanks a lot for your suggestions and knowledge sharing. I like to let you
know that, I completed setting up the stand alone cluster and couple of
data science users are able to use it already for last two weeks. And the
performance is really good. Almost 10X performance improvement compare to
HPC local mode. They tested with some complex data science scripts using
spark and other data science projects. The cluster is really stable and
very performant.

I enabled dynamic allocation and cap the memory and cpu accordingly at
spark-defaults. Conf and at our spark framework code. So its been pretty
impressive for the last few weeks.

Thanks you so much!

Thanks,
Elango


On Tue, 19 Sep 2023 at 6:40 PM, Patrick Tucci 
wrote:

> Multiple applications can run at once, but you need to either configure
> Spark or your applications to allow that. In stand-alone mode, each
> application attempts to take all resources available by default. This
> section of the documentation has more details:
>
>
> https://spark.apache.org/docs/latest/spark-standalone.html#resource-scheduling
>
> Explicitly setting the resources per application limits the resources to
> the configured values for the lifetime of the application. You can use
> dynamic allocation to allow Spark to scale the resources up and down per
> application based on load, but the configuration is relatively more complex:
>
>
> https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
>
> On Mon, Sep 18, 2023 at 3:53 PM Ilango  wrote:
>
>>
>> Thanks all for your suggestions. Noted with thanks.
>> Just wanted share few more details about the environment
>> 1. We use NFS for data storage and data is in parquet format
>> 2. All HPC nodes are connected and already work as a cluster for Studio
>> workbench. I can setup password less SSH if it not exist already.
>> 3. We will stick with NFS for now and stand alone then may be will
>> explore HDFS and YARN.
>>
>> Can you please confirm whether multiple users can run spark jobs at the
>> same time?
>> If so I will start working on it and let you know how it goes
>>
>> Mich, the link to Hadoop is not working. Can you please check and let me
>> know the correct link. Would like to explore Hadoop option as well.
>>
>>
>>
>> Thanks,
>> Elango
>>
>> On Sat, Sep 16, 2023, 4:20 AM Bjørn Jørgensen 
>> wrote:
>>
>>> you need to setup ssh without password, use key instead.  How to
>>> connect without password using SSH (passwordless)
>>> 
>>>
>>> fre. 15. sep. 2023 kl. 20:55 skrev Mich Talebzadeh <
>>> mich.talebza...@gmail.com>:
>>>
 Hi,

 Can these 4 nodes talk to each other through ssh as trusted hosts (on
 top of the network that Sean already mentioned)? Otherwise you need to set
 it up. You can install a LAN if you have another free port at the back of
 your HPC nodes. They should

 You ought to try to set up a Hadoop cluster pretty easily. Check this
 old article of mine for Hadoop set-up.


 https://www.linkedin.com/pulse/diy-festive-season-how-install-configure-big-data-so-mich/?trackingId=z7n5tx7tQOGK9tcG9VClkw%3D%3D

 Hadoop will provide you with a common storage layer (HDFS) that these
 nodes will be able to share and talk. Yarn is your best bet as the resource
 manager with reasonably powerful hosts you have. However, for now the Stand
 Alone mode will do. Make sure that the Metastore you choose, (by default it
 will use Hive Metastore called Derby :( ) is something respetable like
 Postgres DB that can handle multiple concurrent spark jobs

 HTH


 Mich Talebzadeh,
 Distinguished Technologist, Solutions Architect & Engineer
 London
 United Kingdom


view my Linkedin profile
 


  https://en.everybodywiki.com/Mich_Talebzadeh



 *Disclaimer:* Use it at your own risk. Any and all responsibility for
 any loss, damage or destruction of data or any other property which may
 arise from relying on this email's technical content is explicitly
 disclaimed. The author will in no case be liable for any monetary damages
 arising from such loss, damage or destruction.




 On Fri, 15 Sept 2023 at 07:04, Ilango  wrote:

>
> Hi all,
>
> We have 4 HPC nodes and installed spark individually in all nodes.
>
> Spark is used as local mode(each driver/executor will have 8 cores and
> 65 GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
> scheduler.
>
> As this is local mode, we are facing performance issue(as only one
> executor) when it comes dealing with large datasets.
>
> Can I convert this 4 nodes into spark standalone cluster. We dont have
> hadoop so yarn mode is out of 

Re: Spark stand-alone mode

[Patrick Tucci]

Multiple applications can run at once, but you need to either configure
Spark or your applications to allow that. In stand-alone mode, each
application attempts to take all resources available by default. This
section of the documentation has more details:

https://spark.apache.org/docs/latest/spark-standalone.html#resource-scheduling

Explicitly setting the resources per application limits the resources to
the configured values for the lifetime of the application. You can use
dynamic allocation to allow Spark to scale the resources up and down per
application based on load, but the configuration is relatively more complex:

https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

On Mon, Sep 18, 2023 at 3:53 PM Ilango  wrote:

>
> Thanks all for your suggestions. Noted with thanks.
> Just wanted share few more details about the environment
> 1. We use NFS for data storage and data is in parquet format
> 2. All HPC nodes are connected and already work as a cluster for Studio
> workbench. I can setup password less SSH if it not exist already.
> 3. We will stick with NFS for now and stand alone then may be will explore
> HDFS and YARN.
>
> Can you please confirm whether multiple users can run spark jobs at the
> same time?
> If so I will start working on it and let you know how it goes
>
> Mich, the link to Hadoop is not working. Can you please check and let me
> know the correct link. Would like to explore Hadoop option as well.
>
>
>
> Thanks,
> Elango
>
> On Sat, Sep 16, 2023, 4:20 AM Bjørn Jørgensen 
> wrote:
>
>> you need to setup ssh without password, use key instead.  How to connect
>> without password using SSH (passwordless)
>> 
>>
>> fre. 15. sep. 2023 kl. 20:55 skrev Mich Talebzadeh <
>> mich.talebza...@gmail.com>:
>>
>>> Hi,
>>>
>>> Can these 4 nodes talk to each other through ssh as trusted hosts (on
>>> top of the network that Sean already mentioned)? Otherwise you need to set
>>> it up. You can install a LAN if you have another free port at the back of
>>> your HPC nodes. They should
>>>
>>> You ought to try to set up a Hadoop cluster pretty easily. Check this
>>> old article of mine for Hadoop set-up.
>>>
>>>
>>> https://www.linkedin.com/pulse/diy-festive-season-how-install-configure-big-data-so-mich/?trackingId=z7n5tx7tQOGK9tcG9VClkw%3D%3D
>>>
>>> Hadoop will provide you with a common storage layer (HDFS) that these
>>> nodes will be able to share and talk. Yarn is your best bet as the resource
>>> manager with reasonably powerful hosts you have. However, for now the Stand
>>> Alone mode will do. Make sure that the Metastore you choose, (by default it
>>> will use Hive Metastore called Derby :( ) is something respetable like
>>> Postgres DB that can handle multiple concurrent spark jobs
>>>
>>> HTH
>>>
>>>
>>> Mich Talebzadeh,
>>> Distinguished Technologist, Solutions Architect & Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 15 Sept 2023 at 07:04, Ilango  wrote:
>>>

 Hi all,

 We have 4 HPC nodes and installed spark individually in all nodes.

 Spark is used as local mode(each driver/executor will have 8 cores and
 65 GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
 scheduler.

 As this is local mode, we are facing performance issue(as only one
 executor) when it comes dealing with large datasets.

 Can I convert this 4 nodes into spark standalone cluster. We dont have
 hadoop so yarn mode is out of scope.

 Shall I follow the official documentation for setting up standalone
 cluster. Will it work? Do I need to aware anything else?
 Can you please share your thoughts?

 Thanks,
 Elango

>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>

Re: Spark stand-alone mode

[Ilango]

Thanks all for your suggestions. Noted with thanks.
Just wanted share few more details about the environment
1. We use NFS for data storage and data is in parquet format
2. All HPC nodes are connected and already work as a cluster for Studio
workbench. I can setup password less SSH if it not exist already.
3. We will stick with NFS for now and stand alone then may be will explore
HDFS and YARN.

Can you please confirm whether multiple users can run spark jobs at the
same time?
If so I will start working on it and let you know how it goes

Mich, the link to Hadoop is not working. Can you please check and let me
know the correct link. Would like to explore Hadoop option as well.



Thanks,
Elango

On Sat, Sep 16, 2023, 4:20 AM Bjørn Jørgensen 
wrote:

> you need to setup ssh without password, use key instead.  How to connect
> without password using SSH (passwordless)
> 
>
> fre. 15. sep. 2023 kl. 20:55 skrev Mich Talebzadeh <
> mich.talebza...@gmail.com>:
>
>> Hi,
>>
>> Can these 4 nodes talk to each other through ssh as trusted hosts (on top
>> of the network that Sean already mentioned)? Otherwise you need to set it
>> up. You can install a LAN if you have another free port at the back of your
>> HPC nodes. They should
>>
>> You ought to try to set up a Hadoop cluster pretty easily. Check this old
>> article of mine for Hadoop set-up.
>>
>>
>> https://www.linkedin.com/pulse/diy-festive-season-how-install-configure-big-data-so-mich/?trackingId=z7n5tx7tQOGK9tcG9VClkw%3D%3D
>>
>> Hadoop will provide you with a common storage layer (HDFS) that these
>> nodes will be able to share and talk. Yarn is your best bet as the resource
>> manager with reasonably powerful hosts you have. However, for now the Stand
>> Alone mode will do. Make sure that the Metastore you choose, (by default it
>> will use Hive Metastore called Derby :( ) is something respetable like
>> Postgres DB that can handle multiple concurrent spark jobs
>>
>> HTH
>>
>>
>> Mich Talebzadeh,
>> Distinguished Technologist, Solutions Architect & Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 15 Sept 2023 at 07:04, Ilango  wrote:
>>
>>>
>>> Hi all,
>>>
>>> We have 4 HPC nodes and installed spark individually in all nodes.
>>>
>>> Spark is used as local mode(each driver/executor will have 8 cores and
>>> 65 GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
>>> scheduler.
>>>
>>> As this is local mode, we are facing performance issue(as only one
>>> executor) when it comes dealing with large datasets.
>>>
>>> Can I convert this 4 nodes into spark standalone cluster. We dont have
>>> hadoop so yarn mode is out of scope.
>>>
>>> Shall I follow the official documentation for setting up standalone
>>> cluster. Will it work? Do I need to aware anything else?
>>> Can you please share your thoughts?
>>>
>>> Thanks,
>>> Elango
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>

Re: Spark stand-alone mode

[Bjørn Jørgensen]

you need to setup ssh without password, use key instead.  How to connect
without password using SSH (passwordless)


fre. 15. sep. 2023 kl. 20:55 skrev Mich Talebzadeh <
mich.talebza...@gmail.com>:

> Hi,
>
> Can these 4 nodes talk to each other through ssh as trusted hosts (on top
> of the network that Sean already mentioned)? Otherwise you need to set it
> up. You can install a LAN if you have another free port at the back of your
> HPC nodes. They should
>
> You ought to try to set up a Hadoop cluster pretty easily. Check this old
> article of mine for Hadoop set-up.
>
>
> https://www.linkedin.com/pulse/diy-festive-season-how-install-configure-big-data-so-mich/?trackingId=z7n5tx7tQOGK9tcG9VClkw%3D%3D
>
> Hadoop will provide you with a common storage layer (HDFS) that these
> nodes will be able to share and talk. Yarn is your best bet as the resource
> manager with reasonably powerful hosts you have. However, for now the Stand
> Alone mode will do. Make sure that the Metastore you choose, (by default it
> will use Hive Metastore called Derby :( ) is something respetable like
> Postgres DB that can handle multiple concurrent spark jobs
>
> HTH
>
>
> Mich Talebzadeh,
> Distinguished Technologist, Solutions Architect & Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 15 Sept 2023 at 07:04, Ilango  wrote:
>
>>
>> Hi all,
>>
>> We have 4 HPC nodes and installed spark individually in all nodes.
>>
>> Spark is used as local mode(each driver/executor will have 8 cores and 65
>> GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
>> scheduler.
>>
>> As this is local mode, we are facing performance issue(as only one
>> executor) when it comes dealing with large datasets.
>>
>> Can I convert this 4 nodes into spark standalone cluster. We dont have
>> hadoop so yarn mode is out of scope.
>>
>> Shall I follow the official documentation for setting up standalone
>> cluster. Will it work? Do I need to aware anything else?
>> Can you please share your thoughts?
>>
>> Thanks,
>> Elango
>>
>

-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297

Re: Spark stand-alone mode

[Mich Talebzadeh]

Hi,

Can these 4 nodes talk to each other through ssh as trusted hosts (on top
of the network that Sean already mentioned)? Otherwise you need to set it
up. You can install a LAN if you have another free port at the back of your
HPC nodes. They should

You ought to try to set up a Hadoop cluster pretty easily. Check this old
article of mine for Hadoop set-up.

https://www.linkedin.com/pulse/diy-festive-season-how-install-configure-big-data-so-mich/?trackingId=z7n5tx7tQOGK9tcG9VClkw%3D%3D

Hadoop will provide you with a common storage layer (HDFS) that these nodes
will be able to share and talk. Yarn is your best bet as the resource
manager with reasonably powerful hosts you have. However, for now the Stand
Alone mode will do. Make sure that the Metastore you choose, (by default it
will use Hive Metastore called Derby :( ) is something respetable like
Postgres DB that can handle multiple concurrent spark jobs

HTH


Mich Talebzadeh,
Distinguished Technologist, Solutions Architect & Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 15 Sept 2023 at 07:04, Ilango  wrote:

>
> Hi all,
>
> We have 4 HPC nodes and installed spark individually in all nodes.
>
> Spark is used as local mode(each driver/executor will have 8 cores and 65
> GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
> scheduler.
>
> As this is local mode, we are facing performance issue(as only one
> executor) when it comes dealing with large datasets.
>
> Can I convert this 4 nodes into spark standalone cluster. We dont have
> hadoop so yarn mode is out of scope.
>
> Shall I follow the official documentation for setting up standalone
> cluster. Will it work? Do I need to aware anything else?
> Can you please share your thoughts?
>
> Thanks,
> Elango
>

Re: Spark stand-alone mode

[Sean Owen]

Yes, should work fine, just set up according to the docs. There needs to be
network connectivity between whatever the driver node is and these 4 nodes.

On Thu, Sep 14, 2023 at 11:57 PM Ilango  wrote:

>
> Hi all,
>
> We have 4 HPC nodes and installed spark individually in all nodes.
>
> Spark is used as local mode(each driver/executor will have 8 cores and 65
> GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
> scheduler.
>
> As this is local mode, we are facing performance issue(as only one
> executor) when it comes dealing with large datasets.
>
> Can I convert this 4 nodes into spark standalone cluster. We dont have
> hadoop so yarn mode is out of scope.
>
> Shall I follow the official documentation for setting up standalone
> cluster. Will it work? Do I need to aware anything else?
> Can you please share your thoughts?
>
> Thanks,
> Elango
>

Re: Spark stand-alone mode

[Patrick Tucci]

I use Spark in standalone mode. It works well, and the instructions on the
site are accurate for the most part. The only thing that didn't work for me
was the start_all.sh script. Instead, I use a simple script that starts the
master node, then uses SSH to connect to the worker machines and start the
worker nodes.

All the nodes will need access to the same data, so you will need some sort
of shared file system. You could use an NFS share mounted to the same point
on each machine, S3, or HDFS.

Standalone also acquires all resources when an application is submitted, so
by default only one application may be run at a time. You can limit the
resources allocated to each application to allow multiple concurrent
applications, or you could configure dynamic allocation to scale the
resources up and down per application as needed.

On Fri, Sep 15, 2023 at 5:56 AM Ilango  wrote:

>
> Hi all,
>
> We have 4 HPC nodes and installed spark individually in all nodes.
>
> Spark is used as local mode(each driver/executor will have 8 cores and 65
> GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
> scheduler.
>
> As this is local mode, we are facing performance issue(as only one
> executor) when it comes dealing with large datasets.
>
> Can I convert this 4 nodes into spark standalone cluster. We dont have
> hadoop so yarn mode is out of scope.
>
> Shall I follow the official documentation for setting up standalone
> cluster. Will it work? Do I need to aware anything else?
> Can you please share your thoughts?
>
> Thanks,
> Elango
>

Spark stand-alone mode

[Ilango]

Hi all,

We have 4 HPC nodes and installed spark individually in all nodes.

Spark is used as local mode(each driver/executor will have 8 cores and 65
GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as
scheduler.

As this is local mode, we are facing performance issue(as only one
executor) when it comes dealing with large datasets.

Can I convert this 4 nodes into spark standalone cluster. We dont have
hadoop so yarn mode is out of scope.

Shall I follow the official documentation for setting up standalone
cluster. Will it work? Do I need to aware anything else?
Can you please share your thoughts?

Thanks,
Elango

Spark Stand-alone mode job not starting (akka Connection refused)

[T.J. Alumbaugh]

I've been trying for several days now to get a Spark application running in
stand-alone mode, as described here:

http://spark.apache.org/docs/latest/spark-standalone.html

I'm using pyspark, so I've been following the example here:

http://spark.apache.org/docs/0.9.1/quick-start.html#a-standalone-app-in-python

I've run Spark successfully in local mode using bin/pyspark, or even just
setting the SPARK_HOME environment variable, proper PYTHONPATH, and then
starting up python 2.7, importing pyspark, and creating a SparkContext
object. It's running in any kind of cluster mode that seems to be the
problem.

The StandAlone.py program in the example just reads a file and counts
lines. My SparkConf looks like this:

from pyspark import SparkConf, SparkContext
conf = SparkConf()
#conf.setMaster(spark://192.168.0.9:7077)
conf.setMaster(spark://myhostname.domain.com:7077)
conf.setAppName(My application)
conf.set(spark.executor.memory, 1g)

I tried a couple of configurations:

Config 1: (All on one) - master is localhost, slave is localhost
Config 2 (Separate master and slave) - master is localhost, slave is
another host

I've tried a few different machines:
Machine 1: Mac OS 10.9 w/ CDH5 Hadoop distribution, compiled
with SPARK_HADOOP_VERSION=2.3.0-cdh5.0.0 option

Machines 2, 3: Centos 6.4 w/ CDH5 Hadoop distribution, compiled
with SPARK_HADOOP_VERSION=2.3.0-cdh5.0.0 option

Machine 4: Centos 6.4 with Hadoop 1.04 (default Spark compilation)

Here are the results I've had:

Config 1 on Machine 1: Success
Config 1 on Machine 2: Fail
Config 2 on Machines 2,3: Fail
Config 1 on Machines 4: Fail
Config 2 on Machines 1,4: Fail

In the case of failure, the error is always the same.

akka.tcp://sp...@node4.myhostname.domain.com:43717 got disassociated,
removing it.
akka.tcp://sp...@node4.myhostname.domain.com:43717 got disassociated,
removing it.
Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.0.2.55%3A42546-2#-1875068764]
was not delivered. [1] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
AssociationError [akka.tcp://sparkMaster@node4:7077] - [akka.tcp://
sp...@node4.myhostname.domain.com:43717]: Error [Association failed with
[akka.tcp://sp...@node4.myhostname.domain.com:43717]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sp...@node4.myhostname.domain.com:43717]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: node4.myhostname.domain.com/10.0.2.55:43717

It will then repeat this line:
parentName: , name: TaskSet_0, runningTasks: 0

for a while, and then print out this message:
Initial job has not accepted any resources; check your cluster UI to ensure
that workers are registered and have sufficient memory

I have turned the verbosity to DEBUG on all log4j.properties I can find.
There are no firewalls or blocked ports on the internal network.

In all configurations on all machines, when I do sbin/start-master.sh,
sbin/start-slaves.sh, the respective log files always show the correct info
(I have been elected leader! New state: ALIVE or Successfully registered
with master spark://blah-blah:7077). The very nice UIs (on port 8080 for
the masters, port 8081 for the slaves) always show that everything is in
order. The master host shows the workers, the workers acknowledge they have
registered with the master.

When attempting to get 'Config 1' running on any of the machines, I've put
both 'localhost' and the actual fully qualified domain name of the host in
conf/slaves. Results are the same.

In the one case where things are working, I see messages like this in the
log:

Remoting started; listening on addresses :[akka.tcp://
sparkExecutor@192.168.0.9:59049]
Remoting now listens on addresses: [akka.tcp://
sparkExecutor@192.168.0.9:59049]
Connecting to driver: akka.tcp://
spark@192.168.0.9:59032/user/CoarseGrainedScheduler
Connecting to worker akka.tcp://sparkWorker@192.168.0.9:59005/user/Worker
Successfully connected to akka.tcp://
sparkWorker@192.168.0.9:59005/user/Worker
Successfully registered with driver

I've tried many different variables in my spark-env.sh. Currently, in the
one case that works, I set:

STANDALONE_SPARK_MASTER_HOST=`hostname -f`

but that's about it (setting that in the failure cases does not make them
work).
So to me, it seems like the messages from Akka are not getting to the
workers. Any idea why this is?
Thanks for the help!

-T.J.