nfdump install instructions seem inaccurate

[skip cruse]

I noticed when setting up nfdump for spot-ingest that there were some
errors around the version of automake that’s requried. Apparently automake
1.14 is required and after installing that everything worked fine. I did
some digging and it looks like this was already raised via SPOT-178:
https://issues.apache.org/jira/browse/SPOT-178. Perhaps someone could
update the website to reflect this dependency and add the additional setup
steps in there as well?

Cheers,
Skip
-- 
if( bool halfWayThere == true);
printf "WAH! LIVIN ON A PRAYER";

install instructions have a typo

[skip cruse]

I was following the install instructions and noticed that the commands
cannot be copy and pasted as there are missing line breaks in the commands.
Additionally a -j is needed on the tar command to successfully extract a
.bzip2 file and extract wireshark. We should update this page to have
proper line breaking and correct command line flags.

BR,
Skip
-- 
if( bool halfWayThere == true);
printf "WAH! LIVIN ON A PRAYER";

Re: install instructions have a typo

[skip cruse]

PR Opened

On Fri, Feb 14, 2020 at 3:56 PM Tadd Wood  wrote:

> Skip, thanks for pointing that out.  Should be a quick fix which is great.
> Also, this ticket is relevant to that issue:
> https://issues.apache.org/jira/projects/SPOT/issues/SPOT-147
>
> Let us know when you've got a PR and we'll get it reviewed.
>
> Thank you,
> Tadd Wood
>
> On Fri, Feb 14, 2020 at 9:39 AM skip cruse  wrote:
>
> > I was following the install instructions and noticed that the commands
> > cannot be copy and pasted as there are missing line breaks in the
> commands.
> > Additionally a -j is needed on the tar command to successfully extract a
> > .bzip2 file and extract wireshark. We should update this page to have
> > proper line breaking and correct command line flags.
> >
> > BR,
> > Skip
> > --
> > if( bool halfWayThere == true);
> > printf "WAH! LIVIN ON A PRAYER";
> >
>


-- 
if( bool halfWayThere == true);
echo "WAH! LIVIN ON A PRAYER";

update copyright

[skip cruse]

I think we should update the copyright notices on the spot website
http://spot.apache.org/get-started/ to 2020.

-- 
if( bool halfWayThere == true);
echo "WAH! LIVIN ON A PRAYER";

Re: [SPOT-INGEST] Ingest file organization

[Skip Cruse]

We should keep the name /spot-ingest/ for the original ingester, but move the 
new ingester to /spot-ingest-sparkstreaming/ or similar.  Hopefully we can use 
the ticket to track down the files that were created, so we can move them to a 
new home easily.

Get Outlook for iOS


From: Tadd Wood 
Sent: Tuesday, January 14, 2020 5:51 PM
To: dev@spot.incubator.apache.org
Subject: [SPOT-INGEST] Ingest file organization

I noticed that after SPOT-141 was introduced (a new kind of Spot Ingest,
using PySpark Streaming) that it overlaid the new code on top of the old
code on /spot-ingest/. When debugging the code, it makes it hard to
determine which files are relevant to the new or the old ingest process. We
should split them apart. Thoughts?

Thank you,
Tadd Wood

ingest_conf question

[skip cruse]

I was setting up Spot-Ingest, and I noticed that I needed to create a
“ingest_conf.json” file. The documentation on
http://spot.apache.org/doc/#deployment refers to a few things, but doesn’t
go into any details. It does link to the github page at
https://github.com/apache/incubator-spot/tree/master/spot-ingest/README.md,
but that gives a blank config, and doesn’t talk in more detail about how to
determine what values are appropriate.

Perhaps we can find out if people have successfully filled in this config
file, and can share details that don’t talk about their specific
implementation details, but do lead a user to understanding what the values
are intended to be?

-- 
if( bool halfWayThere == true);
echo "WAH! LIVIN ON A PRAYER";

Place where we can download sample test files

[skip cruse]

Does anyone have access to the original Data Samples pointed at here? (
DATA_SAMPLE.md
)
. It looks the links broke sometime back to the s3 bucket. Also would be
great if anyone could share links to other sample datasets for Proxy and
Flow as well. I think something the ingestion code could benefit from
greatly is an option to start-up and ingest a few test files to make sure
that the processors and collectors or working as intended.


cheers,
Skip

wireshark version

[skip cruse]

I noticed when making my changes on PR-159 ( https:/
/github.com/apache/incubator-spot/pull/159) that the Wireshark download
link isn’t actually pointing to the right version of Wireshark that we
claim to support (2.0.1), and the link also appears to be broken. I’m going
to update this link, but I wanted to also check and make sure this is the
correct version of wireshark and if 2.2.3 is actually the supported version.

Cheers,
Skip

Re: [SPOT-266] ml_ops.sh script should use Spark 2?

[skip cruse]

That seems like a fair solution. I noticed there’s also a spark-submit
command that’s required when running this pipeline:worke

r.py . We should make the update here as well since I believe this also
requries Spark 2.1+.

On Fri, Mar 20, 2020 at 4:24 PM Tadd Wood  wrote:

> I notice there’s a dependency on Spark 2.1+ that didn’t get resolved per
> this JIRA (https://issues.apache.org/jira/projects/SPOT/issues/SPOT-266).
> I
> propose adding a check in this script to verify the user’s version of Spark
> and if it's not at least 2.1 the script will exit and indicate that the
> user should upgrade their version of Spark. We also need to replace
> spark-submit with spark-submit2. Any thoughts for/against updating this
> script?
>
> Also would anyone be willing to update the documentation on the Spot
> website to reflect the correct required version of Spark to run Spot?
>
> Thank you,
> Tadd Wood
>


-- 
if( bool halfWayThere == true);
echo "WAH! LIVIN ON A PRAYER";

Re: Hive CLI, Beeline CLI, and SPOT-221

[skip cruse]

+1. I don’t see any obvious downsides, and using beeline could potentially
allow us to eventually use just one interface between Hive and Impala
someday down the road if feel like consolidating some of the code.

On Tue, Apr 28, 2020 at 3:03 PM Jeremy Nelson 
wrote:

> I reviewed https://issues.apache.org/jira/projects/SPOT/issues/SPOT-221
> <
> https://issues.apache.org/jira/projects/SPOT/issues/SPOT-221?filter=allopenissues
> >,
> and in my opinion, this is an easy and straightforward change there is no
> reason not to make. Switching the setup scripts from hive cli to beeline
> cli allows us to set up spot on secure clusters, and doesn’t have any
> obvious downside to this.
>
> Jeremy
>


-- 
if( bool halfWayThere == true);
echo "WAH! LIVIN ON A PRAYER";