Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Storage migration transfers data from one storage device to another. This involves moving blocks of storage and files from storage systems, whether they're on disk, tape or in the cloud. During migration is also an optimal time for organizations to perform data validation and reduction by

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Many thanks for your clarifications. I don't have a team of engineers. Just myself, that I think with much modesty is not little. I'm not familiar with clojure, i know java programming language. The lambda's architecture pipeline i want to build will not be made entirely with clojure. As

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

As much as I would love to convert a new data engineer to the ways of clojure, in my opinion, choosing a language to solve a problem is rarely a wise move. Do you have a team of engineers ready and willing to learn clojure or are you doing this yourself? We do a lot of work with all of the

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Christian writes really good tools. Sparkling is no exception. I have yet to use it in production myself however, since I haven't had the need to use Clojure directly to solve any "data aggregation" problems. Spark and other tools do that well enough, naturally. As far as using a

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Thad, You approach seems very promising to me for a lot of jobs. Spark runs on top of many things. As far as a clojure layer on top, what do you think about sparkling ? On Thu, Jul 4, 2019 at 8:43 AM Thad Guidry wrote: > "Batch" - doing things in

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

probably as Thad says, for a farsighted choice the tool to use for batch processing is Apache Spark. But I'm worried about its learning curve and the time it takes. I don't have much time to develop my map reduce algorithems. I would like to use a consolidated and fairly used tool in

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

"Batch" - doing things in chunks "Processing" - THE WORLD :-) because it means so many different things to so many folks (including your boss) Without a doubt, you will love Apache Spark for your batch processing and writing Spark Programs to conquer any World you are building. Spend time to

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Hi @atdixon and Thad, thanks for your help. I provide more details about my project My big data layer is inspired by Lambda architecture. The pipeline include following layers and related tool choosed to address the issue: - *Nifi* for *data ingestion*, and publisinh data/message on kafka

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

"The best code is never written" https://zeppelin.apache.org/ https://nifi.apache.org/ Thad https://www.linkedin.com/in/thadguidry/ On Tue, Jul 2, 2019 at 11:07 AM orazio wrote: > Hi All, > > I'm newbie on Clojure/Big Data, and i'm starting with hadoop. > I have installed Hortonworks HDP 3.1

Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

My biased first reaction to Hadoop is, do you really need it? It has a separate runtime, some overhead. And it seems to me it much easier to use Kafka, probably connect to get data in/out and Streams/Ksql to process the data. Because of Java interop and the nice generic Kafka Api it's really

Re: Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

I've found Clojure to be an excellent fit for big data processing for a few reasons: - the nature of big data is that it is often unstructured or semi-structured, and Clojure's immutable ad hoc map-based orientation is well suited to this - much of the big data ecosystem is Java or JVM-based

Clojure is a good choice for Big Data? Which clojure/Hadoop work to use?

Hi All, I'm newbie on Clojure/Big Data, and i'm starting with hadoop. I have installed Hortonworks HDP 3.1 I have to design a Big Data Layer that ingests large iot datasets and social media datasets, process data with MapReduce job and produce aggregation to store on HBASE tables. For now, my