Re: Drawing Big Data tech diagrams using Pen Tablets

[ayan guha]

FWIW...I use draw.io and it is pretty neat

On Thu, Sep 13, 2018 at 6:46 AM, Gourav Sengupta 
wrote:

> well, it may be possible to use just excel shapes ( and advanced shapes
> instead of draw.io). I think that free form, hand drawing and changing
> them to solid shapes with two taps in iPad apps and Surface apps render a
> fantastic expressive quality to diagrams. Ofcourse, as with everything, its
> a matter of choice.
>
> Regards,
> Gourav Sengupta
>
> On Wed, Sep 12, 2018 at 7:45 PM Mich Talebzadeh 
> wrote:
>
>> thanks Jorn, pretty neat
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Wed, 12 Sep 2018 at 19:38, Jörn Franke  wrote:
>>
>>> You can try cloud services such as draw.io or  similar.
>>>
>>> On 12. Sep 2018, at 20:31, Mich Talebzadeh 
>>> wrote:
>>>
>>> Hi Gourav,
>>>
>>> I have an IPAD that my son uses it and not me (for games). I don't see
>>> much value in spending $$$ on Surface. Then I had montblanc augmented paper
>>> that kinf of impressive but not really that practical.
>>>
>>> I have Visio 2010 and an upgrade to 2016 will cost good money. So I
>>> decided to get this pen tablet after seeing these diagrams. They are pretty
>>> cool
>>>
>>> 
>>>
>>> In UK it retails at £495 so not cheap but much cheaper than surface pro
>>> which is a premium brand.
>>>
>>> Cheers,
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Wed, 12 Sep 2018 at 18:34, Gourav Sengupta 
>>> wrote:
>>>
 Hi Mich,

 does Apple iPad and Surface not have their own tools for this?

 Regards,
 Gourav Sengupta

 On Tue, Sep 11, 2018 at 8:21 PM Mich Talebzadeh <
 mich.talebza...@gmail.com> wrote:

> Hi,
>
> Sorry this question may not be related to Spark but sure many people
> use MS Visio or other graphics tools for architecture type diagrams with
> big data including Spark.
>
> Visio is time consuming and cumbersome although that is what I use for
> tech diagrams. I tried Montblanc augmented paper
> but
> that is clumsy and expensive.
>
> I was wondering if anyone has tried some tools like Wacom Intuos Pro
> Paper Edition Pen Tablet
> 
> or any similar tools for easier drawing and their recommendation?
>
> Thanks,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any loss, damage or destruction of data or any other property which may
> arise from relying on this email's technical content is explicitly
> disclaimed. The author will in no case be liable for any monetary damages
> arising from such loss, damage or destruction.
>
>
>



-- 
Best Regards,
Ayan Guha

Re: Drawing Big Data tech diagrams using Pen Tablets

[Gourav Sengupta]

well, it may be possible to use just excel shapes ( and advanced shapes
instead of draw.io). I think that free form, hand drawing and changing them
to solid shapes with two taps in iPad apps and Surface apps render a
fantastic expressive quality to diagrams. Ofcourse, as with everything, its
a matter of choice.

Regards,
Gourav Sengupta

On Wed, Sep 12, 2018 at 7:45 PM Mich Talebzadeh 
wrote:

> thanks Jorn, pretty neat
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 12 Sep 2018 at 19:38, Jörn Franke  wrote:
>
>> You can try cloud services such as draw.io or  similar.
>>
>> On 12. Sep 2018, at 20:31, Mich Talebzadeh 
>> wrote:
>>
>> Hi Gourav,
>>
>> I have an IPAD that my son uses it and not me (for games). I don't see
>> much value in spending $$$ on Surface. Then I had montblanc augmented paper
>> that kinf of impressive but not really that practical.
>>
>> I have Visio 2010 and an upgrade to 2016 will cost good money. So I
>> decided to get this pen tablet after seeing these diagrams. They are pretty
>> cool
>>
>> 
>>
>> In UK it retails at £495 so not cheap but much cheaper than surface pro
>> which is a premium brand.
>>
>> Cheers,
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Wed, 12 Sep 2018 at 18:34, Gourav Sengupta 
>> wrote:
>>
>>> Hi Mich,
>>>
>>> does Apple iPad and Surface not have their own tools for this?
>>>
>>> Regards,
>>> Gourav Sengupta
>>>
>>> On Tue, Sep 11, 2018 at 8:21 PM Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 Hi,

 Sorry this question may not be related to Spark but sure many people
 use MS Visio or other graphics tools for architecture type diagrams with
 big data including Spark.

 Visio is time consuming and cumbersome although that is what I use for
 tech diagrams. I tried Montblanc augmented paper
 but
 that is clumsy and expensive.

 I was wondering if anyone has tried some tools like Wacom Intuos Pro
 Paper Edition Pen Tablet
 
 or any similar tools for easier drawing and their recommendation?

 Thanks,

 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com


 *Disclaimer:* Use it at your own risk. Any and all responsibility for
 any loss, damage or destruction of data or any other property which may
 arise from relying on this email's technical content is explicitly
 disclaimed. The author will in no case be liable for any monetary damages
 arising from such loss, damage or destruction.



>>>

Re: Drawing Big Data tech diagrams using Pen Tablets

[Mich Talebzadeh]

thanks Jorn, pretty neat


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 12 Sep 2018 at 19:38, Jörn Franke  wrote:

> You can try cloud services such as draw.io or  similar.
>
> On 12. Sep 2018, at 20:31, Mich Talebzadeh 
> wrote:
>
> Hi Gourav,
>
> I have an IPAD that my son uses it and not me (for games). I don't see
> much value in spending $$$ on Surface. Then I had montblanc augmented paper
> that kinf of impressive but not really that practical.
>
> I have Visio 2010 and an upgrade to 2016 will cost good money. So I
> decided to get this pen tablet after seeing these diagrams. They are pretty
> cool
>
> 
>
> In UK it retails at £495 so not cheap but much cheaper than surface pro
> which is a premium brand.
>
> Cheers,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 12 Sep 2018 at 18:34, Gourav Sengupta 
> wrote:
>
>> Hi Mich,
>>
>> does Apple iPad and Surface not have their own tools for this?
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Tue, Sep 11, 2018 at 8:21 PM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Sorry this question may not be related to Spark but sure many people use
>>> MS Visio or other graphics tools for architecture type diagrams with big
>>> data including Spark.
>>>
>>> Visio is time consuming and cumbersome although that is what I use for
>>> tech diagrams. I tried Montblanc augmented paper
>>> but
>>> that is clumsy and expensive.
>>>
>>> I was wondering if anyone has tried some tools like Wacom Intuos Pro
>>> Paper Edition Pen Tablet
>>> 
>>> or any similar tools for easier drawing and their recommendation?
>>>
>>> Thanks,
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>

Re: Drawing Big Data tech diagrams using Pen Tablets

[Jörn Franke]

You can try cloud services such as draw.io or  similar.

> On 12. Sep 2018, at 20:31, Mich Talebzadeh  wrote:
> 
> Hi Gourav,
> 
> I have an IPAD that my son uses it and not me (for games). I don't see much 
> value in spending $$$ on Surface. Then I had montblanc augmented paper that 
> kinf of impressive but not really that practical.
> 
> I have Visio 2010 and an upgrade to 2016 will cost good money. So I decided 
> to get this pen tablet after seeing these diagrams. They are pretty cool 
> 
> 
> 
> In UK it retails at £495 so not cheap but much cheaper than surface pro which 
> is a premium brand.
> 
> Cheers,
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> 
>> On Wed, 12 Sep 2018 at 18:34, Gourav Sengupta  
>> wrote:
>> Hi Mich,
>> 
>> does Apple iPad and Surface not have their own tools for this? 
>> 
>> Regards,
>> Gourav Sengupta
>> 
>>> On Tue, Sep 11, 2018 at 8:21 PM Mich Talebzadeh  
>>> wrote:
>>> Hi,
>>> 
>>> Sorry this question may not be related to Spark but sure many people use MS 
>>> Visio or other graphics tools for architecture type diagrams with big data 
>>> including Spark.
>>> 
>>> Visio is time consuming and cumbersome although that is what I use for tech 
>>> diagrams. I tried Montblanc augmented paper but that is clumsy and 
>>> expensive.
>>> 
>>> I was wondering if anyone has tried some tools like Wacom Intuos Pro Paper 
>>> Edition Pen Tablet
>>> or any similar tools for easier drawing and their recommendation?
>>> 
>>> Thanks,
>>> 
>>> Dr Mich Talebzadeh
>>>  
>>> LinkedIn  
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>  
>>> http://talebzadehmich.wordpress.com
>>> 
>>> Disclaimer: Use it at your own risk. Any and all responsibility for any 
>>> loss, damage or destruction of data or any other property which may arise 
>>> from relying on this email's technical content is explicitly disclaimed. 
>>> The author will in no case be liable for any monetary damages arising from 
>>> such loss, damage or destruction.
>>>  

Re: Drawing Big Data tech diagrams using Pen Tablets

[Gourav Sengupta]

Hi Mich,

does Apple iPad and Surface not have their own tools for this?

Regards,
Gourav Sengupta

On Tue, Sep 11, 2018 at 8:21 PM Mich Talebzadeh 
wrote:

> Hi,
>
> Sorry this question may not be related to Spark but sure many people use
> MS Visio or other graphics tools for architecture type diagrams with big
> data including Spark.
>
> Visio is time consuming and cumbersome although that is what I use for
> tech diagrams. I tried Montblanc augmented paper
> but
> that is clumsy and expensive.
>
> I was wondering if anyone has tried some tools like Wacom Intuos Pro
> Paper Edition Pen Tablet
> 
> or any similar tools for easier drawing and their recommendation?
>
> Thanks,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>

Fixing NullType for parquet files

[Stephen Boesch]

When this JIRA was opened in 2015 the parquet did not support null types.
I commented on this JIRA in May that - given parquet now does include that
support - can this bug be reopened ?  There was no response. What is the
correct way to request consideration of re-opening this issue?

https://issues.apache.org/jira/browse/SPARK-10943

Permalink

[image: marmbrus]Michael Armbrust
 added
a comment - 15/Oct/15 18:00

Yeah, parquet doesn't have a concept of null type. I'd probably suggest
they case null to a type CAST(NULL AS INT) if they really want to do this,
but really you should just omit the column probably.

Permalink

[image: wabu]Daniel Davis
 added a
comment - 03/May/18 10:14

According to parquet data types
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md, now a
Null type should be supported. So perhaps this issue should be reconsidered?

Permalink

[image: javadba]Stephen Boesch
 added
a comment - 03/May/18 17:08

Given the comment by Daniel Davis can this issue be reopened?

[Help] Set nThread in Spark cluster

[Aakash Basu]

Hi,

API = ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier

val xgbParam = Map("eta" -> 0.1f,

 |   "max_depth" -> 2,

 |   "objective" -> "multi:softprob",

 |   "num_class" -> 3,

 |   "num_round" -> 100,

 |   "num_workers" -> 2)

I'm running a job which will not work until the number of threads of the
API is equivalent to the num_worker set for Spark.

So, in master = local mode, when I do --master local[n] and also set
num_worker for that API as the same value as n, it works.

But, in cluster I do not know which parameter to control which precisely
takes the call of handling the number of threads. I tried with -

1) spark.task.cpus
2) spark.default.parallelism
3) executor cores

But, none of them works, and the speciality of this issue is, it goes into
a halt while distributing the XGBoost model if the above condition is not
met.


My code is as follows, it works in local mode, but not in cluster, any help?

Code:


>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *import org.apache.spark.sql.types.StringTypeimport
> org.apache.spark.sql.types.DoubleTypeimport
> org.apache.spark.sql.types.StructFieldimport
> org.apache.spark.sql.types.StructTypeval schema = new StructType(Array(
> StructField("sepal length", DoubleType, true),  StructField("sepal width",
> DoubleType, true),  StructField("petal length", DoubleType, true),
> StructField("petal width", DoubleType, true),  StructField("class",
> StringType, true)))val rawInput =
> spark.read.schema(schema).csv("file:///appdata/bblite-data/iris.csv")import
> org.apache.spark.ml.feature.StringIndexerval stringIndexer = new
> StringIndexer().  setInputCol("class").  setOutputCol("classIndex").
> fit(rawInput)val labelTransformed =
> stringIndexer.transform(rawInput).drop("class")import
> org.apache.spark.ml.feature.VectorAssemblerval vectorAssembler = new
> VectorAssembler().  setInputCols(Array("sepal length", "sepal width",
> "petal length", "petal width")).  setOutputCol("features")val xgbInput =
> vectorAssembler.transform(labelTransformed).select("features",
> "classIndex")import ml.dmlc.xgboost4j.scala.spark.XGBoostClassifierval
> xgbParam = Map("eta" -> 0.1f,  "max_depth" -> 2,  "objective" ->
> "multi:softprob",  "num_class" -> 3,  "num_round" -> 100,
> "num_workers" -> 2)val xgbClassifier = new XGBoostClassifier(xgbParam).
>   setFeaturesCol("features").  setLabelCol("classIndex")val
> xgbClassificationModel = xgbClassifier.fit(xgbInput)*



Link of the same question I posted in stackoverflow:
https://stackoverflow.com/questions/52290938/set-nthread-in-spark-cluster-for-xgboost

Thanks,
Aakash.